How Kafka Consumer Works?

3 min readMar 28, 2022

This article will help you understand how Kafka Consumer API works. Kafka plays a significant role for companies to implement their event-based architectures. It is not possible to learn Kafka in a single article, but giving simple consumer examples can be a good start for Kafka. Before diving deeper into consumer scenarios, it might be good to explain the main concepts.

Kafka Consumer: Consumer API allows you to read messages from topics in Kafka. Consumer applications need to subscribe to one or more Kafka topics.

Consumer Group: Kafka consumers are necessarily part of a consumer group. In the following examples, we will understand the concept of consumer groups more clearly.

Topic: We can think of the Topic as the category of messages we will send. (Database table or folder)

Partition: Smallest storage unit in a topic. You can configure the partition count for any Kafka topic.

Scenarios :

We configured the test topic as four partitions and will use the same Kafka topic for all scenarios. We will not specify key and partition. Our test Kafka Producer will send a few city names (each city name as a separate message) to the “cities” topic. We will use Confluent Cloud for Kafka, Confluent CLI as a Kafka Producer and my simple Golang code as a Kafka Consumer.

You can see the producer code below.

Scenario 1: 1 Consumer — 1 Consumer Group

In this scenario, “consumer-1” will get all messages from all four partitions of the “cities” topic because it is the only consumer in “consumer-group-1”.

Scenario 2: 2 Consumers — 1 Consumer Group

If we have two consumers, each consumer will get messages from two different partitions. In this example, consumer 1 took the messages from partitions 0 and 1, and consumer 2 took partitions 2 and 3.