Module 15 / Messaging and event logs

Module 15 · lesson 3 of 8 · Read 8 min

Consumer groups

Sharing the work, and rebalancing when a consumer dies.

What you'll learn

  1. One group per service
  2. Partitions cap your parallelism
  3. Offsets and commits
  4. Rebalancing
  5. Consumer lag

In the first lesson of this module we said real systems need two behaviours at once: every service gets every message, but within a service, each message is handled by only one of its instances. Kafka does both with one idea, the consumer group.

This lesson covers how a group shares a topic's partitions, why the number of partitions caps how far a group can scale, and what happens during the awkward few seconds when a consumer joins, leaves or dies.

One group per service

Every Kafka consumer belongs to a group, named by a group.id string. The rules are short:

  1. Every group gets every message in the topic. Groups don't affect each other at all.
  2. Within a group, each partition is assigned to exactly one consumer. That consumer reads all of that partition's messages, in order.

So the notifications service runs its instances with group.id = notifications, and the dispatch service uses group.id = dispatch. Both read the whole orders topic. Within notifications, the partitions are divided up among its instances.

A consumer group is a set of consumers that share a topic's partitions between them, each partition read by exactly one member, while every group gets every message.

Partitions 0 to 5 of orders
→
Group notifications3 consumers, 2 partitions each
Group dispatch2 consumers, 3 partitions each
Two groups reading the same six partitions. Each group gets everything. Within a group, partitions are shared out.

Each group keeps its own offsets: notifications might be at offset 81,400 in partition 0 while dispatch is at 81,435. A slow group never holds up a fast one.