You've used Kafka as a box in many designs in this course: "put a queue here". This lab opens the box. You'll design a distributed, durable message log like Kafka from a blank page, and see why its few core ideas (an append-only log, partitions, replicas that must keep up, and consumers that track their own position) give it both huge throughput and a strong promise: an acknowledged message is never lost.
Module 15 explained how to use Kafka. Here you're building it.
The brief
You're building the event backbone for a large company. Hundreds of services publish events (orders placed, payments captured, pages viewed, locations reported). Other services read them: some in real time, some hours later, some by replaying a week of history.
Who uses it?
- Producers are services that write events. Some write a few a minute, the clickstream writes hundreds of thousands a second.
- Consumers read events, in groups that share the work. Several independent groups read the same stream.
- Platform engineers run the cluster, add machines and replace failed disks.
In scope: topics split into partitions, durable replicated storage, producers with configurable durability, consumer groups with stored offsets, time and size based retention, and log compaction.
Out of scope: exactly-once transactions across topics (we'll mention where they plug in), schema registries, cross-datacentre mirroring, and stream processing frameworks built on top.
Requirements
Functional
- Producers append messages to a named topic. A message has an optional key, a value, and headers.
- Messages with the same key are kept in order relative to each other.
- Consumers read messages in order within a partition, and can re-read from any earlier position.
- Several consumer groups can read the same topic independently. Within a group, each partition goes to one consumer.
- Messages are kept for a configured time (say 7 days) or, for compacted topics, the latest value per key is kept forever.
Non-functional
- Durable: once a write is acknowledged (with the strongest setting), it survives the loss of any one broker, without data loss.
- High throughput: about 1 GB/s of writes at peak across the cluster.
- Low latency: a message is readable by consumers within ~10 ms of being acknowledged, at p99 under normal load.
- Highly available: the cluster keeps accepting writes and serving reads when a broker dies.
- Horizontally scalable: add brokers to add capacity.
The interesting tension: durability wants to wait for copies on several machines, and throughput and latency want to wait for nothing.