Kafka is the tool most people mean when they say "events" at a large company. It's often described as a message queue, but that undersells what's different about it. A traditional queue hands out messages and deletes them once they're acknowledged. Kafka keeps every message, in order, for as long as you tell it to, and lets each reader keep its own bookmark.
That one design choice, a log instead of a queue, explains almost everything else about how Kafka behaves.
Kafka stores each topic as a set of append-only, replicated logs called partitions. Readers track their own position, so messages stay after they're read.
A log
A log here means an append-only list. New messages go on the end. Nothing is changed or removed from the middle. Each message gets a number, called its offset, that says where it sits in the list.
offset: 0 1 2 3
[OrderPlaced] [OrderAccepted] [OrderPlaced] [OrderPickedUp] ...
Readers don't remove messages. A reader just remembers "I've processed up to offset 2" and asks for offset 3 next. Another reader can be at offset 0, reading the same messages for its own reasons, at its own pace.
Compare that to a traditional queue like RabbitMQ or SQS:
| Traditional queue | Kafka | |
|---|---|---|
| After a message is processed | Deleted | Kept until retention expires |
| Who tracks progress | The broker, per message | Each reader, as one offset |
| Many readers | Need a copy each, set up in advance | Just read the same log |
| Read old messages again | Not possible once deleted | Move the offset back |
| Order | Mostly in order, not guaranteed with many readers | Strict order within a partition |
That last-but-one row is a big deal. If a bug in the loyalty service gave the wrong points for three hours, you can fix the bug, move its offset back three hours, and process those orders again. With a queue, those messages are gone.