In June 2011, three LinkedIn engineers, Jay Kreps, Neha Narkhede and Jun Rao, presented Kafka: a Distributed Messaging System for Log Processing at the NetDB workshop. Kafka had been running in production at LinkedIn for more than six months, handling hundreds of gigabytes and close to a billion messages a day.
The paper is short and readable, and almost every design choice in it was unusual for a messaging system at the time. Many are now so standard that it's easy to miss how deliberate they were. It's also a good lesson in what a first version leaves out: there's no replication in it at all.
The problem
LinkedIn generated a lot of what the paper calls log data: user activity (logins, page views, clicks, searches) and operational metrics (call latencies, errors, CPU). That data had traditionally fed offline analytics. What had changed was that it was now used in the product (search relevance, recommendations, ad targeting, abuse detection, the news feed) with delays of no more than a few seconds.
Two families of tools existed, and neither fitted:
- Enterprise messaging systems (the paper names JMS implementations, IBM WebSphere MQ and others) offered rich per-message delivery guarantees, which the paper calls overkill for log data: losing the occasional page view isn't a disaster. They also lacked batching (JMS had no way for a producer to send many messages in one request), had weak support for spreading data across machines, and slowed down badly when messages piled up unconsumed.
- Log aggregators (Facebook's Scribe, Yahoo's Data Highway, Cloudera's Flume) were built to load data into Hadoop for offline use, exposed details such as "minute files", and mostly pushed data to consumers.