Here's a bug that hides in almost every service that publishes events. The order service saves a new order to its database, then publishes OrderPlaced to Kafka. Two separate systems, two separate writes. Most of the time both succeed. Once in a while, one does and the other doesn't, and there's nothing in the code to notice.
This is called the dual write problem. The outbox pattern is the standard, boring and reliable fix, and it's worth knowing by heart.
Two writes, no transaction
The naive code looks reasonable:
async function placeOrder(cart) {
const order = await db.insert('orders', { ...cart, status: 'placed' })
await kafka.send('orders', { key: order.id, value: { type: 'OrderPlaced', ...order } })
return order
}
Walk through what can go wrong:
| What fails | Database | Kafka | Result |
|---|---|---|---|
| Nothing | Order saved | Event published | Fine |
| Process crashes after the insert | Order saved | Nothing | Restaurant never hears about the order |
| Kafka is down or times out | Order saved | Nothing, or maybe something | Same, or a retry publishes twice |
| Reverse the order: publish first, then the insert fails | Nothing | Event published | Restaurant cooks food for an order that doesn't exist |
At 4 million orders a day, a failure rate of even 1 in 100,000 is 40 orders a day where the customer paid and no kitchen saw it. Those turn into angry support tickets and refunds, and they're very hard to debug, because no error was logged at the moment it mattered.
Can we wrap both in one transaction? Not really. The database transaction can't include Kafka. Two-phase commit across the database and the broker is possible in theory. For most of its life Kafka couldn't take part in that protocol at all (recent versions added opt-in support, off by default), and Module 16 explains why you'd avoid it anyway.