Module 17 / Event-driven data and streaming

Module 17 · lesson 3 of 8 · Read 8 min

Change data capture

Turning database changes into a stream.

What you'll learn

  1. The dual-write problem
  2. The database already keeps a list of changes
  3. What a change event looks like
  4. Starting from existing data
  5. What people use it for
  6. The catches

In Module 9 Tiffin added a search index so customers could type "biryani near me". That index is a copy of restaurant and menu data. So is the menu cache in Redis, and so is the analytics warehouse the business team queries every morning. Every time a restaurant changes a dish price, all three copies need to hear about it.

The obvious way is to have the app update each copy after it writes to the database. That way is quietly broken. This lesson shows why, and shows a better source of changes that was sitting inside your database all along.

The dual-write problem

Here's the obvious code:

await db.query('UPDATE menu_items SET price_paise = 26000 WHERE id = 81')
await searchIndex.update(81, { price_paise: 26000 })
await cache.delete('menu:12')

Three things can go wrong.

A crash in the middle. The database update succeeds, then the server dies before the search update. The search index shows ₹240 forever, or until someone edits that dish again.

A failure in the middle. The search cluster is down for a minute. Do you roll back the price change? You can't, it's committed. Do you retry? For how long, while the request hangs?

Two writers racing. Two edits to the same dish happen close together. Server A sets ₹260, server B sets ₹280. In the database, B's update lands last, so the price is ₹280. But the search updates travel separately, and A's update to the index happens to arrive after B's. The index now says ₹260 and the database says ₹280. Nothing crashed. Nothing logged an error. The copies just disagree.

%% caption: Same two writes, applied in different orders in two places. The copies silently diverge.
sequenceDiagram
  autonumber
  participant A as Server A
  participant D as Database
  participant S as Search index
  participant B as Server B
  A->>D: price 260
  B->>D: price 280
  Note over D: ends at 280
  B->>S: price 280
  A->>S: price 260, arrives late
  Note over S: ends at 260
Same two writes, applied in different orders in two places. The copies silently diverge.

The root problem is that the database and the index are two systems with no shared transaction. Writing to both from the app can never be made fully reliable.