Module 12 / Distributed system pitfalls

Module 12 · lesson 6 of 8 · Read 9 min

Counting across machines

Likes, stock and seats when many servers update one number.

Adding one to a number is the first thing anyone learns to program. On one machine with one thread, it just works. Put that number in a shared database and have fifty servers add to it at once, and it quietly gives wrong answers.

Tiffin has numbers like this everywhere: how many "free dessert" coupons are left, how many people liked a dish, how many orders a restaurant has open right now. This lesson is about keeping them right when many servers change them at the same time, and about when "roughly right" is good enough.

The lost update

Tiffin runs a promotion: the first 100 orders at Biryani House tonight get a free gulab jamun. The count of coupons left lives in a table:

CREATE TABLE promos (id INT PRIMARY KEY, remaining INT NOT NULL);

The obvious code reads the number, checks it, and writes it back:

const { remaining } = await db.one('SELECT remaining FROM promos WHERE id = 7')
if (remaining > 0) {
  await db.query('UPDATE promos SET remaining = $1 WHERE id = 7', [remaining - 1])
  grantDessert(order)
}

Now two orders arrive at the same moment, on two different servers:

Time Server 1 Server 2 remaining in the database
1 Reads 1 1
2 Reads 1 1
3 Writes 0, grants dessert 0
4 Writes 0, grants dessert 0

Two desserts, one coupon. Both servers did something perfectly reasonable, but each acted on a value that was already stale by the time it wrote. This is the lost update: one write overwrote another without either noticing.

On a quiet afternoon this almost never happens. At 8pm, when the promotion goes live and 2,000 people tap "Order" in the same minute, it happens constantly, and the restaurant gives away 140 desserts instead of 100.

A lost update happens when two clients read the same value, each change it, and each write it back, so one change silently overwrites the other.