Module 12 / Distributed system pitfalls

Module 12 · lesson 7 of 8 · Read 13 min

Distributed locks

Locks that expire, pauses that last too long, and fencing tokens.

Inside one program, making sure only one thread does something at a time is easy: take a mutex, do the work, release it. If the thread crashes, the whole process goes with it, so nobody is left waiting on a lock that will never be released.

Across machines, you need the same guarantee: only one server does this job right now. That's a distributed lock. It sounds like a mutex with a network in the middle. It's much harder than that, for reasons you've already met in this module: machines crash, pause and disagree about time. This lesson builds a lock step by step and shows where each version breaks.

Why Tiffin needs one

Tiffin assigns riders to orders zone by zone. Every few seconds, a dispatcher looks at all waiting orders and all free riders in, say, Koramangala, and matches them up. If two dispatchers ran for the same zone at the same moment, both might give the same rider two different orders, or give one order two riders.

There are 12 dispatcher servers, for redundancy. Each one loops over zones. Before it dispatches a zone, it needs to know that no other server is doing that zone right now.

In Module 7 we met a gentler version of this problem: making sure a scheduled job runs on only one server. Same idea, higher stakes.