Tiffin's dispatcher passed every test the team wrote. It still assigned rider Ravi to two orders one Friday, because a garbage-collection pause landed between taking a lock and using it, exactly the case from the distributed locks lesson. Nobody had written a test for "pause for 12 seconds at line 84", and nobody could have listed every place a pause might land.
That's the problem with testing distributed systems. The bugs aren't in any one function. They're in rare orderings of events: this message arrives after that timeout, this node pauses during that failover. A normal test runs one ordering, the one that happens on a quiet laptop. This lesson covers the techniques that go looking for the others, from cheapest to most thorough.
Why ordinary tests miss these bugs
Three things make distributed bugs hard to catch:
- Nondeterminism. Thread scheduling, network delays and clock readings differ on every run. A test that fails once in 10,000 runs looks like a flaky test and gets retried until it passes.
- Rare failures. Partitions, disk errors and long pauses happen in production weekly across a big fleet, but almost never in a 30-second test.
- Bugs need combinations. It's rarely one fault. It's a partition and a leader election and a retry, in a specific order.
So you need some way to explore many orderings and failures on purpose, and a precise statement of what "correct" means so you can tell when one goes wrong.