Gossip spreads the news that a node died. But somebody has to decide it died first. That sounds easy: if a node stops answering, it's dead. In Module 12 you saw why it isn't. A node that doesn't answer might be dead, or crashed and restarting, or stuck in a 4-second garbage collection pause, or fine but behind a congested switch. From the outside, all of these look the same: silence.
So every distributed system has a failure detector, a component whose only job is to turn silence into a decision. This lesson is about how to make that decision well.
The problem: two ways to be wrong
A failure detector can make two kinds of mistake:
| Mistake | What happens | Cost in Tiffin |
|---|---|---|
| Too slow: a dead node is still marked alive | Requests keep going to it and time out | Riders' location updates are lost for 30 seconds at dinner time |
| Too eager: a live node is marked dead | Traffic and data move off it, then back again | A GC pause triggers re-replication of 200 GB, which slows the whole cluster |
The second one is sneaky. Declaring a node dead is not free. Its partitions get reassigned, replicas get rebuilt, clients reconnect elsewhere. If the node was only slow, you've paid all that cost for nothing, and you'll pay it again when it comes back. A detector that flips a node between "dead" and "alive" every few seconds (called flapping) can do more damage than the original problem.
There's no setting that avoids both mistakes. Faster detection always means more false alarms. The goal is to make the trade-off explicit and adapt it to the network you actually have.