In the mid-2000s, Google had a pile of systems that each needed exactly one of something: one GFS master, one Bigtable master, one primary per shard of some service. Every team was solving "who's in charge?" on its own. Some used home-made election code that worked when duplicated work was harmless. Others simply paged a human when the leader died. Distributed consensus was understood in theory (Paxos was well known), but getting a production team to build it correctly into their own service was another matter.
Google's answer was not a better consensus library. It was a small, boring, highly available service that every team could share: a lock service with a file system interface. It's called Chubby, and most of the coordination tools you'll use today, from ZooKeeper to etcd, follow the shape it set.
The paper: The Chubby lock service for loosely-coupled distributed systems, Mike Burrows, OSDI 2006, Google.
What Chubby is
A Chubby cell is usually five servers called replicas. They run Paxos to elect one of themselves as master, and the master holds a master lease: the replicas promise not to elect anyone else for a few seconds, and they keep renewing that promise while the master keeps winning votes. Three of the five must be up for the cell to work.
- Writes go from the master to all replicas through the consensus protocol and are acknowledged once a majority has them.
- Reads are answered by the master alone. That's safe only because of the lease: while it's valid, no other master can exist, so the master can't be serving stale data.
- Clients find the master by asking any replica listed in DNS, then talk only to the master.