Failover

It's 8:15pm on a Friday, the busiest fifteen minutes of Tiffin's week. The machine running the primary database has a hardware fault and goes silent.

Here's what customers notice in the first few seconds. Menus still load, because they come from Redis and the replicas. Restaurant lists, search and reviews still work. But every write fails. Nobody can place an order. Kitchen tablets can't accept orders. Riders can't mark food as picked up. Tiffin is, for practical purposes, closed.

You have two replicas with almost all the data. One of them needs to become the new primary, and everything that wrote to the old one needs to start writing to it instead. That process is called failover.

Failover is replacing a failed primary with one of its replicas: detecting the failure, choosing and promoting a replica, pointing all writers at it, and making sure the old primary can't come back and accept writes.

Each part of that sentence has its own trap.

The timeline

Here's what a well-run automated failover looks like:

About 40 seconds of failed writes. Now compare it with a manual failover at 8:15pm: the alert fires, an engineer gets paged, opens a laptop, works out what's going on, finds the runbook, checks which replica is furthest ahead, promotes it, and updates the config. 20 to 40 minutes is typical, at the worst possible time.

Step 1: Is it really dead?

The system watches the primary with heartbeats, small checks every few seconds. If several in a row fail, it declares the primary dead.

The hard part is choosing how long to wait:

• Too short, say 5 seconds, and you'll fail over for things that weren't failures: a disk stall during a heavy checkpoint, a brief network blip, a primary that's slow because it's busy. Every failover has a cost, including possible data loss, so false alarms are expensive.
• Too long, say 5 minutes, and a real failure means 5 minutes of no orders.

Most setups land somewhere between 10 and 60 seconds. Tiffin uses 30.

There's a second subtlety: who decides. If one monitoring server can't reach the primary, maybe the primary is dead, or maybe the monitoring server's own network is broken. Good failover tools ask several observers and need a majority to agree, usually through a small consensus store like etcd or ZooKeeper. Module 12 covers why "I can't reach it" doesn't mean "it's dead", and Module 13 how a group agrees on a new leader.

Step 2: Which replica, and what's lost?

The system picks the replica that has replayed the most of the primary's log, because it's missing the least.

But with asynchronous replication, it's still missing something. Any write the primary confirmed but hadn't yet sent to a replica is gone. Here's what that means for Tiffin at dinner:

writes per second:        1,200
typical replica lag:      100-300 ms
writes lost on failover:  1,200 x 0.1 to 1,200 x 0.3 = 120 to 360 writes

Some of those writes are orders. Customers were told "order confirmed", and the order no longer exists. Some are payment records. Sorting that out by hand takes days.

Synchronous replication

To lose nothing, the primary must wait for at least one replica to confirm it has the change before telling the app "committed". This is synchronous replication. In Postgres, it's configured with settings such as synchronous_standby_names.

The cost is time on every write. A replica in another zone of the same region adds roughly 1-2 ms per commit, which most apps can afford. A replica in another region can add 50-100 ms, which most can't. There's another catch: if the synchronous replica goes down, the primary has nobody to confirm to, and writes stall. So setups usually have two candidates and need confirmation from either one.

Many managed databases do this for you. A "multi-AZ" deployment on most clouds is a primary with a synchronous standby in another zone, and failover typically completes in one to two minutes.

For Tiffin, losing confirmed orders is not acceptable, so the plan is one synchronous standby in a second zone, plus asynchronous replicas for reads. Module 13 goes deeper into the choices in between.

Step 3: Point everyone at the new primary

Promotion itself is quick: in Postgres, one command (pg_promote()) turns a replica into a primary in a second or two. The slow, error-prone part is getting every writer to use it. The common options:

Tiffin's app servers already talk to PgBouncer, so failover just means updating PgBouncer's target and reconnecting. Tiffin's failover tool is Patroni, a popular choice for Postgres. Patroni promotes the replica and repoints the other replicas at it, and it publishes which node is now the primary; a small hook script (or a health-checking proxy such as HAProxy in front of the databases) uses that to repoint PgBouncer.

Step 4: Keep the old primary down

This is the step people skip, and it causes the worst outcome of all.

Suppose the old primary wasn't dead. Its network link was cut for 40 seconds and then came back. It still thinks it's the primary. Some app servers still have connections open to it, and they start writing to it again. Now there are two primaries, each accepting different orders, each handing out order IDs from its own sequence. Order 88,101 exists on both, for different customers.

This is called split brain, and cleaning it up means comparing two diverged databases row by row. It's far worse than 40 seconds of downtime.

The fix is called fencing: making sure the old primary cannot accept writes before the new one starts. Common ways:

• Shut it off. The failover tool uses the cloud API to stop the old machine, or cuts its power. Crude and effective.
• A lease. The primary must keep renewing a lease in the consensus store. If it can't, for example because it's cut off from the network, it demotes itself to read-only before the lease runs out. A new primary is only promoted after the old lease has definitely expired. Patroni works this way.
• Cut its connections. Make sure the pooler and network no longer route anything to it.

Module 12 covers fencing tokens, the general version of this idea.

What the app sees

Even a perfect failover means 30-60 seconds of failed writes and broken connections. The app should handle that on purpose:

• Reconnect automatically. Connection pools should drop dead connections and open new ones. Test that they do.
• Retry carefully. If a POST /orders failed with a connection error, the order may or may not have been committed. Blindly retrying may create a duplicate. This is what idempotency keys are for (Module 12).
• Degrade, don't crash. Menus, search and order history can keep working from replicas and cache. Checkout can show "We're having a moment, please try again in a minute" instead of a stack trace.

Practise it

A failover process that has never been run is a guess. Tiffin's team does a deliberate failover on a quiet Tuesday afternoon once a month. Each time, they measure how long writes failed, check how much was lost, and fix whatever surprised them. The first time, it took 4 minutes, because one service cached the database's DNS name forever. Far better to learn that on a Tuesday afternoon than at 8:15pm on a Friday.

%% caption: Tiffin's failover. The synchronous standby has every committed write, so nothing confirmed is lost.
flowchart TD
  A[("Zone A primary<br/>fails at 8:15pm")]:::hot -. "declared dead" .-> B[("Zone B sync standby<br/>promoted")]:::store
  B --> PB["PgBouncer<br/>repointed"] --> S["App servers<br/>writes resume"]
  R[("Replicas")]:::store -- "now follow the new primary" --> B