Module 18 / High availability and multi-region

Module 18 · lesson 2 of 7 · Read 8 min

Removing single points of failure

Redundancy at every layer, and what it costs.

What you'll learn

  1. Finding them
  2. Spread across zones
  3. Stateless parts: run more than you need
  4. Stateful parts: the hard ones
  5. The hidden ones
  6. What redundancy costs
  7. Untested redundancy is a guess

At 7:52pm on a Friday, a single network switch in Tiffin's data centre fails. Every app server is fine. The database is fine. But the database happens to sit behind that one switch, so for 25 minutes nothing can reach it, and the whole app is down at the start of dinner.

That switch was a single point of failure: one part whose loss takes the whole system down with it. This lesson is about finding those parts, removing the ones that matter, and knowing what that costs.

A single point of failure (SPOF) is any one component, of any kind, whose failure alone is enough to stop the system from working.

"Of any kind" is the important bit. A SPOF can be a server, a cable, a config file, a cloud account, a TLS certificate or one engineer who's the only person who knows the database password.

Finding them

The method is simple and a bit tedious. Draw the path a request takes, from the customer's phone to the database and back. Point at each box and ask: if only this one thing breaks, does ordering stop?

%% caption: Tiffin's request path, with the obvious single points of failure highlighted.
flowchart TD
  CA(["Customer app"]) --> DNS["DNS"]
  DNS --> LB["Load balancer<br/>one instance"]:::hot
  LB --> APP["App servers<br/>dozens of them"]
  APP --> DB[("Database primary<br/>one machine")]:::hot
  APP --> PP["Payment provider"]
Tiffin's request path, with the obvious single points of failure highlighted.

Then go past the drawing. The boxes on the diagram are the easy part. Ask the same question about:

  • the network: switches, the link to the internet, the NAT gateway,
  • the zone or data centre everything sits in,
  • shared services: DNS provider, config store, secrets manager, service discovery,
  • the deploy pipeline and the one CI runner it uses,
  • certificates and domain registrations that expire,
  • people and access: who can log in when things break?

You'll end up with a long list. Not every item needs fixing. The next sections show how to fix the ones that do.