Module 12 / Distributed system pitfalls

Module 12 · lesson 8 of 8 · Read 9 min

Cascading failures

One slow service takes down five. Circuit breakers and bulkheads.

What you'll learn

  1. How a slowdown spreads
  2. Slow is worse than dead
  3. Loops that make it worse
  4. Circuit breakers
  5. Bulkheads
  6. Load shedding
  7. Putting it together

In a system with many services, the most dangerous failures aren't the ones where something breaks. They're the ones where something breaks and takes its neighbours down with it. One slow service, one missing timeout, one eager retry loop, and ten minutes later the whole app is down, including parts that never had a problem.

That chain reaction is a cascading failure. This lesson pulls together everything in the module: you'll see how timeouts, retries and overload combine into one, and meet the three tools built specifically to stop it: circuit breakers, bulkheads and load shedding.

How a slowdown spreads

It's Saturday, 8:20pm. Tiffin's reviews service, which shows star ratings on restaurant pages, starts responding slowly. Its database is running a badly planned query, and calls that took 30 ms now take 8 seconds.

Here's what happens next:

%% caption: The reviews service is the only thing that's actually broken. Everything that depends on it fails because of it.
flowchart TD
  C(["Customers"]) --> GW["API gateway<br/>requests queue up"]
  GW --> RP["Restaurant page service<br/>threads stuck waiting"]
  GW --> OS["Order service<br/>can't get through"]
  RP --> RV["Reviews service<br/>slow, 8 s per call"]:::hot
The reviews service is the only thing that's actually broken. Everything that depends on it fails because of it.
  1. The restaurant page service calls reviews on every page load, with a 10-second timeout. Each call now holds a worker thread for 8 seconds.
  2. Little's law from the timeouts lesson: 400 page loads a second × 8 s = 3,200 threads needed. The service has 200. Within a second its pool is full.
  3. Now every request to the restaurant page service waits, including ones that don't need reviews at all.
  4. The API gateway's connections to the restaurant page service fill up. Requests to other services through the gateway start waiting in the same queue.
  5. Customers see spinners, give up, and pull to refresh. Load goes up.
  6. Checkout goes through the same gateway. Orders stop.