The last lesson ended with a caller staring at silence. A timeout is the simplest thing you can do about it: decide in advance how long you're willing to wait, and stop waiting after that.
It sounds too obvious to need a lesson. But many popular HTTP clients ship with no timeout at all by default, and a missing or badly chosen timeout is behind a surprising number of real outages. This lesson is about choosing the number, and about being honest about what a timeout does and doesn't tell you.
What happens without one
Say Tiffin's order service calls the restaurant service to check that a restaurant is open before accepting an order. Normally that takes 20 ms. One evening the restaurant service's database locks up, and calls to it just hang.
The order service has no timeout on that call. Each incoming checkout request grabs a worker thread, calls the restaurant service, and waits. And waits.
A handy rule, called Little's law, tells you how many requests are in flight at once:
requests in flight = arrival rate × time each request takes
At dinner time, Tiffin takes about 220 checkouts a second. Say the order service runs on two servers, so each gets about 110.
- Healthy: 110/s × 0.02 s = about 2 requests in flight. A pool of 100 threads is plenty.
- Restaurant service hanging, no timeout: 110/s × ∞ = every thread you have. The pool of 100 is full in under a second.
After that, the order service can't serve anything, including requests that never touch the restaurant service. One sick dependency has taken down a healthy service. That's the start of a cascading failure, which gets its own lesson at the end of this module.