Every server has a limit. Say one Tiffin checkout server handles 500 requests a second comfortably. What happens at 700?
The naive hope is that it serves 500 and the other 200 wait a bit. What really happens is worse. Requests pile up in queues. Each one waits longer, so it holds memory and a connection longer. Clients time out after 2 seconds and retry, which adds even more requests. Soon the server spends its time on requests whose callers have already given up. It's 100% busy and doing almost nothing useful.
This lesson is about avoiding that collapse. You'll learn to say no early, to the right requests, so you can keep saying yes to the rest.
Throughput is not goodput
Two numbers matter under load:
- Throughput: how many requests the server processes per second.
- Goodput: how many of those finish in time to be useful, before the client stops waiting.
Below capacity, every request succeeds. At capacity, latency climbs but things mostly work. Past capacity, with unbounded queues, latency grows without limit. If every request waits 3 seconds in the queue and clients give up at 2, goodput drops to zero even though the CPU is pegged. This is called congestion collapse, and retries (Module 12) make it worse because each timeout turns into another request.
The fix is to keep the server near the left side of that picture, no matter how much traffic arrives.