Most failures in a distributed system are short. A packet gets dropped, a server restarts during a deploy, a garbage collection pause freezes a service for 300 ms. Try the same request a moment later and it works. So retrying is the natural next step after a timeout.
But retries are also one of the most common ways a small problem becomes a large outage. A retry is extra load, and it arrives at exactly the moment a service is struggling. This lesson is about retrying in a way that helps a sick service recover instead of finishing it off.
In Module 7 we retried background jobs from a queue, where a job can wait minutes between attempts. Here we're dealing with the harder case: synchronous calls, where a customer is waiting for the answer.
Which failures are worth retrying
Retry only when a second attempt has a real chance of a different result.
| Failure | Retry? | Why |
|---|---|---|
| Timeout | Yes, if the request is safe to repeat | Often a slow moment or one bad server |
| Connection refused or reset | Yes | The server was restarting, or the load balancer picked a dying one |
| 503 Service Unavailable | Yes, with backoff | The service is overloaded or briefly down |
| 429 Too Many Requests | Yes, after the Retry-After time |
You were rate limited (Module 8) and told when to come back |
| 500 Internal Server Error | Maybe, once | Could be a passing glitch, could be a bug that fails every time |
| 400, 401, 403, 404, 422 | No | The request itself is wrong. It will be wrong next time too |
Retrying a 400 is pure waste. Retrying a 500 caused by a null pointer bug triples the load and the log noise without ever succeeding. A 409 Conflict depends on what the server means by it: usually it's final, but the next lesson uses it for "your earlier attempt is still in progress", which is worth retrying after a short wait.