Retries and dead letters

It's 8:10pm on a Friday. Tiffin's email provider starts returning 503 Service Unavailable. Receipt jobs start failing, about 11 a second.

With the queue from the last lesson, nothing is lost: a failed job just isn't acknowledged, and it comes back after the visibility timeout. But "comes back every 30 seconds, forever" isn't a plan. In this lesson you'll decide when to retry, how long to wait between tries, when to give up, and what "giving up" should mean.

Not every failure deserves a retry

The first question about a failed job is: would trying again help?

The first two are transient failures. They go away on their own. The rest are permanent: the same job will fail the same way on every attempt.

Your worker should tell them apart. A permanent failure should skip straight to "give up" (more on that below) instead of burning ten attempts over two hours. A good habit is to have job code raise two kinds of errors: a "retry me" error and a "don't bother" error, and let the queue library handle each.

Waiting between retries

The simplest retry is "try again immediately". During an outage, that's the worst choice. The provider is struggling, and you're sending it 11 jobs a second plus every retry of every failed job. You're adding load to a system that's already falling over.

Instead, wait longer after each failure. This is exponential backoff: double the delay each time, up to a cap.

All ten attempts together span about 85 minutes. A short blip is covered by the first few retries, and a long outage still gets a few tries spread out over more than an hour.

Add jitter

There's one more problem. At 8:10pm, thousands of jobs failed within the same few seconds. With plain backoff, they all retry 10 seconds later, together. Then 20 seconds later, together. The provider sees a wall of traffic every time it tries to recover.

The fix is jitter: pick a random delay instead of the exact one.

function retryDelay(attempt) {
  const base = 10_000            // 10 seconds
  const cap = 60 * 60 * 1000     // 1 hour
  const max = Math.min(cap, base * 2 ** (attempt - 1))
  return Math.random() * max     // "full jitter": anywhere from 0 to max
}

Now the retries spread evenly over each window instead of arriving as a spike. Retries and backoff come up again in Module 12, for calls between services, where getting them wrong can turn a small outage into a big one.

In the Postgres queue from the last lesson, a retry is just pushing run_at into the future and releasing the lock:

UPDATE jobs
SET run_at = now() + :delay * interval '1 millisecond',
    locked_until = NULL
WHERE id = :id;

SQS doesn't do backoff for you. If a worker simply doesn't acknowledge a failed message, it comes back when the visibility timeout ends, every 30 seconds, however many times it has failed. To back off, the worker calls ChangeMessageVisibility on the failed message with the delay it wants (up to 12 hours), so the message stays hidden that long before its next try. The receive count still goes up by one each time, so the dead-letter limit below keeps working.

Poison messages

Now a different failure. A restaurant typed its name with an unusual character that makes the receipt template crash. Every attempt to send that receipt throws the same error.

A job that can never succeed is called a poison message. Without a limit, it causes trouble:

• It retries forever, using worker time on every attempt.
• If it crashes the whole worker process (out of memory, say), it takes other jobs down with it every time it's picked up.
• In an ordered queue (like an SQS FIFO queue), it can block every job behind it in the same message group.
• Your error dashboard fills with the same error, hiding new ones.

So every job type needs a maximum number of attempts. After that, stop trying. But stop in a way that doesn't lose the job.

Dead-letter queues

A dead-letter queue is where jobs go after they've failed too many times: out of the way of healthy work, but kept, so a human can look at them.

%% caption: After the last attempt, the job moves aside instead of disappearing.
flowchart TD
  Q[[Queue]] --> K[Worker]
  K -- "attempts 1 to 10" --> E[Email provider]
  K -. "failed: retry later, with backoff" .-> Q
  K -- "still failing after 10" --> D[["Dead-letter queue<br/>kept for a human"]]:::hot

In SQS you attach a "redrive policy" to a queue with a maxReceiveCount, say 10. After 10 receives without an acknowledgement, SQS moves the message to the dead-letter queue you named. In a Postgres queue, add status and last_error columns to the jobs table, and a dead job is just a status change:

UPDATE jobs SET status = 'dead', last_error = :error WHERE id = :id;

The workers' claim query then also checks status = 'ready'.

A dead-letter queue is not a bin

The worst thing you can do with a dead-letter queue is not look at it. Every job in it is a customer without a receipt, or a restaurant without a payout. Treat it like this:

1. Alert when it's not empty. For important jobs, one dead job is worth a message to the team channel.
2. Read the error. Usually it's a bug or bad data, and the fix is a code change or a data correction.
3. Redrive. Once fixed, move the jobs back to the main queue to run again. SQS has a button for this. In Postgres, it's an UPDATE back to ready with attempts = 0.

Notice step 3. A redriven job might have half-succeeded the first time. That's one more reason jobs must be safe to run twice, which is the next lesson.

Timeouts inside a job

One more trap, and it's subtle. Your visibility timeout is 30 seconds. A worker calls the email provider, which hangs for 45 seconds before answering.

At second 30, the queue decides the worker must be dead and hands the job to another worker. Now two workers are sending the same receipt. When the first one finally finishes and tries to acknowledge, the job may already be gone, or the acknowledgement may fail.

The rule: every external call inside a job needs its own timeout, well below the visibility timeout. For a 30-second visibility timeout, give the email call a 10-second timeout. If it times out, that's a transient failure and the job retries with backoff.

For jobs that are legitimately long, like exporting every order from last quarter to a CSV in 3 minutes, either set a longer visibility timeout for that queue or have the worker extend it while it works. SQS lets a worker call ChangeMessageVisibility every minute or so as a heartbeat. In the Postgres queue, the worker pushes locked_until forward.

What to watch

A background system fails quietly, so you need to make it loud. These four numbers cover most problems:

The last one catches the sneakiest outage: workers that crashed on deploy and never came back. The queue fills, nobody gets receipts, and no errors appear because nothing is running.