Queues and workers

You've decided the receipt email can happen after checkout responds. Now you need somewhere for "send this receipt" to wait, and something to actually send it.

The tempting shortcut is to respond to the customer and then send the email in a background thread on the same server. It works on your laptop. In production it loses receipts, and you won't notice for weeks. This lesson shows why, and builds the boring, reliable alternative: a queue and a pool of workers.

Why not just use a background thread?

Picture the shortcut:

app.post('/orders', async (req, res) => {
  const order = await placeOrder(req.body)
  res.status(201).json(order)
  sendReceipt(order.id) // runs after the response, no await
})

Here's what goes wrong:

• Deploys kill it. In Module 4 you set up rolling deploys. Each server is stopped and replaced. Draining waits for in-flight requests, but this send isn't a request any more, so nothing waits for it. Any receipt that was "about to be sent" when the server stopped is gone. With several deploys a day, that's real customers.
• Crashes kill it. A server that runs out of memory takes every pending email down with it.
• No retries. If the email provider returns an error, the thread logs it and the receipt is lost.
• No visibility. You can't answer "how many receipts are waiting right now?" because the answer is scattered across the memory of every server.
• It competes with customers. At 8pm the same CPU and connections serving checkout are also sending emails.

The core problem: the promise "I'll send this soon" exists only in one process's memory. You need to write it down somewhere that outlives any single server.

The queue: a to-do list that survives

A queue is a durable list of work. Producers add jobs to it, workers take jobs off it, and a job stays in the queue until a worker confirms it's done.

The web server becomes a producer. It adds a job and moves on. A separate pool of workers takes jobs and does them.

%% caption: The web server and the worker never talk directly. The queue sits between them and holds the work.
sequenceDiagram
  autonumber
  participant C as Customer app
  participant W as Web server
  participant Q as Queue
  participant K as Worker
  participant E as Email provider
  C->>W: place order
  W->>Q: add job: receipt for 9812
  W-->>C: order placed
  K->>Q: take the next job
  Q-->>K: receipt for order 9812
  K->>E: send the email
  E-->>K: accepted
  K->>Q: done, remove the job

This split buys you a lot:

• The web server responds as soon as the job is safely stored.
• If a worker crashes, the job is still in the queue. Another worker picks it up.
• Workers run as their own deployment. You can scale them, restart them and deploy them without touching checkout.
• Queue length becomes a number you can graph and alert on.
• If the email provider is down for 20 minutes, jobs pile up and wait instead of failing.

What a job looks like

A job is a small message. Keep it to what the worker needs to find everything else:

{
  "type": "send_receipt",
  "order_id": 9812,
  "enqueued_at": "2026-10-18T14:35:02Z"
}

Put IDs in jobs, not copies of the data. If you put the whole order in the message, and the customer changes their address before the worker runs, the worker sends stale data. Reading the order fresh from the database also keeps messages small, which matters because most queues cap message size.

Taking a job: receive, work, acknowledge

The interesting part is how a worker takes a job without losing it if it dies halfway.

A naive queue removes a job the moment a worker reads it. If the worker then crashes, the job is gone. That's at-most-once delivery: each job runs once or not at all.

Real job queues do it in three steps:

1. Receive. The worker gets a job. The queue doesn't delete it. It hides it from other workers for a while. In Amazon SQS this is the visibility timeout, 30 seconds by default.
2. Work. The worker sends the email.
3. Acknowledge. The worker tells the queue "done", and only then is the job deleted.

If the worker crashes at step 2, it never acknowledges. After 30 seconds the job becomes visible again and another worker takes it. No job is lost.

%% caption: The visibility timeout is how a queue survives crashed workers.
sequenceDiagram
  autonumber
  participant W1 as Worker 1
  participant Q as Queue
  participant W2 as Worker 2
  W1->>Q: receive job
  Q-->>W1: receipt 9812, hidden 30 s
  Note over W1: crashes, never acknowledges
  Note over Q: 30 s later, visible again
  W2->>Q: receive job
  Q-->>W2: receipt 9812
  Note over W2: sends the email
  W2->>Q: acknowledge

The price: if worker 1 sent the email and crashed just before acknowledging, worker 2 sends it again. This is at-least-once delivery. Every job runs one or more times. That's the deal almost every real queue offers, and it's why a whole lesson in this module is about jobs that can run twice.

Choosing a queue

You have more options than you might think:

Kafka often appears in this conversation too, but it's a log rather than a job queue, with different trade-offs. It gets its own module (Module 15).

For Tiffin, with a few jobs a second and Postgres already running, a table in the database is a strong first choice. Let's build one.

A queue in Postgres

CREATE TABLE jobs (
  id           bigserial PRIMARY KEY,
  type         text        NOT NULL,
  payload      jsonb       NOT NULL,
  run_at       timestamptz NOT NULL DEFAULT now(),
  locked_until timestamptz,
  attempts     int         NOT NULL DEFAULT 0
);
CREATE INDEX jobs_ready ON jobs (run_at);

A worker claims one job like this:

UPDATE jobs
SET locked_until = now() + interval '30 seconds',
    attempts     = attempts + 1
WHERE id = (
  SELECT id FROM jobs
  WHERE run_at <= now()
    AND (locked_until IS NULL OR locked_until < now())
  ORDER BY run_at
  LIMIT 1
  FOR UPDATE SKIP LOCKED
)
RETURNING id, type, payload;

locked_until is a hand-made visibility timeout. FOR UPDATE SKIP LOCKED is the clever bit: if another worker is in the middle of claiming a row, this worker skips it instead of waiting. Twenty workers can claim jobs at once without stepping on each other. When the job is done, the worker deletes it:

DELETE FROM jobs WHERE id = :id;

A crashed worker leaves locked_until in the past after 30 seconds, and the row becomes claimable again. Same behaviour as SQS, in about 20 lines of SQL. The same trap applies too: a job that runs longer than its 30-second lease gets claimed by a second worker while the first is still going. The next lesson deals with that.

Saving the order and adding the job

There's a trap between "save the order" and "add the job". They are two writes. What if only one happens?

• Save the order, then the server crashes before adding the job: the customer paid but never gets a receipt.
• Add the job first, then saving the order fails: the worker emails a receipt for an order that doesn't exist.

If the queue is a separate system like SQS or Redis, you can't wrap both writes in one transaction. This is called the dual write problem. A Postgres queue sidesteps it neatly:

BEGIN;
INSERT INTO orders (...) VALUES (...) RETURNING id;      -- id = 9812
INSERT INTO jobs (type, payload)
  VALUES ('send_receipt', '{"order_id": 9812}');
COMMIT;

Both rows commit together or neither does. When you later move to a separate broker, the standard fix is the outbox pattern, which uses this same trick: write an "outbox" row in the transaction, and have a separate process copy it to the broker. Module 15 covers it in full.

How many workers?

Use a simple rule (it's called Little's law): the number of jobs in progress at once equals the arrival rate times how long each job takes.

At 200,000 orders a day, Tiffin's dinner rush runs at about 11 orders a second. Each receipt takes about 1.5 seconds, mostly waiting on the email provider:

11 jobs/s × 1.5 s = about 17 jobs in progress at any moment

Give it headroom for a slow provider: if emails take 4 seconds each, you need 11 × 4 = 44 in progress, so run about 50 concurrent workers. If the provider gets slower still, the queue grows for a while and drains afterwards. Since these jobs mostly wait on the network, that doesn't mean 50 machines. One worker process handling 25 jobs concurrently, on two small machines, covers it.

If jobs arrive faster than workers finish them, the queue grows forever. So watch two numbers:

• Queue depth: how many jobs are waiting.
• Age of the oldest job: how long the unluckiest job has waited. This is the one customers feel. "Receipts are 12 minutes behind" is a clearer alert than "4,000 jobs queued".