Jobs that can run twice

Every queue in this module has made the same promise: no job is ever lost. The other half of that promise, said more quietly, is that some jobs will run twice.

For a receipt email, a duplicate is a mild annoyance. For "add 50 loyalty points", it's free points. For "pay this restaurant ₹84,000", it's a very bad morning for the finance team. You can't make duplicates impossible, so you make them harmless. This lesson shows how.

Where duplicates come from

It's worth knowing exactly how a job ends up running twice, because it happens more often than people expect:

• The worker crashes after the work but before the acknowledgement. The email is sent, the process is killed by a deploy, the job comes back.
• The job is slower than the visibility timeout. The queue assumes the first worker died and hands the job to a second one while the first is still running.
• The acknowledgement gets lost. The worker says "done", the network drops it, the queue never hears it.
• The producer adds the job twice. The web server's call to SQS times out, so it retries. The first call actually succeeded.
• A human redrives a dead job that had half-worked before it failed.

%% caption: The most common way a job runs twice. The work succeeded; only the "done" was lost.
sequenceDiagram
  autonumber
  participant Q as Queue
  participant W1 as Worker 1
  participant E as Email provider
  participant W2 as Worker 2
  W1->>Q: receive job
  W1->>E: send the receipt
  E-->>W1: sent
  Note over W1: crashes before acknowledging
  Note over Q: visibility timeout ends
  W2->>Q: receive job
  W2->>E: send the receipt again
  E-->>W2: sent, a second time

Why can't the queue just guarantee "exactly once"?

Because the work and the acknowledgement happen in two different systems. The email goes to the email provider. The acknowledgement goes to the queue. There's no way to make "send email" and "tell the queue" happen as one atomic step. Whichever you do first, a crash in between leaves them disagreeing. Module 15 looks at what systems mean when they advertise "exactly once" and why it only holds inside a closed system.

So the job itself has to cope.

A job is idempotent if running it twice has the same effect as running it once.

Some operations are idempotent already

Look at what the job actually does. Some writes are safe to repeat as they are:

Setting a value is idempotent. Changing a value by an amount is not. So the first trick is often just to rewrite the job: instead of "add 50 points", recompute "points = sum of all rewards earned". Running a recompute twice gives the same answer.

When you can't rewrite it that way, you need the job to remember what it has already done.

Record what you did, with a unique constraint

Let's fix the loyalty points job. Instead of only incrementing a number, keep a ledger of every reward, and let the database refuse a duplicate:

CREATE TABLE points_ledger (
  id          bigserial PRIMARY KEY,
  customer_id bigint NOT NULL,
  order_id    bigint NOT NULL,
  reason      text   NOT NULL,
  points      int    NOT NULL,
  UNIQUE (order_id, reason)
);

The job inserts a ledger row and updates the balance only if the insert actually happened, all in one statement:

WITH added AS (
  INSERT INTO points_ledger (customer_id, order_id, reason, points)
  VALUES (:customer_id, :order_id, 'order_reward', 50)
  ON CONFLICT (order_id, reason) DO NOTHING
  RETURNING customer_id, points
)
UPDATE customers c
SET points = c.points + added.points
FROM added
WHERE c.id = added.customer_id;

Run it once: one ledger row, balance up by 50. Run it again: the insert hits the unique constraint, returns nothing, and the update touches no rows. Run it twice at the same time on two workers: the database lets exactly one insert win, and the other waits and then does nothing.

That last case matters. A common mistake is to write it as "check, then act":

const already = await db.query('SELECT 1 FROM points_ledger WHERE order_id = $1', [orderId])
if (!already.rows.length) {
  await db.query('UPDATE customers SET points = points + 50 ...')
  await db.query('INSERT INTO points_ledger ...')
}

Two workers can both run the SELECT, both see nothing, and both add points. The unique constraint is what turns "please don't do this twice" into "the database will not let this happen twice". Let the database enforce it, not your if.

A general version of this is a processed_jobs table with a unique job ID: insert the ID in the same transaction as the job's database writes, and skip the job if the ID is already there. It works for any job whose effects all live in your database.

When the work leaves your database

The ledger works because the effect (the points) and the record ("I did this") are in the same database, so one transaction covers both. Emails, SMS and payments aren't in your database. Now you have two options.

Option 1: the other side deduplicates for you

Many payment APIs accept an idempotency key: a string you send with the request. If they see the same key twice, they don't do the work again. They return the result of the first call. Stripe calls it the Idempotency-Key header, and most payment providers have something similar. Providers only remember a key for a limited time, often around a day, so check the window before counting on it for a job redriven days later.

The key must come from the job, not from the attempt. Something like payout-restaurant-381-2026-W42 is the same on every retry. A random UUID generated inside the job is different every time, so it protects nothing. Idempotency keys for your own API, where you are the one deduplicating, are covered in Module 12.

Option 2: pick which way you'd rather be wrong

Most email providers don't deduplicate. So you mark the order and send the email, and you have to choose the order:

There's no third option that's always right. The window is small: a crash between the two steps, or a timeout where the provider accepted the email but its answer never reached you. Either failure is rare. For receipts, a rare duplicate beats a missing receipt. So: check receipt_sent_at, skip if set, otherwise send and then set it.

Design jobs to be idempotent from the start

Some habits that make this easy:

• Put stable IDs in the job. "Reward order 9812" can be deduplicated. "Add 50 points to customer 77" can't, because a second legitimate reward looks identical.
• Prefer "make it so" over "do this". "Make order 9812's status delivered" is safe to repeat. "Advance order 9812 to the next status" is not.
• Store results you computed. If a job calculates an amount, save it with a unique key before acting on it, so the retry uses the same number.
• Test it. In your test suite, run every job twice and assert the result is the same as running it once. It's the cheapest bug-catcher you'll ever write.