Lab: Receipts and payouts

In this lab you'll take two jobs Tiffin does badly today and redesign them with everything from this module: a slow receipt email that makes checkout slow and sometimes breaks it, and a weekly restaurant payout that someone currently runs by hand. One of them sends emails. The other moves money. You'll see how the same tools apply to both, and where money demands more care.

As always, decide for yourself before you open each answer.

The brief

Two years after launch, Tiffin looks like this:

• 200,000 orders a day across 3,000 restaurants, average order ₹400.
• 60% of orders land between 7pm and 10pm: 120,000 in 10,800 seconds ≈ 11 orders a second, with spikes to about 13.
• Eight stateless app servers behind a load balancer, one Postgres primary with read replicas, Redis for caching.

Receipts. Checkout charges the card (800 ms typical), saves the order, then emails a receipt through an email provider before responding. The email call takes 1.5 seconds typically and 8 seconds at p99. About 0.5% of calls fail, and the provider had two 20-minute outages last quarter. During those outages, checkout returned errors and Tiffin lost an estimated 1,800 orders.

Payouts. Every Monday, each restaurant is paid for the previous week (Monday 00:00 to Sunday 23:59, India time): the sum of its delivered orders minus Tiffin's 20% commission, minus refunds. Today an engineer runs a script on her laptop on Monday morning. It calls the payment provider's payouts API once per restaurant. Last month her laptop's Wi-Fi dropped halfway through, she re-ran the script, and 340 restaurants were paid twice.

Some quick maths on payouts: 1,400,000 orders a week × ₹400 × 80% ≈ ₹45 crore a week, or about ₹1.5 lakh per restaurant. Finance has two requirements: never pay a restaurant twice for the same week, and never skip one.

Step 1: What stays in checkout?

The receipt fails both tests from "Now or later?": the customer doesn't need it to see "Order placed", and a failed email shouldn't cancel a paid order. Moving it out takes checkout from about 2.3 seconds typical (and 10+ at p99) down to about 0.8 seconds. More importantly, the next 20-minute email outage costs zero orders.

Step 2: Pick the queue

Saving the order and enqueuing the job in one transaction is the most valuable property here. With SQS, you'd be writing to two systems, and a crash in between loses a receipt or sends one for an order that doesn't exist. You'd need an outbox to fix that, which is more machinery than Tiffin needs today.

BEGIN;
INSERT INTO orders (...) VALUES (...) RETURNING id;             -- 9812
INSERT INTO jobs (type, payload) VALUES ('send_receipt', '{"order_id": 9812}');
COMMIT;

Step 3: Size the workers and set retries

Two details matter here. First, the email call needs a timeout of about 10 seconds, well below the 30-second visibility timeout, or a hung call lets a second worker start the same job. Second, check the email provider's own rate limit before draining at full speed. If it allows 100 emails a second, 33 is fine. If it allows 20, you'd configure the workers to stay under it, and the backlog takes longer to clear.

Step 4: Make the receipt job safe to run twice

Notice we didn't reach for anything clever. The duplicate window can't be closed when the email provider can't deduplicate, so you choose which way to be wrong and move on.

Step 5: Schedule the payouts

Now the money. First, how does the payout get triggered, exactly once a week, without anyone's laptop?

WITH claimed AS (
  INSERT INTO job_runs (job_name, scheduled_for)
  VALUES ('weekly-payouts', '2026-10-18 20:30:00+00')
  ON CONFLICT DO NOTHING
  RETURNING scheduled_for
)
INSERT INTO jobs (type, payload)
SELECT 'pay_restaurant',
       jsonb_build_object('restaurant_id', r.id, 'period_start', '2026-10-12')
FROM restaurants r, claimed;

Splitting the run into one job per restaurant matters. If the payment provider fails for restaurant 381, only that job retries. Under the old script, one failure stopped the whole run halfway, which is exactly how 340 restaurants got paid twice.

Step 6: Never pay twice

This is the step that matters most. A pay_restaurant job can run twice for every reason in "Jobs that can run twice", and each duplicate is about ₹1.5 lakh.

CREATE TABLE payouts (
  id            bigserial PRIMARY KEY,
  restaurant_id bigint      NOT NULL,
  period_start  date        NOT NULL,
  amount_paise  bigint      NOT NULL,
  status        text        NOT NULL DEFAULT 'pending',  -- pending, paid, failed
  provider_ref  text,
  created_at    timestamptz NOT NULL DEFAULT now(),
  UNIQUE (restaurant_id, period_start)
);

Here's the dangerous crash played out, for restaurant 381's payout row 4471:

%% caption: The retry finds the stored row and repeats the call with the same key, so the provider sends money once.
sequenceDiagram
  autonumber
  participant W as Payout job
  participant P as Postgres
  participant X as Payment provider
  W->>P: insert payout, on conflict skip
  P-->>W: row 4471, pending
  W->>X: pay, key payout-4471
  X-->>W: paid
  Note over W: crashes before saving
  Note over W,X: the job runs again
  W->>P: insert payout, on conflict skip
  P-->>W: row 4471 exists, same amount
  W->>X: pay, key payout-4471
  X-->>W: the original payout
  W->>P: save provider_ref, paid

There are three layers of protection here, and each covers a different gap:

Store money as integer paise, not floating-point rupees: 0.1 + 0.2 is not 0.3 in floating point, and finance will find every rounding error.

What if the payment provider didn't support idempotency keys? Then, before sending, the job would ask the provider "do you have a payout with my reference payout-4471?" and only send if not. That's weaker, because two jobs could ask at the same moment, so you'd also hold a row lock on the payout row (SELECT ... FOR UPDATE) while asking and sending. For money, idempotency keys are worth choosing a provider over.

Step 7: Know when it breaks

Reconciliation deserves its own mention. Every other protection is about your code behaving correctly. Reconciliation checks the result against an outside record, so it catches the bugs you didn't think of.

The final design

%% caption: Checkout only waits for what the customer needs. Receipts and payouts run behind it, retried, deduplicated and watched.
flowchart TD
  CA([Customer app]) -- checkout --> AS["App server<br/>validate, charge, save"]
  AS -- "order and job, one transaction" --> PG[("Postgres<br/>orders, jobs, payouts")]:::hot
  SC["Scheduler<br/>Sunday 20:30 UTC"] -- "job_runs claim, once a week" --> PG
  PG --> RW["Receipt workers<br/>50 concurrent, backoff"]
  RW --> EP[Email provider]
  PG --> PW["Payout workers<br/>one job per restaurant"]:::hot
  PW -- "idempotency key" --> PP[Payment provider]

Checkout is about three times faster and no longer fails when the email provider does. Receipts survive a 20-minute outage and catch up within about 20 minutes of recovery. Payouts run themselves every Monday as one job per restaurant, and a restaurant can't be paid twice for a week, however many times a job runs.

Build it on AWS

Tiffin runs in Mumbai (ap-south-1) on ECS with Fargate, an Application Load Balancer and RDS for PostgreSQL. The default AWS answer for background work is SQS plus workers, and later modules use it. This lab is the exception worth understanding: its own design keeps the job queue inside Postgres, and on AWS that's still right. The AWS-specific decisions are elsewhere: which service sends the email, and what triggers Monday's payouts. Prices are us-east-1 list prices as of late 2026, fine for comparing options; Mumbai's differ, so check the AWS pricing calculator before quoting them.

The architecture

%% caption: Receipts and payouts on AWS. The job queue stays in RDS; SES sends the email; EventBridge Scheduler starts Monday's run.
flowchart TD
  APP["App service<br/>ECS on Fargate"] -- "order and job, one transaction" --> RDS[("RDS for PostgreSQL<br/>orders, jobs, payouts")]:::hot
  EBS["EventBridge Scheduler<br/>Monday 02:00, Asia/Kolkata"] -- "RunTask" --> PT["Payout planner<br/>one-off ECS task"]
  PT --> RDS
  RDS --> W["Worker service<br/>ECS on Fargate, 2 tasks"]
  W --> SES["Amazon SES<br/>receipts"]:::hot
  W -- "through a NAT gateway" --> PP[Payment provider]

Argument 1: shouldn't the queue be SQS, now that we're on AWS?

SQS would cost almost nothing here: about 6 million jobs a month, each a send, a receive and a delete, is 18 million requests, roughly $7 at $0.40 per million. Cost isn't the question. The question is the one Step 2 answered: the order and its receipt job must be saved together or not at all. RDS can do that in one transaction. SQS can't join a Postgres transaction, so moving to SQS today means building the outbox (Module 15) to get back a guarantee Tiffin already has for free.

The jobs table's cost on RDS is real but small: about 13 inserts and 13 deletes a second, plus vacuum cleaning up the dead rows. Watch two numbers on the database: dead tuples on the jobs table, and how long the SKIP LOCKED claim query takes at dinner.

Recommendation: keep the jobs table. Move to SQS behind an outbox when job volume reaches hundreds a second, when the jobs table shows up in the primary's top queries, or when another team needs to consume the same events. That's the day Module 15's design starts paying for itself.

Argument 2: SES is the email provider, so learn its rules first

Receipts are a textbook SES workload: one message per order, sent through SendEmail once per recipient. At 200,000 orders a day, that's about 6 million emails a month, about $600 at $0.10 per thousand. But SES has rules that change the design:

• New accounts start in the sandbox: at most 200 messages in 24 hours, one per second, and only to verified addresses. Request production access well before launch; AWS says the first response can take up to 96 hours.
• The sending rate is a quota, and it must cover the backlog drain. Step 3 drains a backlog at 33 receipts a second. If the account's maximum send rate is lower, SES rejects the extra sends with a ThrottlingException ("Maximum sending rate exceeded") and drops those messages; it doesn't queue them for you. Request a rate comfortably above 33, and have the workers pace themselves to the account's limit (the GetAccount API returns it).
• Reputation is per account. If bounce or complaint rates climb too high, AWS can pause sending for the whole account. Receipts go to people who just ordered, so they rarely bounce or get reported as spam. Marketing email does both. Send marketing from a separate AWS account, so a bad promotional campaign can never stop order receipts.
• Bad addresses are handled for you. The account-level suppression list (on by default for accounts created after November 2019) stops SES delivering to addresses that hard-bounced or complained. Those sends still count against the daily quota, but no longer hurt your bounce rate.
• There is no idempotency key. Step 4's choice stands: receipt_sent_at in the order row, and a rare duplicate receipt is the accepted way to fail.

Argument 3: what starts Monday's payouts?

The laptop is replaced by EventBridge Scheduler: a cron schedule evaluated in the Asia/Kolkata time zone, firing at 02:00 every Monday with its flexible time window turned off, and starting a one-off ECS task (the RunTask target). That task makes the job_runs claim and enqueues 3,000 pay_restaurant jobs in the same statement, as in Step 5. It finishes in seconds.

Three details matter:

• Keep the job_runs claim. The scheduler retries a target it couldn't start, and a retry can't be told apart from a second run. The claim already makes a second run harmless, so nothing changes. The scheduler doesn't replace the claim; it replaces the laptop.
• Give the schedule a dead-letter queue, and alarm on it. Together with the 03:00 "no run yet" alert from Step 7, a missed trigger pages someone within an hour.
• Payouts leave through a NAT gateway with an Elastic IP. Payment providers often allowlist the IP addresses that may call their payouts API. That IP is now part of the payout system: changing it is a support ticket with the provider, not a config change. Keep the provider's API key in Secrets Manager, readable only by the worker's task role.

What it costs

That's about 10 cents per thousand orders, and more than 90% of it is SES. The payouts that move ₹45 crore a week cost almost nothing to run. Their cost is entirely in getting them right.

The same design on GCP and Azure

The design moves unchanged. On Google Cloud the email provider is a third party, which makes Step 4's "the provider can't deduplicate" just as true.