Lab: Survive the dinner rush

This lab puts the whole module together. You'll get a brief with real numbers, and for each step you decide first, then compare with one way of reasoning it through. There's rarely a single right answer, but there are plenty of wrong ones, and they tend to show up at 8pm.

The brief

Marketing has planned Tiffin's biggest promotion yet: 50% off every order this Saturday, from 7pm to 10pm. They expect dinner traffic to roughly triple. It's Tuesday. You have four days.

Here's where Tiffin stands today:

The goal: Saturday's dinner goes by with no outage, no lost orders, and p99 latency under 800 ms.

Step 1: How many servers?

10,800 / 980 ≈ 11  ->  11 servers for the load
+ 1 spare (N+1)     ->  12 servers

The maximum-size setting is the kind of thing nobody remembers until it bites. Autoscaling limits, cloud account quotas on how many machines you can run, and IP address ranges in your network can all quietly cap you below what you need. Check all of them on Tuesday, not Saturday.

Step 2: When do the servers arrive?

Managed load balancers scale themselves too, but they also take time to grow. For a jump of 3x that's usually fine. If you ever expect something like 10x in a minute, ask your cloud provider about pre-warming the load balancer.

Step 3: Find the state

You can't just add servers until you know they'll behave. The team searches the code and finds four places that keep state. Check each one.

UPDATE promos
SET used = used + 1
WHERE code = 'SAT50' AND used < 20000
RETURNING used;

Here's one discounted checkout on Saturday, now that the counter lives in the database. It doesn't matter which server the load balancer picks:

%% caption: Any of the 12 servers can take the order, because the promo counter is claimed in the shared database.
sequenceDiagram
  autonumber
  participant C as Customer app
  participant L as Load balancer
  participant S as Any app server
  participant D as Database
  C->>L: POST /orders with SAT50
  L->>S: least outstanding requests
  S->>D: claim a slot if used < 20000
  alt a row comes back
    D-->>S: slot claimed
    S-->>C: order placed at 50% off
  else no row
    D-->>S: nothing returned
    S-->>C: the promotion is used up
  end

Step 4: Look behind the servers

Adding app servers doesn't add database capacity. It moves pressure onto the database. Check it before Saturday does.

12 servers x 8 processes x 10 connections = 960 connections
max_connections                           = 300

Step 5: Configure the front door

Step 6: Deploys and rehearsal

The final design

%% caption: Tiffin for Saturday. Stateless app servers scaled ahead of time, state in shared stores, and a database that has just enough room for one night.
flowchart TD
  CA(["Customer app"]) --> LB["Load balancer<br/>least outstanding, GET-only retries"]:::hot
  LB --> fleet
  subgraph fleet ["App servers: 12 at dinner, autoscale to 14"]
    direction LR
    S1["App server"] ~~~ S2["App server"] ~~~ S3["App server"]
  end
  fleet --> R[("Redis<br/>sessions, menus")]:::store
  fleet --> DB[("Database<br/>orders, promo counter, 32 cores")]:::store
  fleet --> OS[("Object storage<br/>receipts, photos")]:::store

And the plan in one list:

1. Raise the autoscaling maximum to 14, check account quotas, and schedule 12 servers from 6:30pm to 10:30pm.
2. Move the promo counter into the database with an atomic update. Move receipts to object storage.
3. Cut database pool sizes so total connections stay under the limit even at the autoscaling maximum, take the heaviest reads off the database, and scale it up on Thursday morning.
4. Health checks test only the server itself. The load balancer retries only safe requests.
5. Ship the promotion behind a flag midweek, freeze deploys on Saturday, and rehearse at 130% of forecast with a server killed mid-test.

Notice where the real risk ended up: not in the app servers, which you can now add almost freely, but in the one database every server shares. Making that database scale is the whole of the next module.

Build it on AWS

The plan above is cloud-neutral. On AWS, Tiffin's app servers are tasks in an ECS service on Fargate, behind an Application Load Balancer, with RDS for PostgreSQL behind them (see the first version lab). Most of the plan maps straight across. What's interesting is where AWS's own limits, defaults and missing settings change the checklist. Prices are Mumbai (ap-south-1) on-demand list prices as of late 2026; check the AWS pricing calculator before quoting them.

The architecture

%% caption: Saturday on AWS. Capacity scheduled ahead, the database scaled up on Thursday, state in shared services.
flowchart TD
  CA(["Customer app"]) --> ALB["Application Load Balancer<br/>least outstanding requests"]:::hot
  SCH["Scheduled scaling<br/>12 tasks from 18:30 IST"] -.-> SVC
  ALB --> SVC["ECS service on Fargate<br/>8 vCPU tasks, max 14"]
  SVC --> VK[("ElastiCache<br/>sessions, menus")]:::store
  SVC --> PG[("RDS for PostgreSQL<br/>32 vCPU for the weekend")]:::store
  SVC --> S3[("S3<br/>receipts")]:::store

Argument 1: the quota you'll hit before the traffic

The lab warned that limits, not servers, cap you. On AWS, the first one is Fargate's account quota: by default, 6 on-demand vCPUs per Region. Tiffin's account has been raised since then, or five 8-vCPU tasks wouldn't run at all, but Saturday needs more than it has ever used:

14 tasks x 8 vCPU = 112 vCPUs at the autoscaling maximum

Fargate raises quotas automatically as usage grows, but only gradually. Request the increase on Tuesday, in the Service Quotas console, and check the answer before Thursday's rehearsal. The other Fargate limits are comfortable: a service can start 500 tasks a minute in Mumbai, and Tiffin needs seven.

Then the clock. A scheduled action is just "set the minimum to 12 at this time", and cloud schedules are often written in UTC. 18:30 in India is 13:00 UTC, so set the time zone on the schedule explicitly and check the first action fires when you expect. Also time how long a new task takes to become healthy. Most of it is pulling the container image, so a big image means a slow scale-out. ECS can lazy-load large images, which helps for images over a few hundred megabytes.

The load balancer scales itself, and a 3x rise over an hour is usually fine. For a sharper jump, an ALB can reserve capacity in advance: the minimum reservation is 100 load balancer capacity units, which in Mumbai costs about $0.80 an hour. That's cheap insurance for a four-hour window.

Argument 2: health checks and retries, the AWS way

Two of the lab's settings don't map straight onto AWS, and both are worth knowing.

Retries. The lab's load balancer retried failed requests once. An ALB has no such setting: none of its target group options retries a request against another target. So "retry only GETs" is enforced where retries actually happen, in the app's HTTP clients and the mobile app. The bug to hunt for is a client library with automatic retries switched on for every method, including POST /orders.

Health checks. The lab removed the SELECT 1 from the health check so a slow database couldn't mark every server unhealthy. On ECS that matters even more, but for a different reason than you might expect. If every target fails its health check, the ALB actually fails open and keeps sending traffic to all of them. ECS, however, replaces tasks that fail their load balancer health checks. A database-dependent health check on a slow Saturday therefore makes ECS stop and restart your app tasks at 8pm, the worst possible moment. Keep /healthz about the process only, and give new tasks a health check grace period, which defaults to zero, so a slow-starting task isn't killed before it's ready.

One more default: when a task is removed, the ALB keeps it draining for 300 seconds. At 22:30, when the schedule scales back down, that's fine. For deploys, set it a little above your slowest request (the payment call), not five minutes.

Argument 3: the database plan in RDS terms

Connections. The lab's database refused connections above max_connections = 300. On RDS for PostgreSQL the default comes from the instance's memory, capped at 5,000, and a db.m7g.4xlarge with 64 GiB gets the cap. So on default RDS settings, Saturday's 960 connections wouldn't be refused. Don't relax: each connection is a separate PostgreSQL process with its own memory, and a thousand of them, mostly idle, still eat RAM the database would rather use for caching. Cut the pools to 2 per process as the lab says. RDS Proxy does the pooling as a managed service, at $0.017 per database vCPU-hour in Mumbai, about $200 a month for this database. It's the next module's topic.

Scaling up. Moving from 16 to 32 vCPUs is an instance class change, and on a Multi-AZ database it triggers a failover: typically 60 to 120 seconds of connection errors while the database's DNS name moves to the upgraded standby. That's why it happens on Thursday at 4am, with the app's database driver set to reconnect. Then decide whether to keep it:

db.m7g.4xlarge, Multi-AZ:  $3.832/hour
db.m7g.8xlarge, Multi-AZ:  $7.664/hour   (+$3.832/hour)
Thursday 4am to Monday 4am = 96 hours    ->  ~$370 for the weekend
Keeping it for good                       ->  ~$2,800 more a month

Scale back down on Monday at 4am, with one more brief failover, unless the weekend's numbers say Friday dinners now need it too.

What it costs

That's 2 cents per discounted order. At an average order of ₹400, the discounts themselves come to about ₹40 lakh, so the extra infrastructure is a rounding error. The expensive mistakes are the free ones: a quota nobody raised, or a health check that restarts the fleet.

The same design on GCP and Azure

The checklist carries over: every platform has account quotas, minimum and maximum instance settings, and its own health check behaviour. Find each one on Tuesday.