Lab: Protect the order API

In this lab you'll design Tiffin's rate limits from scratch. There's no single right answer. What matters is that every number traces back to real traffic, and that no hungry customer at 8pm ever sees "too many requests". Decide each step before opening the reasoning.

The brief

Tiffin is two years old and doing well in its city. The setup:

• About 3,600 requests a second from real customers at the dinner peak, on 8 stateless app servers behind the managed load balancer from Module 4, with nginx as a reverse proxy in front of each app server.
• Redis is already running for caching (Module 3).
• Customers log in with a phone number and a 4-digit OTP sent by SMS. Each SMS costs ₹0.25.
• Today there are no rate limits at all.

Last month went badly. Four incidents landed on your desk:

1. The scraper. Menu requests at 2,500 a second for two hours, logged out, from 40 IP addresses that all belong to a cloud provider. Dinner-time latency doubled.
2. The coupon guesser. One account tried 30,000 coupon codes in a night and found three private influencer codes. They were shared on a deals forum and cost ₹1.2 lakh before anyone noticed.
3. SMS pumping. 200,000 OTPs sent in one night to numbers the attacker profits from. Bill: ₹50,000.
4. Our own bug. App version 5.3 retried POST /orders in a tight loop whenever the request timed out. Some customers got the same order three or four times.

The head of engineering's brief: "Stop all four. Don't block a single real customer at dinner. And cap how much an SMS attack can ever cost us."

Step 1: Measure normal

You can't set a limit without knowing what normal looks like. You pull a week of logs. Here's what real customers do, counted per user (or per IP for logged-out traffic):

Step 2: Pick the key for each limit

No single key works everywhere. IP is weak (carrier NAT) but it's all you have before login. User ID is best once you have it. Phone number protects the target. A global total protects your wallet.

Step 3: Set the numbers

Check each one against its incident.

Scraper. Each cloud IP now gets 60 menus a minute instead of 3,750. Across 40 IPs that's 2,400 a minute, or 40 a second, down from 2,500 a second. A determined scraper will rent more IPs or switch to mobile proxies, so this is a speed bump, not a wall. The real protection for capacity is making scraping cheap: menus are cached (Module 3), so even scraped requests barely touch the database.

Coupon guesser. 10 wrong codes an hour means 30,000 guesses would take 125 days on one account. Each new account needs a new phone number and an OTP, which is itself limited.

SMS pumping. The hard stop is 10,000 OTPs an hour. Even if nobody answers the page for the whole 8-hour night of last month's attack, that's at most 80,000 × ₹0.25 = ₹20,000, down from ₹50,000. In practice the alarm at 6,000 an hour pages someone within minutes, and the attack is blocked far below the hard stop. The hard stop has a price: once it trips, real customers can't get an OTP either until the hour rolls over. That's why it sits well above the normal peak, and the alarm fires long before it. (Rate limits aren't the only defence here: blocking premium-rate number ranges and countries Tiffin doesn't serve helps just as much.)

Our own bug. 10 orders in 10 minutes still lets a broken app place several duplicates. A rate limit can't tell "the same order again" from "a new order". The real fix is an idempotency key, so a retried request with the same key returns the first order instead of creating a new one. That's Module 12. The rate limit is a backstop that stops a runaway loop from placing hundreds.

Step 4: Choose the algorithm and where each limit runs

The reasoning:

• The IP seatbelt goes at nginx because it needs no business knowledge and it's cheapest to reject a flood before it reaches an app server. nginx can map cloud IP ranges to the strict zone. Two details come from nginx sitting behind the load balancer. Every request arrives from the load balancer's address, so nginx must take the client's IP from the X-Forwarded-For header with its real_ip module, trusting that header only from the load balancer's (and CDN's) addresses; otherwise every phone shares one bucket, or a client can forge its IP. And each nginx instance counts on its own while the load balancer spreads one IP's requests over all eight, so set each instance to an eighth of the fleet-wide number: 750 a minute for mobile and home IPs, about 8 for cloud ranges. That's rough, which is fine for a seatbelt.
• The per-user bucket goes in middleware because only the app knows who's logged in. It runs before any handler, so a rejected request costs one Redis call and nothing else.
• Feature limits live inside the features because only the coupon code knows a coupon was wrong, and only the OTP code knows which phone number is being targeted.

Here's one OTP request passing every limit on its way to the SMS provider:

%% caption: A logged-out OTP request. Cheap checks first, the money-saving global cap last.
sequenceDiagram
  autonumber
  participant P as Phone
  participant N as nginx
  participant A as App server
  participant R as Redis
  participant S as SMS provider
  P->>N: send OTP to my number
  Note over N: IP seatbelt, counted locally
  N->>A: forward
  A->>R: per-phone and per-IP logs
  R-->>A: under both limits
  A->>R: global hourly count
  R-->>A: under the cap
  A->>S: send the SMS
  A-->>P: code sent

The extra Redis load is roughly 5,000 calls a second at about 0.5 ms each. One Redis handles that easily. Limits get their own Redis instance so a cache flush or cache eviction can't reset everyone's counters.

Step 5: What happens over the limit, and when Redis breaks

The pattern: fail open where the limit protects capacity and blocking would cost real orders. Fail closed where the limit protects money or security and there's a cheap way for the customer to carry on.

App version 5.4 is shipped before the limits go live, so Tiffin's own app honours Retry-After and backs off instead of hammering a 429.

Step 6: Roll it out

The final design

%% caption: Tiffin's rate limits. Loose and cheap at the edge, per user in the app, strict inside the features that cost money or invite attacks.
flowchart TD
  P([Phone]) --> CDN["CDN<br/>cached menus"]
  CDN --> LB["Load balancer<br/>managed, layer 7"]
  LB --> NG["nginx on each app server<br/>IP seatbelt, strict for cloud IPs"]:::hot
  NG --> AS["App servers<br/>per-user token bucket"]
  AS --> F["Feature code<br/>OTP, coupon, order"]
  AS --> R[("Redis, limits only<br/>shared counters, Lua scripts")]:::hot
  F -- "strict feature limits" --> R
  F -- "OTP: global hourly cap first" --> SMS[SMS provider]

Read it back against the brief. The scraper is slowed at nginx and mostly hits cache. The coupon guesser gets 10 tries an hour. An SMS attack has a known worst-case bill and pages someone early. A buggy app stops at 10 orders. And every limit sits at or above the busiest real customer and was proven in shadow mode first.

Build it on AWS

Tiffin runs in Mumbai (ap-south-1): phones talk to an Application Load Balancer, the app runs on ECS with Fargate, menus are served from ElastiCache for Valkey and an in-process cache (Module 3), and CloudFront serves only dish photos. On AWS the lab's nginx disappears, and its job moves to AWS WAF. The business limits stay in the app, exactly as designed. Prices are us-east-1 list prices as of late 2026, fine for comparing options; Mumbai's differ, so check the AWS pricing calculator before quoting them.

One departure from the final design above: it draws a CDN with cached menus in front of the API. On AWS, Module 3 already decided not to cache menus in CloudFront, and the scraper still mostly hits a cache, just Valkey's instead of the edge's. Routing every API request through CloudFront only for this would add about $3,100 a month in request charges at the US rate (the 3.1 billion requests a month worked out under Argument 1, at $0.01 per 10,000), with India's edge rate higher still.

The architecture

%% caption: Rate limits on AWS. WAF on the load balancer does the IP seatbelts; the app does per-user and feature limits against its own Valkey.
flowchart TD
  P([Phone]) --> ALB[Application Load Balancer]
  WAF["AWS WAF<br/>IP seatbelts, cloud IPs strict"]:::hot -.- ALB
  ALB --> APP["App service, ECS on Fargate<br/>per-user bucket, feature limits"]
  APP --> VK[("ElastiCache Serverless for Valkey<br/>limits only")]:::hot
  APP -- "OTP, after the global cap" --> SMS["AWS End User Messaging SMS<br/>monthly spend quota"]

Argument 1: where the IP seatbelt runs, and how it spots cloud IPs

WAF can attach to CloudFront or to the ALB. Two facts decide it. AWS's IP-reputation rule groups look at the request's source IP, and Tiffin's API traffic doesn't pass through CloudFront: phones connect straight to the ALB. So the seatbelt goes on the ALB, where the source is the phone. (If the API ever moves behind CloudFront, move the web ACL with it: an ALB behind CloudFront only sees CloudFront's addresses.)

The lab's rules translate directly:

• Mobile and home IPs: a rate-based rule, 6,000 requests per 60-second window per IP. WAF's minimum window is 60 seconds, which is fine for a seatbelt.
• Cloud IPs: add AWS's managed AWSManagedRulesAnonymousIpList rule group, but set its rules to Count so they only label requests. Then a second rate-based rule, limited to requests carrying the HostingProviderIPList label, allows 60 a minute.
• The gap: AWS documents that its hosting-provider list doesn't include AWS's own IP addresses, so plan as if a scraper renting EC2 instances slips past it. AWS publishes its IP ranges, so keep a WAF IP set built from that list, refreshed on a schedule, and give it the same strict limit.

WAF counts across the whole load balancer, not per server as nginx did, so the limit is fleet-wide. Its counts are approximate, though, so a client can get somewhat past the number before being limited. That's fine for a seatbelt, and it's exactly why the coupon, OTP and order limits stay in the app, where they're exact.

The cost: WAF charges $0.60 per million requests it inspects. Tiffin peaks at 3,600 requests a second; assume about a third of that, 1,200 a second, averaged over the day:

1,200/s x 2.6 million s = 3.1 billion requests a month x $0.60 per million = ~$1,870
plus the web ACL ($5) and its rules and rule group ($1 each)            = ~$1,880

That's the biggest line in this section. It buys the scraper fix, and it's the place to add bot rules later without touching the app. (The Bitly lab also attaches its WAF to the ALBs, but for a different reason: there CloudFront answers most requests, the bots land on cache misses, and the ALB has to read the client's address from a header CloudFront adds.)

Argument 2: a cache just for counters

The lab gives limits their own Redis, so a cache flush can't reset everyone's counters. On ElastiCache there are two ways to buy it.

Node-based:  one cache.r7g.large primary + one replica in another zone
             2 x $0.1752 x 730 h                                ~ $256 a month
Serverless:  ~5,000 calls/s at peak, ~1,700/s on average = 4.4 billion calls a month
             at ~2 ECPU each (Lua scripts cost more than a GET) x $0.0023 per million  ~ $20
             + storage, billed at its 100 MB minimum (the counters are tiny)            ~  $6
                                                                ~ $26 a month

At this size, ElastiCache Serverless wins by ten times, and it runs across Availability Zones without a replica to manage. Two things to check before relying on it. First, it adds a proxy hop, so measure the limiter's p99 against the lab's 10 ms timeout before going live. Second, a Lua script that touches several keys needs them in the same hash slot. Name keys with a hash tag, like {user:881}:bucket and {user:881}:orders, so one user's keys always sit together.

The node-based cluster becomes the cheaper choice once traffic is high enough that a busy node costs less than the ECPUs. The rate limiter lab reaches that point at a far larger scale.

Argument 3: the SMS bill's last line of defence

OTPs go through AWS End User Messaging SMS. Two things here matter before the first customer logs in.

The spending quota. Every account has a monthly SMS spending quota. New accounts start at $1.00 a month, so production needs an increase through Service Quotas. When sending would push the month past the quota, AWS stops sending, within minutes, so a burst can overshoot a little. That's a useful backstop for the lab's global cap, with two catches. It's monthly, so it can't replace the hourly cap that pages someone at 6,000 an hour. And when it trips, it stops all SMS, real customers' OTPs included, until the month ends or the quota is raised. Set it well above a normal month, say double, and alarm on your own OTP counts long before you reach it.

Registration. Sending to Indian numbers on local routes requires registering your sender details and message templates before you can send. It takes time, so start the paperwork weeks before launch, not the week of.

The lab's ₹0.25 per SMS is the provider price this design was costed on. Check AWS's current price for Indian destinations before comparing.

What it costs

That's about 60 cents per million requests, almost all of it WAF. The limits themselves cost about as much as a team lunch. The protection at the edge is what you're paying for.

The same design on GCP and Azure

The design is the same everywhere: rough limits at the edge, exact limits in the app, and a hard cap on anything that costs money per message.