Rolling out a limit

You now know how to count requests, which algorithm to use, where the limit lives, and how to share it across servers. That's how a limit works. This lesson is about how a limit gets switched on, which is where most rate-limiting incidents actually happen. A limit with a perfect algorithm and the wrong number blocks hungry customers at 8pm, and nobody finds out until the support queue fills up.

The safe path has four stages: run it in shadow mode, read what shadow mode saw, enforce it for a slice of traffic, then watch and tune it for as long as it exists.

The limit Tiffin needs

When a customer types a delivery address, the app calls Tiffin's server, which calls a maps provider for autocomplete suggestions. Tiffin pays the provider ₹0.40 per call under its contract. Last month the bill was three times the forecast. The logs show why: a few hundred accounts call autocomplete thousands of times a day, almost certainly scripts harvesting address data on Tiffin's account.

The team wants a per-user limit on GET /addresses/autocomplete. But what number? A customer typing "14 Koramangala 5th Block" fires a request on most keystrokes, so honest use is bursty. Guess too low and real customers can't enter their address. That's worse than any maps bill.

Stage 1: shadow mode

In shadow mode, a limit does everything except block. It counts each request, decides whether this one would be rejected, logs that decision, and then lets the request through anyway.

In the app, it's one branch:

const rule = limits.get('autocomplete_per_user') // { limit, windowSec, mode, enforcePercent }
const d = await limiter.check(`ac:${user.id}`, rule)

if (!d.allowed) {
  log.info({ event: 'rate_limit', rule: 'autocomplete_per_user', user: user.id,
             count: d.count, mode: rule.mode })
  if (rule.mode === 'enforce' && inSlice(user.id, rule.enforcePercent)) {
    return res.status(429).set('Retry-After', d.retryAfterSec).json({ error: 'slow_down' })
  }
}
return handleAutocomplete(req, res)

Three details matter.

The counters must run for real. Shadow mode isn't "log the request". It goes through the same Redis counter and the same algorithm as enforcement will, so what you see is exactly what enforcement would have done.

The mode comes from config, not code. limits.get reads from a config store that every server reloads every few seconds. Moving a rule from shadow to enforce, or back, is a config change that takes effect in seconds, with no deploy. That's also your kill switch.

Shadow a lower number than you plan to enforce. Tiffin's candidate is about 100 calls per 10 minutes, so the shadow rule uses 30. Shadow mode only logs requests over the line, and you want to see the users just below your real limit too, not only the ones far above it.

At the edge, nginx has the same idea built in. limit_req_dry_run on counts as usual but lets everything through, and the $limit_req_status variable records what would have happened, so you can put it in the access log. Here's a real line from a test where 30 requests arrived at once against a limit with a burst of 20:

127.0.0.1 "GET /restaurants/42/menu HTTP/1.1" 200 REJECTED_DRY_RUN

The status is 200: the customer got their menu. The last field says the limit would have refused it. Cloud web application firewalls have the same switch; on AWS WAF, you set a rate-based rule's action to Count instead of Block.

Stage 2: read the shadow data

After a week, Tiffin has every would-block event. The question is the same for every limit: who would we have blocked, and were they real customers?

The best evidence of a real customer is money. So join the shadow log against orders, and split users into those with a paid order in the last 30 days and everyone else. Then look at each user's busiest 10 minutes of the week. With logs in S3, this is one Athena query:

WITH peaks AS (            -- each user's busiest 10 minutes of the week
  SELECT user_id, max(count) AS peak
  FROM shadow_events
  WHERE rule = 'autocomplete_per_user'
  GROUP BY user_id
)
SELECT
  o.user_id IS NOT NULL            AS has_paid_order,
  count(*)                         AS users,
  approx_percentile(p.peak, 0.99)  AS p99,
  approx_percentile(p.peak, 0.999) AS p999,
  max(p.peak)                      AS max
FROM peaks p
LEFT JOIN paid_last_30_days o ON o.user_id = p.user_id
GROUP BY 1;

What came back:

Two very different groups. Real customers who type a lot top out around 84 calls in their busiest 10 minutes; one customer edited the same address several times on a bad network. The no-order group is mostly a few hundred accounts sending thousands. A limit of 150 per 10 minutes sits almost twice above the busiest real customer and cuts the scripts down by 90% or more.

Before settling on it, read a sample of the rows between the two groups by hand. Here, the users with no orders who peaked between 85 and 150 were mostly new customers who signed up and typed their address several times before ordering. They're real customers too, just without history. A limit of 150 leaves them alone.

What shadow data can't tell you

Shadow mode shows what would have happened if the limit had existed. Once it does exist, behaviour changes, in both directions:

• Clients that get blocked may retry. Tiffin's own app, if it retries a 429 immediately, will send more requests once enforcement starts, not fewer. Check that the app honours Retry-After before you enforce.
• Scripts adapt. Someone harvesting addresses will spread the work over more accounts. The per-user limit still caps each account, but expect the would-block count to fall faster than the bill does, and plan a second limit, such as per device or per IP, if it does.
• Product changes move "normal". If next month's app update adds autocomplete to the search box, honest usage doubles overnight.

Stage 3: enforce for a slice

Even with good data, don't flip the whole rule at once. Enforce it for a small slice of users, chosen by hashing the user ID, so the same user is always in or out:

const inSlice = (id, pct) => (hash(id) % 100) < pct

Tiffin's rollout:

Every step happens on a weekday afternoon, when traffic is low and the whole team is around. No step happens on a Friday. The weekend at 50% is deliberate: it's the busiest traffic of the week, seen by half the users, before the other half get it.

The most useful number to watch is 429s sent to users with a paid order in the last 30 days, as a share of their requests. It should be zero or near it. Tiffin alerts if it goes above 0.1% for 15 minutes; when that fires, the response is to move the rule back to shadow first and investigate second.

Stage 4: keep tuning

A limit isn't done when it reaches 100%. It's a number tied to how people use the app, and that keeps changing.

• Keep logging blocks with the same fields as shadow mode, forever. The same query then shows who is being blocked today.
• Re-run the analysis every quarter, and before any launch that changes how often clients call an endpoint.
• Check the headroom. Tiffin grows about 10% a month. A per-user limit doesn't grow with it, but per-IP limits do get closer to real traffic as more customers share each carrier's addresses.
• Keep the kill switch working. Test it once a quarter: flip one rule to shadow and back.