Labs

System lab · medium · 33 min

Rate limiter: Shared API quotas

A shared quota check in front of every endpoint, fast enough to be invisible.

A rate limiter answers one question, over and over: should this request be allowed? It sits in front of every API call, so it has two jobs that pull against each other. It has to be correct enough that a client can't sneak past its limit, and fast enough that nobody notices it's there.

This lab builds a shared limiter for a company's whole API from a blank page. Module 8 introduced the algorithms on a single server. Here you'll make them work across hundreds of servers and more than one region.

The brief

Your company runs a public API, like a payments or messaging platform. Thousands of customers call it with API keys. Some send a few requests a day. A few send tens of thousands a second. Every so often, one of them has a bug and sends a million.

You're building the service that every API server asks before handling a request.

Who uses it?

  • API servers and the gateway call the limiter on every request.
  • Product teams define rules: "100 requests a second per API key", "5 login attempts a minute per IP".
  • Customers never call it directly, but they feel it: either a 429 Too Many Requests or nothing at all.

In scope: per-key and per-IP limits, multiple rules per request, a shared count across all API servers, response headers that tell clients their budget, and running in two or more regions.

Out of scope: DDoS protection at the network layer (that's a job for the CDN or a scrubbing service, long before traffic reaches you), billing and quotas measured in months, and bot detection.

Requirements

Functional

  • Given a request's attributes (API key, IP, endpoint), decide allow or deny.
  • Support several rules per request, for example per-key per second and per-key per day.
  • Rules are configurable without a deploy.
  • Denied requests get 429 with a Retry-After header. All responses carry remaining-budget headers.

Non-functional

  • Low latency: the check adds under 1 ms at p99 to each request, within a region.
  • Accurate enough: a client can exceed its limit by a few percent briefly, but not by 2×.
  • Highly available: if the limiter is unhealthy, the API must keep working.
  • Scales with the API: handles every request the API handles, at peak.

Look at the third one carefully. The limiter protects the API, so it must never become the reason the API is down.