Open the Tiffin app and watch the network tab. In the first second, the home screen fires about 12 requests: your profile, saved addresses, the active order, banners, nearby restaurants, offers, the cart, and a few more. Then nothing for ten seconds while you scroll. Then a burst of 4 when you open a restaurant. Then quiet.
Real clients are bursty. A limit of "5 requests per second" would reject half of a perfectly normal app launch. A limit of "300 per minute" allows the launch, but it would also allow a script to send all 300 in the first second of the minute, which is exactly the kind of spike you're trying to prevent.
What you want is two rules at once: short bursts are fine, but the average over time must stay low. The token bucket says exactly that, with two numbers.
The idea
Picture a bucket that holds tokens.
- The bucket has a capacity, say 50 tokens. It starts full.
- Tokens drip in at a steady refill rate, say 5 per second. Once the bucket is full, extra tokens spill over and are lost.
- Each request takes one token out. If there's a token, the request is allowed. If the bucket is empty, the request is rejected.
A token bucket allows bursts up to its capacity, and limits the long-run average to its refill rate.
The two numbers do different jobs, and that's the whole point:
| Number | Controls | Tiffin value |
|---|---|---|
| Capacity | The biggest burst allowed at once | 50 |
| Refill rate | The steady average allowed over time | 5 per second (300 per minute) |
Walk through an app launch
A customer opens Tiffin. The bucket is full at 50.
- 12 requests arrive in the first second. 12 tokens are taken. 38 left, plus 5 refilled during that second.
- The customer scrolls for 10 seconds. 50 tokens refill, but the bucket caps at 50.
- They open a restaurant: 4 requests. 46 left.
The customer never got close to empty. Now a script:
- It sends 50 requests instantly. All allowed. Bucket empty.
- After that, it gets exactly 5 requests a second, no matter how fast it sends. Everything else is rejected.
The script got one burst, then got throttled to the average. The customer never noticed the limiter existed.
The maximum over any period
Over any span of T seconds, a client can make at most:
capacity + refill rate × T
With 50 and 5 per second: at most 55 in one second, 350 in one minute, 18,050 in one hour. Compare the fixed window's "up to 2x at the boundary". The token bucket has no boundary. Its worst case is a known, fixed burst on top of the average.
You don't need a timer
The obvious way to build this is a background timer that adds tokens to every bucket every second. With a million users, that's a million updates a second for buckets nobody is using. Don't.
Instead, refill lazily. Store two things per bucket: how many tokens it had, and when you last looked. When a request arrives, work out how many tokens would have dripped in since then, add them (up to capacity), and then decide.
// bucket = { tokens: number, last: number (ms) }
function allow(bucket, capacity, ratePerSec, now = Date.now()) {
const elapsed = (now - bucket.last) / 1000
bucket.tokens = Math.min(capacity, bucket.tokens + elapsed * ratePerSec)
bucket.last = now
if (bucket.tokens < 1) return false
bucket.tokens -= 1
return true
}
// a new user starts with a full bucket
const bucket = { tokens: 50, last: Date.now() }
Two numbers per user, a little arithmetic per request, and no work at all for users who aren't sending anything. Tokens are stored as a decimal, so a client that waits 0.3 seconds gets 1.5 tokens back, not zero.
This also gives you a free answer to "when can I try again?". If the bucket has 0.4 tokens and the rate is 5 per second, the client needs 0.6 more tokens, which takes 0.6 / 5 = 0.12 seconds. You'll send that back to the client in the next lesson.
Lazy refill depends on now. If two servers with different clocks update the same bucket, one can hand out tokens "from the future". Inside one process, use a clock that only moves forward. When the bucket is shared across machines, take the time from one place (we'll use Redis's own clock in the "Limits across many servers" lesson). Module 12 explains why machines disagree about the time.
Picking the two numbers
Use your logs again, but look at two different things.
For capacity, look at bursts. What's the most requests a real client sends in a few seconds? For Tiffin, an app launch is about 12, and a fast customer opening three restaurants in a row adds another 12. Capacity 50 covers that with room to spare.
For the refill rate, look at averages. What does a heavy real user send per minute, sustained? From the last lesson, the 99.9th percentile was 90 per minute. A refill of 5 per second (300 per minute) is over 3x that.
A useful check: could an abuser do real harm at exactly the limit? A scraper limited to 5 requests a second can still download about 18,000 menus an hour from one account. If that matters, the answer isn't a tighter bucket for everyone. It's a separate, stricter limit on the menu endpoint, or detecting the scraper in other ways. We'll come back to this in the lab.
Not all requests cost the same
A bucket can charge more than one token per request. Searching restaurants hits the database harder than reading a cached menu, so Tiffin could charge 5 tokens for a search and 1 for a menu. The client's budget now reflects the load it causes, not just its request count. Public APIs like GitHub's do something similar for expensive queries.
The leaky bucket
You'll also hear about the leaky bucket. Flip the picture: requests pour into a bucket (a queue) and leak out of a hole in the bottom at a fixed rate. If the bucket is full, new requests overflow and are rejected.
The difference that matters: a leaky bucket used as a queue delays requests to smooth them out, so what comes out the bottom is a perfectly even stream. A token bucket never delays. It decides yes or no instantly and lets bursts through.
| Token bucket | Leaky bucket (as a queue) | |
|---|---|---|
| Bursts | Allowed, up to capacity | Absorbed, then released at a steady pace |
| Extra latency | None | Requests wait in the queue |
| Output | Bursty, average capped | Smooth |
| Good for | Protecting your API from clients | Feeding something that needs a steady rate, like an SMS provider capped at 50 messages a second |
You've probably used the leaky bucket already without knowing it. Nginx's limit_req works this way:
limit_req_zone $binary_remote_addr zone=perip:10m rate=10r/s;
server {
location /api/ {
limit_req zone=perip burst=20 nodelay;
limit_req_status 429;
}
}
Here each IP may average 10 requests a second. burst=20 lets up to 20 extra requests queue up. Without nodelay, nginx would hold those 20 and release them at 10 a second, adding latency. With nodelay, it serves them immediately but still counts them, which behaves much like a token bucket with capacity 20. The 10 MB zone holds state for roughly 160,000 IP addresses. Nginx rejects with 503 by default, which is why limit_req_status 429 is there. We'll see why that code matters in the next lesson.
Token bucket vs windows
| Fixed window | Sliding window counter | Token bucket | |
|---|---|---|---|
| Stores per client | 1 counter | 2 counters | Tokens + timestamp |
| Allows bursts | The whole limit at once, 2x at a boundary | The whole limit at once, after a quiet minute | On purpose, up to capacity |
| Biggest burst in a couple of seconds (300 a minute, capacity 50) | 600 | About 300 | About 60 |
| Easy to explain to users | Very ("300 a minute") | Fairly | Harder ("50 at once, 5 a second") |
Tiffin uses the token bucket for its customer-facing API because the app is naturally bursty. It keeps the exact sliding log for small security limits like OTP attempts, where "exactly 5, no bursts" is what you want.
Size a bucket for the kitchen tablet
Say each Tiffin kitchen tablet polls for new orders every 5 seconds, as it did back in Module 2. When the tablet reconnects after a Wi-Fi drop, the app fetches the order list, the menu (to show availability toggles) and the restaurant's settings, about 6 requests at once. Staff sometimes tap "refresh" a few times in a row. Pick a capacity and refill rate. Then: how many requests could a broken tablet in a tight loop send in one hour?
Show one answer
Normal traffic is 1 request every 5 seconds, or 0.2 per second. A reconnect plus a few impatient taps is maybe 6 + 5 = 11 requests at once.
Pick capacity 20 (room for a reconnect plus taps) and refill 1 per second (5x normal polling). Normal tablets never come close to empty.
A broken tablet in a tight loop gets at most 20 + 1 × 3,600 = 3,620 requests an hour, instead of the hundreds of thousands a loop could otherwise send. If a bad release broke all 3,000 of today's tablets at once, that's at most about 3,000 requests a second in total: painful, but a known number you can absorb or shed while you roll back, rather than an unbounded flood.
What to remember
- A token bucket has two numbers: capacity (the biggest burst) and refill rate (the long-run average). Each request takes a token; no token, no request.
- Over any T seconds, a client gets at most capacity + rate × T. There's no window boundary to exploit.
- Refill lazily: store tokens and the last update time, and compute the refill when a request arrives. No timers.
- Pick capacity from real bursts and the refill rate from real heavy-user averages. Requests can cost more than one token.
- A leaky bucket queues requests and releases them at a steady rate. Nginx's
limit_reqis one. Use it when the thing behind it needs a smooth stream.