Limits across many servers

Everything so far has quietly assumed one server holding all the counters in its memory. Tiffin hasn't looked like that since Module 4. It runs eight stateless app servers behind a load balancer that sends each request to whichever server is free.

Put a token bucket in each server's memory and see what happens. A customer's requests are spread across all eight servers. Each server sees only an eighth of them, and each server has its own full bucket. A limit of "5 a second" quietly becomes "40 a second". Add servers for a busy Saturday and the limit grows again, without anyone touching it.

This is the same problem you met with sessions in Module 4: state that lives in one server's memory stops being true when there are many servers. The fix is similar too.

Three ways to share a limit

Split the limit

The cheapest idea: with 8 servers and a limit of 300 a minute, each server allows 37 a minute. No shared state at all.

It works only if the load balancer spreads each client's requests perfectly evenly, which it doesn't. A customer who happens to hit server 3 a lot gets blocked at 37 while servers 1 to 8 together saw 60. Worse, the number has to change every time you add or remove a server. Fine as a rough seatbelt (nginx's per-IP limit on each nginx instance works this way), not as a real per-user limit.

A shared counter in Redis

The usual answer is to keep the counters somewhere every server can reach. Tiffin already has Redis from Module 3, and Redis is a good fit: in memory, single-threaded (so each command runs alone), and fast.

%% caption: Every server asks the same Redis, so a user's budget is the same wherever their request lands.
flowchart TD
  LB[Load balancer] --> S1[Server 1] & S2[Server 2] & S8[Server 8]
  S1 & S2 & S8 --> R[("Redis<br/>one counter per user")]:::hot

Fixed window, and a nasty race

A fixed window in Redis looks like two commands:

const key = `rl:${userId}:${Math.floor(Date.now() / 60000)}`
const count = await redis.incr(key)       // creates the key at 1 if missing
if (count === 1) await redis.expire(key, 60)
if (count > 300) return reject()

INCR is atomic, so two servers can never both see the same count. Good. But look at the gap between INCR and EXPIRE. If the server crashes, is restarted by a deploy, or loses its connection right there, the key is created without an expiry. Here the key changes every minute so a stuck key just wastes memory. But many people write the simpler version with a fixed key like rl:42, and then a stuck key means that user stays at their count forever. Once they hit 300, they're blocked until someone deletes the key by hand.

The fix is to send both commands as one unit that runs completely or not at all:

MULTI
INCR rl:42:29341522
EXPIRE rl:42:29341522 60 NX
EXEC

MULTI/EXEC runs the commands together with nothing in between. NX on EXPIRE (Redis 7 and later) only sets the expiry if the key doesn't have one, so the window isn't pushed back on every request.

Token bucket, and the read-then-write race

A token bucket is harder. The logic is "read tokens and time, compute, write back". If two servers do that at the same moment for the same user:

%% caption: Both servers read before either writes, so both see the last token.
sequenceDiagram
  autonumber
  participant S1 as Server 1
  participant R as Redis
  participant S2 as Server 2
  S1->>R: read user 42's bucket
  R-->>S1: 1 token left
  S2->>R: read user 42's bucket
  R-->>S2: 1 token left
  S1->>R: allow, write 0
  S2->>R: allow, write 0

Two requests allowed with one token. Under a burst from a script, this happens all the time, and the limit leaks.

Any check-then-update on a shared counter must happen in one atomic step, or two servers will both see the same "last token" and both spend it.

Redis lets you run a small Lua script atomically: while it runs, no other command touches Redis. The whole token bucket fits in one:

-- KEYS[1] = bucket key, ARGV[1] = capacity, ARGV[2] = refill per second
local capacity = tonumber(ARGV[1])
local rate = tonumber(ARGV[2])
local t = redis.call('TIME')
local now = tonumber(t[1]) + tonumber(t[2]) / 1000000

local b = redis.call('HMGET', KEYS[1], 'tokens', 'last')
local tokens = tonumber(b[1]) or capacity
local last = tonumber(b[2]) or now

tokens = math.min(capacity, tokens + (now - last) * rate)
local allowed = 0
if tokens >= 1 then
  tokens = tokens - 1
  allowed = 1
end

redis.call('HSET', KEYS[1], 'tokens', tokens, 'last', now)
redis.call('EXPIRE', KEYS[1], math.ceil(capacity / rate) + 1)
return allowed

Two details worth noticing:

• The time comes from Redis (TIME), not from the app servers. Eight servers have eight slightly different clocks. One clock in one place means refills are computed consistently.
• The key expires once the bucket would be full again. A missing key is treated as a full bucket, so deleting idle buckets changes nothing and keeps memory down to active users only. With capacity 50 and 5 per second, that's 11 seconds.

The servers load the script once and call it by its hash, so each request is one round trip.

Is Redis fast enough?

Back-of-the-envelope for Tiffin at the dinner peak:

• About 3,600 requests a second reach the app. Say each does 1.5 limit checks on average (the general limit, plus a feature limit on some).
• That's about 5,400 Redis calls a second. A single Redis instance handles tens of thousands of small scripts a second. Plenty of room.
• In the same data centre, a round trip to Redis takes roughly 0.2 to 0.5 ms. A typical Tiffin request takes 30 to 80 ms. The limiter adds about 1%.
• Memory: say 200,000 active buckets at about 100 bytes each is 20 MB. Nothing.

So for Tiffin, one shared Redis is the right answer. Two things to keep it that way: give limits their own Redis instance, and keep a short timeout on the call. Sharing the cache's instance means a cache flush, or the cache's eviction policy throwing out keys under memory pressure, quietly resets everyone's counters.

When Redis is down

Now the uncomfortable question. Redis has a blip at 8:15pm. Every limit check fails. What should the server do?

• Fail open: allow the request. Customers keep ordering. For a few minutes there's no rate limiting.
• Fail closed: reject the request. Nothing gets abused, and nothing gets ordered either.

For most of the API, fail open. A rate limiter that takes Tiffin down because the limiter itself broke has done more damage than any scraper. But fail open safely: keep a simple in-memory limiter on each server (the "split the limit" approach) as a fallback, so a flood still meets some limit while Redis is away.

For a few endpoints, the answer flips. Sending OTPs costs money and invites abuse. If Redis is down, fall back to a strict local limit, or fail closed with a "try again in a minute" message. Losing a few logins for a minute is cheaper than an SMS bill.

And set a short timeout, say 5 to 10 ms, on the Redis call. A Redis that is slow, not dead, is worse than one that's down: every request waits for it.

Local counters with periodic sync

Sometimes even one Redis call per request is too much: a very high-volume API, or a single hot key like a huge partner sending 20,000 requests a second through one counter.

Then flip it around. Each server counts locally and, once a second, sends its total to Redis (INCRBY key delta) and reads back the global total. Between syncs, each server decides on its own, using the last global total plus its own local count.

The price is overshoot. Between syncs, each server can't see what the others allowed. If every server assumes the whole remaining budget is its own, 8 servers could each allow 1,000 in the same second against a limit of 1,000 a second. The usual fix is to let each server spend only its share of what's left (remaining ÷ 8) until the next sync. Then the overshoot is limited to roughly one sync interval of uneven traffic, and it corrects itself at the next sync. For a "protect our capacity" limit, that's often fine. For "5 OTP guesses", it's not, so those stay on exact Redis checks.

const key = `otp:${phone}`
const sent = Number(await redis.get(key) ?? 0)
if (sent >= 3) return reject()
await redis.set(key, sent + 1, 'EX', 600)
await sendSms(phone)

MULTI
INCR otp:+919800000000
EXPIRE otp:+919800000000 600 NX
EXEC