Lab: Cache the menus

In this lab you'll design Tiffin's menu cache from start to finish, using everything in this module. The goal sounds simple: make menus fast at dinner time. The catch is in the second half of the brief: never show customers dishes that are gone.

As before, each step asks you to decide first. Write your answer down before opening the reasoning.

The brief

It's Friday, 8:10pm. Here's what the team sees:

• Tiffin has 3,000 restaurants in its city. Menus are available in English and Hindi.
• At the dinner peak, the app requests ~400 menus a second.
• Each menu takes ~25 ms of database time to build (categories, items, option groups, offers). A finished menu is about 30 KB of JSON.
• The database has 16 cores, and runs at 85% CPU at 8pm. Menu responses have a p99 of 900 ms (99% of responses are faster than that; the slowest 1% are slower).
• Restaurants make about 30,000 menu changes a day, mostly marking dishes sold out or back in stock during service.
• The app runs on one server today. Next quarter it will be several (Module 4), so the design should still work then.

Product's requirements:

1. Menus respond in under 50 ms on the server, p99, at dinner time.
2. A dish marked sold out disappears from customers' menus within 30 seconds.
3. Price changes show within 5 minutes. Customers are always charged the current price.
4. An order for a sold-out or removed dish is never accepted.
5. Marketing runs "Chef's Friday": a featured restaurant gets a push notification to 2 million customers. Expect up to 20,000 menu requests a second for that one restaurant for a few minutes.

Step 1: What exactly do you cache?

menu:v1:{restaurant_id}:{lang}      e.g. menu:v1:42:hi

Is one big value a problem when a single dish changes? Every change throws away the whole menu, and the next request rebuilds it. Let's check that's affordable. 30,000 changes a day against 3.6 million menu reads is about 120 reads per change. Most changes happen during service, so say one or two changes a second at peak, which means one or two extra 25 ms rebuilds a second. That's nothing.

If sold-out toggles were far more frequent (say, every few seconds per restaurant), a better split would be a large menu value that changes rarely plus a tiny "sold-out item ids" set per restaurant, merged on read. That's a good optimisation to know about. We don't need it yet.

Step 2: Where does each part live?

The reason for no phone max-age is the 30-second budget. If the phone may keep a menu for 30 seconds, and the server side can also be a few seconds stale, a sold-out dish could show for longer than 30 seconds. An ETag gives most of the bandwidth saving of a phone cache, since an unchanged menu costs a few hundred bytes instead of 30 KB, while always reflecting the server's view.

How much memory does Redis need?

3,000 restaurants x 2 languages x 30 KB = 180 MB

Compress the JSON (menus compress about 4 times) and it's under 50 MB. Even a small managed Redis instance with 1 GB of memory is far more than enough.

Step 3: The read path

async function getMenu(restaurantId, lang) {
  const key = `menu:v1:${restaurantId}:${lang}`

  let cached = null
  try {
    cached = await redis.get(key)   // client configured with a 30 ms timeout
  } catch (err) {
    metrics.increment('menu_cache.error')
  }
  if (cached !== null) return cached               // already JSON, send as is

  const menu = JSON.stringify(await loadMenuFromDb(restaurantId, lang))
  const ttl = 240 + Math.floor(Math.random() * 60)  // 4 to 5 minutes, with jitter
  redis.set(key, menu, { EX: ttl }).catch(() => {})
  return menu
}

The TTL isn't what keeps menus fresh: explicit deletes do that. It's a backstop that bounds the damage of a missed delete. So how long should it be? Requirement 3 says prices must show within 5 minutes, so the backstop can't be longer than that, or a single missed delete breaks the promise. That's why it's 4 to 5 minutes, not the 10 minutes we used earlier in the module. Is that affordable? 3,000 x 2 keys each reloading every 5 minutes is at most 20 rebuilds a second, about half a core of database time. Yes.

Step 4: The write path

async function updateMenuItem(itemId, changes) {
  const item = await db.updateMenuItem(itemId, changes)      // 1. the truth first
  const keys = LANGS.map(l => `menu:v1:${item.restaurantId}:${l}`)
  await redis.del(keys)                                      // 2. drop every copy
  setTimeout(() => redis.del(keys).catch(() => {}), 2000)    // 3. once more, later
  return item
}

Why the second delete two seconds later? Remember the rare race from the invalidation lesson: a reader who loaded the old menu just before the write can store it just after our delete. The delayed delete catches almost all of those. Anything it misses is bounded by the 5-minute TTL.

Here's that race, and the second delete catching it:

%% caption: A slow reader puts the old menu back just after the first delete. The delete 2 seconds later removes it.
sequenceDiagram
  autonumber
  participant X as Slow reader
  participant K as Kitchen tablet
  participant A as App server
  participant D as PostgreSQL
  participant R as Redis
  X->>D: load menu 42 (old)
  K->>A: paneer tikka sold out
  A->>D: update the menu item
  A->>R: delete both language keys
  X->>R: store the old menu
  Note over R: stale copy is back
  A->>R: delete again, 2 s later
  Note over R: the next read loads the truth

That covers what customers see. Requirement 4 is about what they can buy, and no cache design can guarantee it. So checkout never trusts the cached menu. Inside the order transaction, the server reads each item from the database:

SELECT id, name, price_paise, is_available
FROM menu_items
WHERE restaurant_id = :restaurant_id AND id = ANY(:item_ids)
FOR SHARE;

If any item is missing or unavailable, the order is refused with item_unavailable. Prices on the order come from this query, not from the app or the cache. FOR SHARE stops a concurrent "sold out" from slipping in between this check and the order being saved.

Step 5: Dinner-time stampedes

Tiffin deploys a new menu format on a Friday, and v1 becomes v2.

The jittered TTL from step 3 also matters here: menus rebuilt together during a cold start would otherwise all expire together 5 minutes later and stampede again.

Step 6: Chef's Friday

Now the hot key. Restaurant 42 is featured, and gets 20,000 menu requests a second.

20,000 requests/s x 30 KB = 600 MB/s ≈ 4.8 Gbit/s from one Redis key

Notice that the local cache also protects the database if Redis goes down: each server keeps serving popular menus for a few seconds while it falls back. And because marketing plans Chef's Friday ahead, you can ask them to warn the team, so the featured menu is pre-loaded before the notification goes out.

Check the numbers

Did we meet the brief? Estimate the hit ratio at peak:

• Misses from changes: 1 to 2 a second.
• Misses from TTL expiry: at most 6,000 keys every 5 minutes, ~20 a second, and in practice fewer, since unpopular menus aren't cached at all.
• Long tail of rarely-viewed menus: say another ~10 a second.

~30 misses out of 400 requests ≈ 92 to 95% hit ratio
Menu database time: 30 x 25 ms ≈ 0.75 cores, down from 10

The database drops from 85% CPU to around 25%, and a cache hit takes about a millisecond on the server. Requirement 1 is met with room to spare.

The final design

%% caption: Tiffin's menu caching. Reads stop early; writes go to the truth and clear every copy; checkout always reads PostgreSQL.
flowchart TD
  CA(["Customer app"]) -- "dish photos" --> CDN["CDN"]
  CA --> A["App server<br/>local menu cache, 5 s"]:::hot
  KT(["Kitchen tablet"]) -- "marks sold out" --> A
  A -- "menus, 5 min TTL" --> R[("Redis")]:::store
  A -- "misses, writes, checkout" --> PG[("PostgreSQL<br/>the truth")]:::store
  A -. "delete now, and again after 2 s" .-> R

Build it on AWS

Tiffin's version one already runs on ECS, RDS and CloudFront in Mumbai (see the first version lab). This module adds one managed service, a cache, and the interesting decisions are about which flavour, what it really costs, and the one limit that the Chef's Friday design quietly depends on. Prices are Mumbai (ap-south-1) on-demand list prices as of late 2026; check the AWS pricing calculator before quoting them.

The architecture

%% caption: The menu cache on AWS. A serverless Valkey cache between the tasks and RDS; menus stay off the CDN.
flowchart TD
  CA(["Customer app"]) --> ALB["Application Load Balancer"]
  KT(["Kitchen tablet"]) --> ALB
  CA -- "dish photos only" --> CF["CloudFront"]
  ALB --> T["ECS tasks<br/>in-process menu cache, 5 s"]:::hot
  T --> VK[("ElastiCache Serverless<br/>for Valkey")]:::store
  T --> PG[("RDS for PostgreSQL<br/>16 vCPU, Multi-AZ")]:::store

Argument 1: serverless or nodes, Valkey or Redis?

Valkey is the open-source fork of Redis that ElastiCache now leads with. It speaks the same protocol, so the app code doesn't change, and it's priced lower. ElastiCache sells it two ways: nodes you size yourself, or a serverless cache billed for the data it holds and the work it does. The serverless bill has two parts: data stored, with a 100 MB minimum for Valkey, and ECPUs, where a simple GET or SET costs 1 ECPU per kilobyte moved. Price Tiffin's menus both ways:

Serverless Valkey:
  storage:  50 MB compressed, billed at the 100 MB minimum
            0.1 GB x $0.054/GB-hour x 730        = ~$4
  reads:    3.6M menu reads a day x 8 KB = 8 ECPU each
            = 864M ECPU a month x $0.0015/million = ~$1.30
                                                   ~$6 a month
Nodes:      2 x cache.t4g.small (primary + replica in another zone)
            2 x $0.0328/hour x 730               = ~$48 a month
Serverless Redis OSS (for comparison):
            1 GB minimum x $0.081 x 730           = ~$59 a month before any reads

Serverless Valkey wins by a mile at this size, and it already spreads its data across zones, so there's no failover to plan. Compression matters twice: it shrinks the stored data and cuts the ECPUs per read from 30 to 8. Nodes become the better deal once a cache is large and busy around the clock, which is a later module's problem.

Argument 2: Chef's Friday meets a per-key ceiling

Serverless caches scale out by adding shards, but one key always lives on one slot, and AWS documents a ceiling of 30,000 ECPUs a second per slot (90,000 if clients read from replicas). Now redo the lab's hot-key maths in ECPUs:

20,000 requests/s x 30 KB uncompressed = 600,000 ECPU/s on one key
20,000 requests/s x 8 KB compressed    = 160,000 ECPU/s on one key
ceiling per slot                       =  30,000 ECPU/s

Compression alone isn't enough: more than four out of five requests would be throttled. The 5-second in-process cache is what makes it work. Each task asks Valkey for menu 42 at most once every 5 seconds, so even 20 tasks send about 4 requests a second, or 32 ECPUs. So on a serverless cache, the in-process layer isn't an optimisation you can drop when someone tidies up the code. Write that down next to the code.

A cold start, such as the v1 to v2 format switch, is safer than it looks. An empty serverless cache handles 30,000 ECPUs a second, and Tiffin's 400 menus a second at 8 KB is about 3,200. For a planned event far bigger than that, ElastiCache lets you set a minimum capacity in advance, and it takes effect within an hour.

Argument 3: why not cache menus in CloudFront too?

It's the obvious next idea, so price it. A sold-out dish must vanish within 30 seconds, so cached menus would need invalidating on every change:

Invalidations: 30,000 menu changes a day x 30 = 900,000 paths a month
               first 1,000 free, then $0.005 each            = ~$4,500 a month

The alternative is a short TTL, about 10 seconds, with no invalidation. That keeps the 30-second promise, but it still costs money: 108 million menu requests a month at $0.012 per 10,000, plus about 860 GB of data at $0.109 per GB, comes to roughly $225 a month. That's nearly forty times the cache, for customers who are all in one city a few milliseconds from Mumbai. The edge earns its keep for photos, which never change. Menus stay in Valkey.

Argument 4: what the cache buys on the database bill

The real return is on RDS. At 85% CPU, the next step would be doubling the database:

db.m7g.4xlarge, Multi-AZ (today):   $3.832/hour = ~$2,800 a month
db.m7g.8xlarge, Multi-AZ (next):    $7.664/hour = ~$5,600 a month

The cache takes database CPU from 85% to about 25%. With traffic growing about 10% a month (Module 4), it takes roughly 11 months to climb back to 70%. So a $6-a-month cache postpones a $2,800-a-month upgrade by most of a year. It's tempting to bank the savings by shrinking the database to 8 vCPU instead. At 10% growth a month, the smaller instance would be back at 70% in about four months, followed by another resize with a failover. Keep the headroom.

What it costs

That's well under one cent per thousand menu reads served from the cache.

The same design on GCP and Azure

Each provider prices and limits its cache differently. Look up the minimum bill and the per-key ceiling on whichever one you use, since those are the two numbers this lab turned on.