Why cache?

Tiffin has grown. Two years in, it serves 3,000 restaurants and about 200,000 orders a day in its home city. And every evening around 8pm, the database runs hot.

When the team looks at what it's doing, the answer is a little embarrassing. Most of the database's time goes on answering the same question over and over: "what's on restaurant 42's menu?". Thousands of customers ask it every hour. The answer changes maybe ten times a day.

That's the situation caching was invented for. In this module we'll learn to cache well: where caches live, how to fill them, and the hard part, how to keep them from showing stale data. This first lesson is about why, and when caching is the wrong answer.

Most reads ask the same question

Let's look at Tiffin's menu traffic at dinner time.

• 600,000 customer sessions a day, each looking at about 6 menus: 3.6 million menu reads a day.
• 60% of those fall in the 7pm to 10pm rush, and the busiest half-hour runs at about twice the rush average. That's ~400 menu reads a second at the peak.
• A full menu isn't one simple lookup. It joins categories, items, option groups ("spice level", "add raita") and offers. Call it 25 ms of database time per menu.

400 reads/s x 25 ms = 10 seconds of database work every second

That's 10 CPU cores, out of the database's 16, spent on menus alone.

Now the other side. How often do menus change?

• 3,000 restaurants, each editing their menu or marking a dish sold out maybe 10 times a day: 30,000 changes a day.
• 3.6 million reads / 30,000 changes = about 120 reads for every change.

And here's the important part: the answer is the same for everyone. Restaurant 42's menu looks identical to every customer who opens it. So 119 times out of 120, the database is doing 25 ms of work to produce an answer it has already produced, byte for byte.

A copy that's closer and cheaper

The fix is to keep the answer somewhere fast after computing it once, and serve the next request from there.

A cache is a fast, temporary copy of data whose real home is somewhere slower.

Two words in that definition matter. Copy: the database stays the source of truth. The cache can be thrown away at any moment and rebuilt. Temporary: copies go out of date, and every cache needs a plan for when to stop trusting a copy. Most of this module is about that plan.

For Tiffin, the first cache will be an in-memory store called Redis, sitting next to the server:

%% caption: Most menu requests stop at the cache. Only misses reach the database.
flowchart TD
  CA(["Customer app"]) --> S["Server"]
  S -- "menu for 42?" --> C[("Cache")]:::hot
  C -- "found, 0.5 ms" --> S
  S -. "only when the cache misses" .-> D[("Database")]:::store

Fetching a stored menu from Redis takes about half a millisecond, most of which is the network hop. The database query took 25 ms. That's 50 times less time per request, and far less work for the database.

Hit ratio: the number that matters

Not every request finds its answer in the cache. When it does, that's a hit. When it doesn't, that's a miss, and the request goes to the database as before (and usually stores the answer for next time).

The share of requests that hit is the hit ratio, and it decides almost everything.

Notice how the gains keep coming at the top end. Going from 90% to 99% looks like a small step, but it cuts database load by another 10 times. A miss costs 50 times more than a hit, so misses dominate the average.

Average time = hit ratio x 0.5 ms + miss ratio x 25.5 ms
At 95%:  0.95 x 0.5 + 0.05 x 25.5 ≈ 1.75 ms
At 99%:  0.99 x 0.5 + 0.01 x 25.5 ≈ 0.75 ms

(A miss costs 25.5 ms: the 0.5 ms check that found nothing, plus the 25 ms query.)

What decides the hit ratio? Mainly how often the same key is read before it changes or expires. Menus are great: 120 reads per change, and popular restaurants get read far more than that. A customer's own order history is weaker: it's read only by that one customer, a few times.

What caching buys you

Three things, in roughly this order of importance:

1. Less load on the database. This is usually the main reason. The database is the hardest part of the system to scale (Modules 5 and 9 are all about that), so every query it doesn't have to run buys time before you need those modules.
2. Faster responses. 0.5 ms instead of 25 ms. For a phone 60 ms away, this alone doesn't transform the experience, but it removes the slowest part of the server's work and keeps response times steady under load.
3. Cheaper scaling. A Redis server with a few gigabytes of memory is far cheaper than a bigger database, and much simpler than adding database replicas.

What caching costs you

Caching isn't free, and it's easy to underestimate the cost.

• Stale data. The cache holds a copy. When the real data changes, the copy is wrong until something fixes it. A customer might see a dish that sold out a minute ago. This is the main cost, and lessons 4 and 5 deal with it.
• A new moving part. Redis has to be run, monitored and upgraded. If it slows down or fails, you need a plan (lesson 6).
• New kinds of bugs. "Works when I test it, wrong for some users" is a classic caching bug, because what you see depends on what was cached when.
• A dependency you don't notice. After a year at a 99% hit ratio, the database is sized for 1% of the real load. If the cache empties, the full load arrives at once. We'll deal with that too.

When not to cache

Caching helps when the same answer is read many times before it changes. So it helps little, or hurts, when:

• Data is rarely read twice. A one-off export, a receipt opened once. You pay to store it and never get a hit.
• Data changes almost as often as it's read. A rider's live location changes every few seconds. Caching it adds complexity and staleness for little gain. (Though storing only the latest value in a fast store is a different idea, which we'll see in lesson 4.)
• Data must be exactly right, right now. Is this payment complete? Is this coupon already used? Decisions involving money or uniqueness must read the source of truth.
• The query is already fast. Looking up one order by its primary key takes about 1 ms. A cache saves almost nothing and adds a way to be wrong.

A good habit: before adding a cache, find the queries that actually cost the most. Look at the database's slow-query log or its per-query statistics and sort by total time (time per call x number of calls). The menu query, at 25 ms x 400 a second, will be near the top. A 2-second report that runs twice a day won't be, and caching it achieves nothing.