Where caches live

"Add a cache" sounds like one decision. It's really two: what to cache, and where. Data can be cached on the customer's phone, at a server in their city, inside your app server's memory, or in a shared cache like Redis. Each place has a different speed, a different reach, and a very different answer to "how do I fix it when it's wrong?".

In this lesson we'll walk the path from the phone to the database, look at each place a cache can sit, and decide what Tiffin should cache where.

The layers, from near to far

%% caption: Each layer can answer a request before it travels further. The nearer the layer, the faster, and the harder to correct.
flowchart TD
  P(["Phone<br/>app and browser cache"]):::hot --> CDN["CDN<br/>servers near the user"]
  CDN --> SM["Server memory<br/>inside the app"]
  SM --> R[("Redis<br/>shared cache")]:::store
  R --> D[("Database<br/>the source of truth")]:::store

Let's take them in turn.

On the phone: the client cache

The fastest request is the one never sent. Phones and browsers keep copies of responses they've already received, and HTTP has a whole system of headers for telling them what they may keep and for how long.

For a dish photo, the server can send:

Cache-Control: public, max-age=31536000, immutable

That says: anyone may cache this, for a year, and it will never change. The phone won't ask for it again.

"Never change" sounds risky for a photo the restaurant might replace. The trick is to put a version or a hash of the content in the URL:

https://img.tiffin.app/dishes/7/a3f9c2e1.jpg

When the restaurant uploads a new photo, it gets a new hash, and the menu points at a new URL. The old one stays cached forever and simply stops being used. This is called cache busting, and it's the standard way to cache static files: never change what's at a URL; change the URL.

For data that does change, like a menu, you can still allow a short client cache:

Cache-Control: private, max-age=30
ETag: "menu-42-v17"

private means only this phone may cache it, not shared caches along the way. After 30 seconds the phone asks again, but sends If-None-Match: "menu-42-v17". If nothing changed, the server replies 304 Not Modified with no body: a tiny response instead of 30 KB.

The big limitation of the client cache: once it's on the phone, you can't take it back. If you send max-age=86400 on a menu and the price changes, that customer sees the old price for up to a day. So client caching suits things that never change (versioned files) or things where a little staleness is fine (and checkout re-checks prices on the server anyway).

At the edge: the CDN

A CDN (content delivery network) is a company that runs caching servers in hundreds of locations around the world. You point a domain, like img.tiffin.app, at it. When a phone asks for a photo, the request goes to the nearest CDN location. If that location has it cached, it answers immediately. If not, it fetches it once from your server (or object storage), keeps it, and serves everyone nearby from then on.

Remember the estimate from Module 2: photos alone would push about 80 Mbit/s out of Tiffin's server at dinner time, and that was at launch-year traffic. Today it would be about four times that. With a CDN, almost all of that is served by the CDN's machines. Our server barely sees photo traffic at all.

CDNs are great for:

• Images, video, app bundles, JavaScript and CSS: big files, same for everyone.
• Public API responses that are identical for everyone in an area, if you're careful with headers.

They're wrong for anything personal. A CDN that caches /me/orders and serves one customer's orders to the next person is a serious data leak. Responses that depend on who's asking must be marked private or no-store.

Correcting a CDN is possible but slow-ish: you send a purge request and wait seconds to minutes for every location to drop the copy. Versioned URLs avoid needing that at all. We'll see CDNs again in Module 18, for routing users around the world.

Inside the server: in-process memory

The app server itself can keep data in a variable: a map from key to value, living in its own memory.

const local = new Map() // key -> { value, expiresAt }

It's the fastest cache there is. No network, no serialisation, about a ten-thousandth of a millisecond. But it has real limits:

• It's private to one server process. When Tiffin runs ten servers (Module 4), there are ten separate caches, each with its own copy, each going stale on its own schedule. Deleting a key on one doesn't touch the others.
• It vanishes on restart. Every deploy starts with an empty cache. If you deploy at 8pm, the database gets the full load at the worst moment.
• It uses the server's memory. Cache too much and the app runs out.

So in-process caching suits small, hot data that changes rarely and where a little staleness across servers is fine: delivery fee rules, feature flags, the list of cities. Keep it for seconds to minutes and let it refresh on its own.

A shared cache: Redis

Redis (and its older cousin Memcached) is a separate server that keeps data in memory and answers lookups over the network. Every app server talks to the same Redis.

GET menu:42        -> the cached menu, or nothing
SET menu:42 "<json>" EX 600
DEL menu:42

Because there's one shared copy:

• All servers see the same thing. Delete a key once and every server gets fresh data on the next request.
• It survives app deploys. Restart the app servers as often as you like; the cache is still warm.
• It's still very fast. About 0.5 ms including the network round trip inside the data centre. One Redis server can handle 100,000 or more simple operations a second.

Its cost is that it's another server to run and monitor, and another thing that can fail. Most cloud providers offer it as a managed service, which takes care of most of that.

The database has a cache too

One more layer you get for free. Databases keep recently used data pages in memory (PostgreSQL calls this its shared buffers, and also relies on the operating system's file cache). That's why a query is much faster the second time: the data is already in memory.

This is why a database with enough RAM to hold its hot data feels fast, and why the same database slows down sharply once the hot data no longer fits. It doesn't remove the work of running the query, though. Even from memory, our menu query still joins hundreds of rows. A Redis cache stores the finished answer.

Tiffin's choices

Notice the pattern: big, unchanging, public things go as close to the user as possible. Shared, changing things go in Redis, where one delete fixes them. Personal and money-related things aren't cached at all.