Replication lag

The week after Tiffin added read replicas, a new kind of support ticket started arriving:

"I placed an order, the app said it was confirmed, and then it wasn't in My Orders. I panicked and ordered again. Now I've been charged twice."

Nobody could reproduce it. The order was in the database. It showed up fine when support looked. The bug only existed for about a third of a second, which was exactly long enough to cost Tiffin a refund and a customer's trust.

This lesson is about that third of a second: replication lag, the delay between a change landing on the primary and the same change landing on a replica.

What went wrong

Here's the sequence, with timings:

%% caption: The write went to the primary. The read that followed it went to a replica that hadn't caught up.
sequenceDiagram
  autonumber
  participant C as Customer's app
  participant P as Primary
  participant R as Replica
  C->>P: place order, t = 0 ms
  P-->>C: committed, t = 40 ms
  C->>R: open My Orders, t = 120 ms
  Note over R: lag 300 ms, order not there yet
  R-->>C: empty list
  Note over C: customer panics

Every part did its job. The primary saved the order. The replica was replaying changes as fast as it could. The app followed the rule from the last lesson: order history is a read, and reads go to replicas. Together they produced a page that said the order didn't exist.

How big is the lag?

Within one region, under normal load, a Postgres replica is usually a few milliseconds to a few tens of milliseconds behind. Most of the time nobody could notice.

But "usually" isn't "always". Lag spikes when:

• A big write lands at once. A backfill that updates 2 million rows creates a burst of log records that replicas have to chew through.
• The replica is busy. Heavy reads compete for CPU and disk with the replay work, which in Postgres runs in a single process.
• A long query on the replica holds replay back, as you saw in the last lesson.
• The network hiccups, or the replica restarts and has to catch up.

In those moments, lag can grow to seconds or even minutes. Your design has to survive the bad moments, not just the average.

You can see lag directly. On a Postgres replica:

SELECT now() - pg_last_xact_replay_timestamp() AS lag;

On the primary, the pg_stat_replication view shows each replica's position and lag. One caution: the replica query above measures the time since the last replayed transaction, so on a quiet database with no writes, it grows even though the replica is fully caught up. Monitoring tools usually write a tiny "heartbeat" row every second to keep the number honest.

Read your own writes

The guarantee the customer expected has a name.

Read-your-writes consistency: after you change something, your own later reads always see that change. Other people may see it a little later.

Notice it's narrow. Nobody else needs to see the new order instantly. The restaurant tablet can find out a few hundred milliseconds later without harm. Only the person who made the change needs to see it at once, because they know it should be there.

There are four common ways to provide it. They can be combined.

1. Don't read at all

The cheapest fix is often not to read. The POST /orders response already contains the new order. Have the app put it straight into the list it shows, instead of asking the server for the whole list again.

This fixes the most common case, the screen right after the action, at zero cost. It doesn't help if the customer refreshes or opens the page from another device.

2. Read the user's own data from the primary

Some data is only ever read by the person who owns it and changes it: their orders, their addresses, their saved cards. Reading that from the primary always shows their latest writes.

This costs some primary capacity, but less than you'd fear. One customer's order list is a tiny, indexed query, and customers open it far less often than they browse menus. Menus and search, the big read load, stay on replicas.

3. Use the primary for a short time after a write

When a user writes anything, remember the time, for example in their session. For the next few seconds, send all of that user's reads to the primary. After that, back to replicas.

async function readPool(session) {
  const recentlyWrote = Date.now() - (session.lastWriteAt ?? 0) < 5_000;
  return recentlyWrote ? primary : replicas;
}

Simple, and it covers most real cases. The weakness is the fixed window: if lag ever spikes beyond 5 seconds, the bug comes back exactly when the system is already under strain.

4. Wait until the replica has caught up

The most precise approach. Every change on the primary has a position in the log, called the LSN (log sequence number) in Postgres. After a write, note the position:

-- on the primary, right after the write commits
SELECT pg_current_wal_lsn();          -- e.g. 3A/7F21C8D0

Store it in the user's session. Before reading from a replica for that user, check whether the replica has replayed at least that far:

-- on the replica
SELECT pg_last_wal_replay_lsn() >= '3A/7F21C8D0'::pg_lsn AS caught_up;

If it has, read from the replica. If not, wait a few milliseconds and check again, or just use the primary. There's no guessing about windows: the read is fresh exactly when it needs to be. The cost is extra code, and an extra check before each read for users who've recently written.

Tiffin uses the first two. The app shows the new order from the response, and "My orders" and order details read from the primary. Restaurant lists, menus, search and reviews stay on replicas.

Monotonic reads

Lag causes a second, stranger bug when you have more than one replica.

A customer is watching their order. Their app refreshes the status every few seconds, and each refresh goes to whichever replica the load balancer picks:

8:31:02  replica 1 (lag 20 ms)    ->  "Out for delivery"
8:31:07  replica 2 (lag 3 s)      ->  "Preparing"
8:31:12  replica 1                ->  "Out for delivery"

The order went backwards in time. Each answer was a true snapshot of the past, but stitched together they make no sense.

The guarantee that's missing is called monotonic reads: once you've seen data as of some moment, you never later see an older version of it.

The usual fix is to pin each user to one replica, for example by hashing their user ID to choose it. One replica might be behind, but it only ever moves forward, so the user never sees time go backwards. If that replica dies, the user moves to another and might see one small jump.

Read-your-writes and monotonic reads are two of several consistency guarantees. Module 13 covers the full family.

Other bugs lag causes

Once you know the pattern, you'll spot it in other places:

• Reading from a replica, then writing based on it. Check stock on a replica ("2 left"), then decrement on the primary. If the replica was behind, the stock may already be 0, and you oversell. Any read that feeds a write must come from the primary, ideally in the same transaction.
• A background job that runs right after a write. The order is saved, a job is queued to send the confirmation email, and the job looks up the order on a replica 5 ms later: "order not found". Pass the data in the job, or read it from the primary (Module 7).
• Filling a cache from a replica. A restaurant marks biryani sold out. The app writes to the primary and deletes the cached menu (Module 3). The very next request misses the cache, reads the menu from a replica that's 200 ms behind, and puts the old menu back in the cache, where it stays for the full 10-minute TTL. Either fill caches after a write from the primary, or delete the cache entry again after a short delay.

Watch lag, and act on it

Treat lag as one of the database's vitals:

• Graph it for every replica, and alert when it passes something like 1 second for more than a minute.
• Take badly lagging replicas out of rotation. The health check for a replica can include its lag: if it's more than, say, 10 seconds behind, mark it unhealthy so reads go to the others.
• Slow down big writes. Run backfills and bulk updates in small batches with pauses, and watch lag while they run.