Live updates across many servers

Everything so far has quietly assumed one server. The customer's SSE connection and the rider's WebSocket both landed on the same machine, so when the rider sent "picked up", the server could find the customer's connection in its own memory and write to it.

Tiffin hasn't had one server since Module 4. It now runs 8 app servers behind a load balancer. The rider's socket landed on server 3. The customer's tracking screen is connected to server 7. Server 3 receives "picked up" and has no idea where the customer is.

This lesson is about getting a message from the server that hears the news to the server that holds the connection.

The problem in one picture

In Module 4 we made servers stateless so any server could handle any request. That's still true for normal requests. But an open connection is state that can't move: it's a socket owned by one process on one machine. Only server 7 can write to the customer's screen.

So there are really two questions:

1. When something happens, how does it reach the server holding the right connection?
2. How does that server know which of its connections care?

The second is the easy one. Each server keeps an in-memory map, like order 123 -> [connection A, connection B], updated as clients connect and disconnect. The first is where the design choices are.

Option 1: Sticky routing

Can we just make sure the rider and customer for an order land on the same server? The load balancer could route by order ID.

It falls apart quickly. A rider carrying two orders would need to be on two servers at once. A customer watching their order and a restaurant watching all of its orders need different groupings. And when a server dies, everything assigned to it has to move. Sticky routing fits narrow cases, like a chat room where everyone is in exactly one room, but it's not a general answer.

Option 2: Every server polls the database

Each server checks the database once a second: "have any orders my clients care about changed since my last check?"

SELECT id, status, version
FROM orders
WHERE id = ANY(:watched_order_ids)
  AND updated_at > :last_check;

This sounds like a step backwards, since we spent a whole lesson getting away from polling. But look at the numbers. Before, 10,000 phones were polling. Now 8 servers poll, once a second each. That's 8 queries a second, batched, instead of thousands. The phones still get pushed updates.

It's simple and works with what Tiffin already has. The costs: up to a second of extra delay, and the query gets heavier as each server watches more orders. One trap: a transaction that commits slightly late can carry an updated_at older than your last check, and the row is skipped forever. Overlap each check by a couple of seconds and use version to ignore what you've already pushed. For a modest number of servers and relaxed latency, it's a respectable choice that people overlook.

Option 3: A shared message channel

The standard answer is to put a pub/sub system between the servers. Servers publish messages to named channels, and any server subscribed to a channel receives every message published to it.

Pub/sub is a messaging pattern where publishers send to a named channel and every current subscriber of that channel receives a copy.

Tiffin already runs Redis from Module 3, and Redis has a built-in pub/sub feature. Here's how it fits:

1. When the customer's tracking screen connects to server 7, server 7 subscribes to channel order:123.
2. When the rider's "picked up" reaches server 3, server 3 saves it to the database, then publishes to order:123.
3. Redis delivers the message to every server subscribed to order:123. That's just server 7.
4. Server 7 looks up its map and writes to the customer's connection.

%% caption: The server that hears the news and the server that holds the connection meet in Redis.
sequenceDiagram
  autonumber
  participant C as Customer app
  participant S7 as Server 7
  participant R as Redis
  participant S3 as Server 3
  participant RA as Rider app
  C->>S7: start tracking order 123
  S7->>R: SUBSCRIBE order:123
  RA->>S3: picked up
  Note over S3: saves to the database
  S3->>R: PUBLISH order:123
  R-->>S7: message on order:123
  S7-->>C: picked up

// On server 7, when a customer starts watching order 123
await subscriber.subscribe(`order:${orderId}`, (message) => {
  for (const conn of watchers.get(orderId) ?? []) conn.send(message)
})

// On server 3, after saving the status change
await redis.publish(`order:${orderId}`, JSON.stringify(update))

Servers unsubscribe when the last watcher of a channel disconnects, so a server only receives messages it needs.

Redis pub/sub forgets

There's one property of Redis pub/sub you must design around: it's fire and forget. A message goes to whoever is subscribed at that instant. If server 7 was restarting, or its connection to Redis blipped, the message is gone. Redis doesn't store it.

That's fine as long as the database is the source of truth and the channel only carries notifications. Two habits make it safe:

• On every connect and reconnect, subscribe, then send the client the current state from the database, then stream updates (skipping any with an older version). Subscribing first means nothing slips through between the read and the subscribe, and a missed message gets fixed the next time the client reconnects.
• Prefer messages that carry full state ("status is now picked_up, version 8") over deltas ("status moved one step forward"). If one is lost, the next one still leaves the client correct.

If you can't tolerate any gaps, Redis also has streams, which keep messages so a subscriber can read from where it left off. Full message brokers like Kafka go further. We'll study those properly in Module 15.

How far does one Redis go?

A single Redis instance handles a few hundred thousand pub/sub messages a second on decent hardware. Tiffin's order status changes are well under 100 a second at peak. Even 1,500 rider location updates a second is far from the limit. One Redis is plenty for one city, with a replica for failover.

Option 4: A connection directory

Pub/sub sends each message to every subscribed server. For something like chat at a huge scale, a different approach routes messages directly. Keep a directory in Redis that maps each user to the server holding their connection, like customer:42 -> server 7. To send a message, look up the server and call it directly.

This saves fan-out work when there are many servers and each message is for one person, but you now have to keep the directory accurate as connections come and go, and clean it up when a server dies without saying goodbye. Most apps don't need it until they're very large.

A separate tier for connections

Once live updates matter, many teams split the servers in two:

• Connection servers hold sockets and SSE streams, and do almost nothing else: authenticate, subscribe, forward.
• App servers run the business logic and handle normal API requests, as before.

The reason is deploys. Tiffin's app servers deploy several times a day. If they also held the connections, every deploy would disconnect thousands of customers and riders. Connection servers have very little code, so they change rarely, and customers stay connected through app deploys.

%% caption: App servers publish. Connection servers subscribe and hold the sockets. Each can be deployed and scaled on its own.
flowchart TD
  C([Customer app]) -- "live stream" --> CS["Connection servers<br/>hold the sockets"]:::hot
  C -- "normal API requests" --> AS["App servers<br/>business logic"]
  AS -- "publish changes" --> R[["Redis pub/sub"]]
  R -- "deliver to subscribers" --> CS

When a server dies

Say connection server 2 crashes while holding 5,000 customer streams. All 5,000 clients notice within seconds and try to reconnect. If they all reconnect immediately, the surviving servers take 5,000 new connections, each with a login check, a database read for current state, and a Redis subscribe, in the same second. That spike can knock over a second server, whose clients then reconnect too. This is a reconnect storm.

The defences are the ones you've already seen: random jitter on the reconnect delay (SSE's retry field helps here), exponential backoff on repeated failures, and serving the "current state" read from cache rather than the primary database.