Lab: Live order tracking

• Freshness: the rider's position on the customer's map is at most about 5 seconds old. Status changes ("picked up", "arrived") appear within 2 seconds.
• Smoothness: the icon moves smoothly, not in jumps. (This is a client problem, as we'll see, not a data rate problem.)
• Privacy: the customer sees the rider's location only between pickup and delivery, and only for their own order.
• Bad networks: if the rider loses signal, the customer sees "last updated 40 seconds ago" rather than a frozen icon pretending to be live. If the customer loses signal, the map catches up as soon as they're back.

Notice what isn't required: a full history of every point on the screen, or sub-second accuracy. A delivery map is not a racing game.

(a) Orders on the road: 11 orders a second x 15 minutes of riding (900 seconds) = about 10,000 orders on the road at any moment.

(b) Rider messages in: 6,000 riders / 4 seconds = 1,500 location messages a second.

(c) Customer messages out: half of 10,000 is 5,000 open maps, each getting one position every 4 seconds = about 1,250 messages a second.

Add the customers watching status before pickup (about 10,000 orders are still in the kitchen, so roughly 5,000 more screens if half have the app open) and you have around 10,000 open customer connections plus 6,000 rider connections.

Each message is under 200 bytes, so bandwidth is about 300 KB a second each way. That's nothing. The hard parts here won't be throughput. They'll be connections, fan-out across servers, and failure.

A WebSocket. Traffic flows both ways all the time: locations up, offers and alerts down. A POST every 4 seconds would mean 1,500 full HTTP requests a second through the app servers, each with several hundred bytes of headers, and a separate channel would still be needed for offers.

The socket carries a ~70-byte location frame with no headers, and fewer radio wake-ups help a battery that has to last a 10-hour shift. The rider app gets backoff with jitter on reconnect, ping and pong every 25 seconds, and a "what's my current state" call after reconnecting.

One more decision: the phone sends its GPS reading's own timestamp with each location. If the phone was in a tunnel and sends five buffered points at once, the server can tell which is newest and ignore the old ones.

Writing every point to Postgres is possible, since 1,500 small writes a second is within reach of a good primary, but it's a poor use of it. Over a 3-hour dinner peak that's 1,500 x 10,800 = about 16 million rows a night, almost none of which anyone reads. Those writes compete with order placement on the one database that can't scale writes easily (Module 5).

What people actually read is the latest position. So:

• Latest position in Redis: SET rider:881:loc {lat, lng, ts} EX 60. The 60-second expiry means a rider who disappears simply ages out, rather than sitting on the map forever.
• A sampled history in Postgres: one point every 30 seconds per active delivery, for disputes like "the rider never came". That's 10,000 / 30 = about 330 rows a second, a much smaller table, and it can be written in batches.

SSE. The customer only listens. The occasional action, like "call rider" or "cancel", is a normal request. SSE works through Tiffin's existing load balancer and cookies, and reconnects by itself with Last-Event-ID.

Today's 10-second poll is too slow for a 5-second freshness target. Polling every 4 seconds to meet it would mean 2,500 requests a second from all 10,000 open screens, two and a half times today's load, most of them from status screens where nothing has changed. That's possible, but SSE gives fresher updates for less work.

For smoothness, don't send more points. The app animates between the last position and the new one over 4 seconds, so the icon glides. This is how most delivery and ride-hailing apps make the car move smoothly without a high update rate. Sending a point every second would quadruple the messages for almost no visible gain.

Each message carries full state, not a delta:

id: 1042
event: rider
data: {"lat":12.9716,"lng":77.5946,"ts":1739121010,"status":"picked_up"}

Use Redis pub/sub with one channel per order: order:{id}.

• When a customer opens tracking for order 123, the server holding their stream first checks that the order belongs to them. Then it subscribes to order:123 and sends the current state: the status from the database, and the rider's last position from Redis if the order has been picked up.
• When rider 881 sends a location, their server writes it to Redis, then reads the rider's picked-up orders from a small Redis set, rider:881:orders, and publishes to each of their channels. A rider carrying two orders means two publishes. The set is updated when the rider picks up or delivers an order, which happens a few times an hour, so the 1,500 pings a second never touch Postgres. Update it only after the status change has committed in Postgres. And because a lost update could leave a delivered order in the set, the customer's stream stops forwarding rider positions once it has sent delivered.
• Status changes from the kitchen or the rider are published to the same channel.

Privacy is enforced in two places. Only the order's customer can subscribe, checked when the stream opens. And the location is only published while the order is in the picked-up state, so nothing leaks before pickup or after delivery.

Why one channel per order rather than per rider? A channel per rider would mean a customer's server receives the rider's location even after their own order is delivered and the rider moves on to the next customer. Per order, the channel naturally goes quiet when the order is done.

Volume: one publish per picked-up order every 4 seconds, so 10,000 / 4 = 2,500 publishes a second, plus a handful for status changes. Only about half reach a subscriber (the maps that are open); the rest are dropped by Redis at almost no cost. One Redis handles that easily. Keep the connections on a small connection tier (4 servers, each holding about 4,000 connections, with room for one to fail) so app deploys during dinner don't disconnect anyone.

(a) Connection server crashes. About 4,000 connections drop: roughly 2,500 customer SSE streams and 1,500 rider sockets. Each customer's SSE client reconnects on its own, landing on the other 3 servers; the server's earlier retry: value of a few seconds plus client jitter spreads them out. The rider apps reconnect with backoff and jitter and ask for their current state, so an offer sent during the gap isn't lost. On reconnect, each customer stream sends current state first: one Redis read for the rider's location and a status read from a replica or cache. That's 4,000 cheap reads spread over a few seconds, and the 3 survivors now hold about 5,300 connections each, which is why the tier was sized with a spare.

(b) Rider dead zone. No locations arrive. The rider's key in Redis expires after 60 seconds. The customer's app looks at the ts on the last position, and once it's more than about 20 seconds old, shows "Rider's location last updated 40 seconds ago" (counting up) instead of pretending. When the rider reconnects, the app sends its buffered points, the server keeps only the newest by timestamp, and the map jumps forward.

(c) Redis fails over. For a few seconds, publishes fail and subscriptions drop. Servers reconnect to the new primary, re-subscribe to their channels, and re-send current state to their clients. Riders' latest positions might be a few seconds stale or missing until their next update arrives 4 seconds later. Nothing is lost that matters, because the database holds the order status, and positions are refreshed every 4 seconds anyway. This is exactly why the design treats Redis as a fast notification path, not the source of truth.

The ALB closes connections that have been idle for its idle timeout, 60 seconds by default. A rider's socket pings every 25 seconds, and a moving order sends a position every 4 seconds, so both stay busy. But an order in the kitchen may send nothing for minutes, so its SSE stream sits silent until the ALB cuts it.

The fix is a heartbeat: the connection service sends an SSE comment line (: ping) every 20 to 25 seconds on every open stream. It costs a few bytes, keeps the ALB from treating the stream as idle, and lets the server notice dead clients sooner. You could raise the ALB's idle timeout instead (it goes up to 4,000 seconds), but that only hides the problem from one hop, and any proxy between the phone and you can have its own shorter timeout.