Signing in on a live connection

Every lesson in this module has quietly assumed one thing: the server knows who is on the other end of the connection. The SSE lesson said cookies work out of the box. The WebSockets lesson said to check identity during the handshake. Both are true for a web page. But Tiffin's customer app and rider app are native phone apps, and a live connection that stays open for hours raises questions an ordinary request never does. What happens when the login expires halfway through? How do you throw a banned rider off a socket that's already open?

How Tiffin's apps sign in

When a customer logs in with their OTP, Tiffin's auth service gives the app two tokens:

• An access token, valid for 15 minutes. The app sends it on every API call, as a header: Authorization: Bearer eyJhbGc.... It's a signed token (a JWT), so any server can check it without asking the auth service.
• A refresh token, valid for 60 days, kept in the phone's secure storage (the iOS Keychain or Android Keystore). When the access token is about to expire, the app swaps the refresh token for a new access token.

Short access tokens mean a stolen one is only useful for minutes. Logging someone out, or banning them, just means refusing their refresh token. Within 15 minutes, every access token they hold has expired.

That design is built around short requests. A menu request lasts 50 ms, far inside the token's 15 minutes. A live connection can outlast many tokens.

Native apps can just send the header

Here's the good news first. The limits the earlier lessons mentioned belong to the browser: its built-in EventSource and WebSocket can't set an Authorization header. Native apps aren't bound by that. On Android, OkHttp builds both its SSE and WebSocket connections from an ordinary request. On iOS, URLSession can open a WebSocket from a URLRequest. Both let you set any header:

// Android, OkHttp: the WebSocket handshake carries the same header as every other API call
val request = Request.Builder()
    .url("wss://api.tiffin.app/rider/socket")
    .header("Authorization", "Bearer ${tokens.currentAccessToken()}")
    .build()
client.newWebSocket(request, listener)

So Tiffin's native apps use exactly the same token, in exactly the same header, as their normal API calls. No cookies, nothing in the URL. The connection server checks the token's signature and expiry during the handshake, and either accepts the connection or answers 401 and closes it.

One rule to go with it: identity belongs to the connection, not the messages. Once the handshake proves this is rider 5521, every message on that socket is from rider 5521. If a message says {"type":"accept","rider_id":5522}, ignore the ID in it. And when a customer's app subscribes to /orders/123/events, check that order 123 is theirs. It's easy to authenticate the connection and then forget to authorize what it asks for.

The token expires, the connection doesn't

The rider's socket opens at 6pm with a token valid until 6:15. At 6:16 the token has expired, but the socket is still open, and the server checked the token only once. Nothing forces anything to happen.

For a rider, the risk is small. For a stolen phone, or a rider deactivated for fraud at 7pm, it's real: the socket keeps receiving delivery offers, with customer names and addresses, until the next time it happens to reconnect. That could be hours.

There are three ways to handle it:

Server closes at expiry is simple and works for any connection. For SSE, the server ends the stream. The app gets a fresh token and reconnects, sending Last-Event-ID, so no updates are lost. For a WebSocket, the server closes with a code of its own: codes 4000 to 4999 are reserved for applications, and Tiffin uses 4001 to mean "token expired, refresh and come back".

Is reconnecting every 15 minutes expensive? Tiffin has about 10,000 open tracking screens and 6,000 rider sockets at the dinner peak:

16,000 connections / 900 seconds = about 18 reconnects a second

That's nothing next to the thousands of ordinary API requests a second Tiffin already serves at dinner. Spread the closes out, though: if every rider logged in at 6pm, all 6,000 tokens expire at 6:15 together. Close each connection a little early, at a random moment in its last minute, so they don't all reconnect at once.

Refresh in band avoids the reconnect. Two minutes before expiry, the rider app sends:

{"type": "auth", "token": "eyJhbGc...new..."}

The server checks it like a handshake token, makes sure it belongs to the same rider, and extends the deadline. Tiffin's rider app does this, and the server still closes with 4001 if the deadline passes without a new token. SSE can't do this, because the client can't send anything on the stream.

Throwing someone off now

Fifteen minutes is fine for "the token expired". It's too long for "this rider's account was just suspended for fraud". For that, Tiffin's auth service publishes a message on the same shared channel that carries order updates between servers (the last lesson's Redis pub/sub):

PUBLISH revocations '{"user_id": 5521, "reason": "suspended"}'

Every connection server subscribes and closes any connection that belongs to user 5521, with a code that tells the app to show the login screen rather than retry. If a server misses the message, the 15-minute expiry still gets it.

Reconnecting with a stale token

The most common bug here isn't in the server. It's in the client's reconnect loop.

Many SSE and WebSocket libraries reconnect on their own, and they reuse the request they were given at the start, including its headers. So when the app reconnects at 6:16, it sends the 6:00 token, gets a 401, waits a few seconds, sends the same token again, and gets another 401. Forever. Multiply by every phone after a deploy restarts the connection servers, and you have a steady storm of failed handshakes, all of which hit the auth check.

The rules for the client:

• Read the current token at every reconnect, never capture it once when the connection is first built.
• On 401 or close code 4001, refresh first, then reconnect. If the refresh itself fails, the login is gone: show the login screen and stop.
• Back off with jitter, as the WebSockets lesson said, so a fleet of phones doesn't reconnect in lockstep.

The browser's EventSource has the opposite behaviour, and it's worth knowing for the web dashboard. The standard says that if the server answers with anything other than 200, the connection fails and is not retried. A 401 stops it dead, and your code has to notice the error event and start a new EventSource after refreshing.

The web dashboard: cookies or tickets

Restaurants watch incoming orders on a web dashboard, in a browser. There, the browser's EventSource and WebSocket can't set the header, so there are two good options.

Cookies. The dashboard logs in and gets an HttpOnly, Secure session cookie. The browser sends it on the SSE request and the WebSocket handshake automatically. For WebSockets, also check the Origin header during the handshake and refuse anything that isn't Tiffin's own site. Browsers attach cookies to a WebSocket handshake started by any page, so without that check, a malicious site the restaurant owner visits could open a socket as them.

A one-time ticket. If the dashboard uses tokens instead of cookies, it first makes a normal API call, which can carry the header, to get a short ticket:

POST /live/tickets            (Authorization: Bearer ...)
-> {"ticket": "q7Zr...9vQ", "expires_in": 30}

GET /restaurants/88/events?ticket=q7Zr...9vQ

The ticket is random, valid for 30 seconds, and works once. In Redis that's SET ticket:q7Zr...9vQ 88 EX 30 to create it and GETDEL to use it, so a second attempt finds nothing.

Why not just put the access token in the URL? Because URLs get written down everywhere: load balancer logs, proxy logs, browser history, error trackers. AWS's Application Load Balancer, for one, records the request URL exactly as the client sent it, query string and all. A 15-minute token sitting in a log file that dozens of people can read is a 15-minute token anyone can use. A ticket that has already been used is worthless by the time anyone reads the log.

If API Gateway holds the sockets

On AWS, API Gateway can hold WebSockets for you. Two of its fixed rules shape the design: a Lambda authorizer can only run on the $connect route, at the handshake, never on later messages; and every connection is closed after at most 2 hours, or 10 minutes idle. So with API Gateway, in-band refresh is your own code's job, in the route that handles the auth message, and every client must handle a forced reconnect at least every two hours anyway, which conveniently re-checks its token.