Long polling

In the last lesson, polling Tiffin's tracking screen every 2 seconds came to 5,000 requests a second at dinner time, and more than 99% of them came back with "nothing changed". The fix is a small change in who waits. Instead of the server answering "nothing yet" straight away, it holds on to the request and answers only when there's something to say.

That's long polling. It's still plain HTTP, still one request and one response, but the empty answers mostly disappear.

Hold the question open

With normal polling, the conversation goes like this, over and over:

Client: Anything new?    Server: No.
Client: Anything new?    Server: No.
Client: Anything new?    Server: No.
Client: Anything new?    Server: Yes, rider picked up.

With long polling, it goes like this:

Client: Anything new? I'll wait.
        ... 40 seconds pass ...
Server: Yes, rider picked up.
Client: Anything new? I'll wait.

Long polling is a request the server deliberately leaves unanswered until there is news or a timeout passes.

The client sends a request. If there's news, the server answers immediately. If not, the server parks the request and waits. The moment news arrives, it answers. The client reads the answer and immediately sends the next request. So there is almost always one request open, waiting.

%% caption: The request sits open on the server until there is something to say.
sequenceDiagram
  autonumber
  participant C as Customer app
  participant S as Server
  participant K as Kitchen tablet
  C->>S: anything new? I'll wait
  Note over S: parks the request
  K->>S: marks order 123 ready
  S-->>C: yes, order ready
  C->>S: anything new? I'll wait
  Note over S: parks it again

Delay drops to roughly one network round trip, because the answer goes out the moment the news exists. And the number of requests drops to roughly the number of updates, plus one per timeout.

Always send a timeout

The server can't hold a request forever. Everything between the phone and your server has an idle timeout: the mobile network, corporate proxies, and your own load balancer. An AWS Application Load Balancer, for example, closes connections that sit idle for 60 seconds by default, and nginx gives up on a slow upstream after 60 seconds unless you change it.

So the server answers "nothing new" after a fixed wait, usually 20 to 30 seconds, safely below every timeout in the path. The client treats that empty answer as normal and asks again.

Let's redo the Tiffin maths. 10,000 tracking screens, each order with six updates over 30 minutes, and a 30-second timeout:

timeouts per screen:  1,800 seconds / 30 = 60 requests
updates per screen:   6 requests
total per screen:     ~66 requests in 30 minutes

all screens:  10,000 x 66 / 1,800 seconds = ~370 requests per second

Compare that with 5,000 requests a second for 2-second polling, and the delay is now better too: under a second instead of an average of one second. You got both, which is rare in this course.

Don't lose news in the gap

There's a subtle hole. Between the server answering one long poll and the client's next request arriving, maybe 100 milliseconds pass. If the rider is assigned during that gap, there's no open request to answer. Is the news lost?

It is, unless the client says where it got up to. Every update gets an increasing number (a version or event ID), and the client sends the last one it saw:

GET /orders/123/updates?after=5

The server checks first: is there anything newer than 5? If yes, answer immediately. Only if not does it park the request. Now the gap doesn't matter, because the next request picks up anything that happened during it. This "send me everything after N" idea, called a cursor, appears in every live-update technique in this module. Remember it.

What it costs the server

Long polling moves the cost from requests to open connections. At any moment, about 10,000 requests are sitting open on Tiffin's servers, doing nothing.

Whether that's a problem depends entirely on how your server handles requests.

This catches teams out. A Rails app with 5 servers and 16 threads each can serve 80 requests at once. Add long polling for 10,000 screens and the first 80 waiting customers block everyone else, including people trying to place orders. If your app runs on thread-per-request, serve long polls from a separate small service built on an event loop, not from the main app.

How does the parked request hear the news?

The waiting request lives in one server's memory. The news comes from somewhere else: the kitchen tablet marks the order "ready" with a request that may land on a different server.

On a single server, it's easy. Keep a map in memory from order ID to waiting requests. When the order changes, look up the map and answer them.

const waiting = new Map() // orderId -> list of open responses

function onOrderChanged(orderId, update) {
  for (const res of waiting.get(orderId) ?? []) res.json([update])
  waiting.delete(orderId)
}

With several servers behind a load balancer (Module 4), the kitchen's update lands on server 2 while the customer's request waits on server 5. Server 5 never hears about it. That's the central problem of live updates at scale, and it gets its own lesson later in this module. For now, notice that it's not specific to long polling. Every technique where the server pushes has to solve it.

Where long polling still shows up

Long polling was the main way to build chat on the web before browsers had better tools, and it's still common today because it's just HTTP:

• Telegram's bot API lets a bot fetch messages with a long-poll timeout instead of receiving webhooks.
• Consul's blocking queries let a client "watch" a key by sending a request that waits until the value changes past a given index. That's long polling with a cursor.
• Many apps use it as a fallback when a better connection type is blocked by a proxy.

Its weaknesses are real, though. Each update costs a full HTTP request and response, headers and all. If updates arrive many times a second, the client spends its time reconnecting. And every update is followed by a fresh request, so there's a small window where the client is between requests. For a stream of frequent updates, you want one connection that stays open and carries many messages. That's the next lesson.