Load balancers

Tiffin now runs on several identical servers. The customer app, though, only knows one address: api.tiffin.app. Something has to sit at that address, take every incoming request, and hand it to one of the servers. That something is a load balancer.

It sounds like a simple job, and the basic version is. But the choices you make here decide whether a dead server causes one failed request or a full outage, and whether a slow checkout clogs up everyone's menus.

A load balancer is a single entry point that receives requests and forwards each one to one of several servers, while keeping track of which servers are healthy.

What a load balancer does

%% caption: Every app talks to one address. The load balancer decides which server answers.
flowchart TD
  CA(["Customer app"]) --> LB
  KT(["Kitchen tablet"]) --> LB
  RA(["Rider app"]) --> LB
  LB["Load balancer<br/>api.tiffin.app"]:::hot --> S1["Server 1"] & S2["Server 2"] & S3["Server 3"]

Its jobs, roughly in order of importance:

1. Pick a server for each request or connection.
2. Stop sending traffic to servers that are down, and start again when they recover.
3. Hide the servers. Clients never learn their addresses, so you can add, remove and replace servers freely.
4. Often, handle HTTPS. The load balancer holds the TLS certificate, decrypts the traffic, and talks plain HTTP to the servers on the private network. This is called TLS termination, and it saves each server the CPU cost of encryption.
5. Drain servers gracefully before they're removed, so in-flight requests can finish.

You can run one yourself (HAProxy, nginx and Envoy are the common open-source ones) or use your cloud's managed load balancer. For most teams, managed is the right call: it's already redundant, scales itself, and nobody gets paged when its machine dies.

Picking a server

The rule for choosing a server is called the algorithm. Here are the ones you'll actually meet:

Why round robin isn't always fair

Round robin assumes every request is roughly equal. In Tiffin, they aren't. A menu read takes 20 ms. A checkout that waits on the payment provider takes 1.5 seconds.

Picture three servers with round robin. By bad luck, server 2 gets several checkouts in a row. It now has 30 slow requests in progress while servers 1 and 3 have 3 each. Round robin keeps handing server 2 its equal share anyway, so the queue there grows, and every menu request that lands on server 2 waits behind checkouts.

Least connections fixes this by looking at what's actually in progress. Server 2 is busy, so new requests go elsewhere until it catches up. For any app with a mix of fast and slow requests, it's a better default than round robin.

The power of two choices row is worth a note. Checking every server for every request gets expensive with hundreds of servers and several balancers. Picking two at random and using the less busy one spreads load almost as well, and Envoy's "least request" mode works this way.

Hashing

Sometimes you want the same user to land on the same server, for example to reuse an in-memory cache. Hashing the user ID to pick a server does that. The catch is that adding or removing a server changes which server most users map to. Consistent hashing reduces that reshuffling, and you'll meet it properly in Modules 14 and 19. For a normal web app, you usually don't need this: the next lesson shows how to make servers stateless so it doesn't matter which one answers.

Layer 4 or layer 7

Load balancers work at one of two levels, named after layers of the network model.

Layer 4 balancers work with TCP connections. They see IP addresses and ports, and nothing inside the traffic. They pick a server when a connection opens and pass bytes back and forth. They're very fast and can handle millions of connections, but they can't read URLs, headers or cookies.

Layer 7 balancers understand HTTP. They read each request, so they can route by path (/rider/* to the rider servers) or header, balance each request separately even over one long-lived connection, retry a failed request on another server, and add headers like the client's real IP address.

For Tiffin's HTTP API, layer 7 is the natural choice. Layer 4 shines in front of non-HTTP traffic, like a database proxy, or at very high connection counts.

Reverse proxy, load balancer, gateway

You'll hear three names for the box at the front, often for the same piece of software. They're worth separating, because they describe different jobs, not different products.

A reverse proxy is a server that accepts requests on behalf of other servers, forwards them, and returns the answers as if they were its own.

"Reverse" because a normal (forward) proxy acts for the client, like an office proxy that fetches web pages for employees. A reverse proxy acts for the servers: clients think they're talking to api.tiffin.app, and never see what's behind it.

A reverse proxy is useful even with a single server behind it. The jobs it does:

• TLS termination. It holds the certificate and does the encryption, so the app server speaks plain HTTP on the private network.
• Compression. It gzips or Brotli-compresses responses. A 30 KB menu JSON shrinks to about 5 KB, which matters on a phone.
• Caching. It can keep responses to public, slow-changing requests, like GET /restaurants/42/menu, and answer repeats without touching the app at all.
• Absorbing slow clients. A phone on a weak network might take 3 seconds to upload a request or download a response. The proxy buffers it, so an app worker is busy for 20 ms instead of 3 seconds. This is one of the oldest reasons to put nginx in front of an app server.
• Routing. /rider/* goes to one pool, /static/* to a folder of files, everything else to the API.
• Hiding origins. Clients only ever see the proxy's address, so the servers behind it can change freely and aren't directly attackable from the internet.

Now the three names:

So a load balancer is a reverse proxy that specialises in choosing between identical servers, and an API gateway is a reverse proxy that specialises in the rules for an API. Tools blur the line on purpose. nginx started as a web server and reverse proxy and does load balancing. HAProxy is mostly a load balancer that does proxy jobs well. Envoy does all three and is the engine inside many gateways and service meshes. Module 11's API gateway lesson covers the gateway role in depth.

For Tiffin today, one managed layer 7 load balancer does the reverse-proxy jobs that matter: TLS, routing and health checks. Compression and caching of menus can live there too, or in the CDN once Tiffin has one (Module 3).

Health checks

A load balancer is only as good as its knowledge of which servers are healthy. It learns that in two ways.

Active checks. Every few seconds, the balancer calls an endpoint like GET /healthz on each server. If it fails twice in a row, the server is taken out. After three successes in a row, it's put back. Requiring several results in a row stops one slow response from flapping a server in and out.

Passive checks. The balancer also watches real traffic. If a server returns errors or times out on several real requests, it's taken out for a while even before the next active check.

What should /healthz check?

This is where many teams cause their own outages.

The tempting answer is "check everything": the database, Redis, the payment provider. Then the health check really means "can this server do its job?" But think about what happens when the database gets slow at 8pm. Every server's health check fails at the same moment, because they all share that database. The load balancer removes every server. Now there's nothing to send traffic to, and a slow database has become a total outage.

A better rule:

• The health check should answer "is this server process able to serve requests?" Is it up, has it finished starting, is it not stuck?
• It should not fail because of a shared dependency that every server depends on equally. Taking servers out won't fix a shared problem.

A slow database is a real problem, but it's one for your alerts and your error handling, not for the load balancer.

Draining

When you remove a server on purpose, for a deploy or a scale-down, you don't want to cut off requests halfway. Draining (also called deregistration delay) handles this: the balancer stops sending new requests to the server but lets in-flight ones finish, up to a time limit like 30 seconds. Then the server can shut down cleanly. You'll lean on this heavily in the deploys lesson.

Who balances the load balancer?

If every request goes through one box, haven't you just moved the single point of failure?

You have, unless it's redundant too. Managed cloud load balancers run as many machines behind the scenes, spread across zones, so this is handled for you. If you run your own, the usual setup is two balancers sharing a floating IP address: one is active, and if it stops responding, the standby takes over the address within a few seconds. Tools like keepalived do this.

Spreading traffic across whole regions with DNS and anycast is a different level of the same idea, covered in Module 18.

Tiffin's setup

Here's the plan written out, in plain terms rather than any one product's syntax:

listener:      HTTPS on port 443, certificate for api.tiffin.app
algorithm:     least outstanding requests (least connections, counted per request)
targets:       app servers on port 8080, spread over 2 zones
health check:  GET /healthz every 5s, 2s timeout
               remove after 2 failures, restore after 3 successes
draining:      30 seconds
retries:       only for GET requests, at most 1, on a different server

That last line matters. If the balancer automatically retries a POST /orders that timed out, the first attempt may have actually succeeded, and the customer gets charged twice. Retrying safely needs idempotency keys, covered in Module 12. Until then, let the balancer retry only requests that are safe to repeat.