Deploying without downtime

Tiffin's team ships new code about five times a week. On one server, a deploy meant stopping the app and starting the new version: about 40 seconds where every request failed. The team learnt to deploy at 4pm, when traffic is low, and hoped nobody noticed.

With several servers behind a load balancer, there's no longer any reason for a deploy to cause downtime. There's always another server that can take the traffic while one is being updated. But "no downtime" is only half the goal. New code is the single most common cause of outages, so the other half is: when the new version is broken, find out fast, and affect as few people as possible.

This lesson covers the three standard ways to release: rolling, blue-green and canary.

First: shut down gracefully

Before any strategy works, each server needs to stop politely. If you just kill the process, every request it was handling fails. By Little's law (from the first lesson), a server handling 500 req/s at 50 ms each has about 25 requests in flight at any moment. Kill it, and all 25 fail, possibly including someone's checkout.

A graceful shutdown goes like this:

1. The server receives a stop signal (on Linux, usually SIGTERM).
2. It starts failing its readiness check, so the load balancer stops sending new requests.
3. It waits a few seconds for the balancer to notice, then stops accepting new connections.
4. It finishes the requests already in progress.
5. It closes database connections cleanly and exits.

In a Node.js server, the core of it is a few lines:

let ready = true;
app.get('/healthz', (req, res) => res.status(ready ? 200 : 503).end());

process.on('SIGTERM', () => {
  ready = false;                       // load balancer starts draining us
  setTimeout(() => {
    server.close(() => process.exit(0)); // finish in-flight, then exit
  }, 10_000);
});

Combined with draining on the load balancer (from the load balancers lesson), this makes removing a server invisible to customers. Every strategy below depends on it.

Rolling deploys

The simplest strategy: replace servers a few at a time.

For each batch: drain the old servers, start new ones with the new version, wait for their health checks to pass, then move to the next batch. If the new servers fail their health checks, the rollout stops.

For Tiffin, with 6 servers updated 2 at a time and about 3 minutes per batch, a deploy takes around 9 minutes.

Two things to watch:

• Capacity. If you take 2 of 6 servers out at once, you're running on two-thirds capacity during the deploy. That's fine at 4pm, and dangerous at 8pm. Many platforms let you surge instead: start new servers first, then remove old ones, so capacity never drops.
• Rollback is just another rolling deploy. If something goes wrong after the rollout, rolling back to the old version takes another 9 minutes.

Rolling deploys are the default on most platforms, including Kubernetes, and they're a good one. But they have a property you can't avoid: for those 9 minutes, both versions are serving traffic at the same time.

Two versions at once

During a rolling deploy, a customer's menu request might hit v2 and their checkout a second later might hit v1. Both versions read and write the same database. So every change must work with both the old code and the new code running together.

That's easy for most changes. It's hard for database schema changes. Say you want to rename customers.phone to customers.phone_number. If v2's migration renames the column, every v1 server still running immediately starts failing with "column phone does not exist".

The fix is to split one breaking change into several safe ones. This is called expand and contract:

-- release 1
ALTER TABLE customers ADD COLUMN phone_number text;

-- release 2 (in batches, so you don't lock the table)
UPDATE customers SET phone_number = phone
WHERE phone_number IS NULL AND id BETWEEN 1 AND 50000;

-- release 5, only once no running code reads or writes "phone"
ALTER TABLE customers DROP COLUMN phone;

Why can't release 4 drop the column itself? Because during release 4's rolling deploy, release 3 servers are still running, and they still write phone. Drop it too early and they fail.

It's five deploys instead of one. That's the price of never going down. At any point, the version before and the version after can run side by side, and you can roll back one step safely.

The same rule applies to your API. Old versions of the Tiffin app stay on people's phones for months, so the server must keep accepting what they send. Module 2 covered versioning the API for exactly this reason.

Blue-green deploys

Instead of replacing servers one by one, run two complete environments. Blue is live. You deploy the new version to green, test it there with real infrastructure but no customers, and then switch the load balancer from blue to green in one step.

The big win is rollback. If v2 misbehaves, switch the load balancer back to blue, and you're on the old version in seconds, not minutes. Blue stays running for a while after each release for exactly that reason.

The costs:

• Double the servers during a release. With autoscaling, this is temporary, but it's still real money at scale.
• The database is shared. Blue and green talk to the same database, so the expand-and-contract rule still applies. Rolling back the code doesn't roll back a migration.
• Cold start. Green's in-memory caches and connection pools are empty when it suddenly gets 100% of traffic. Warm them up before the switch.

Canary releases

Coal miners used to carry a canary underground. If there was gas, the bird showed signs first, and the miners got out. A canary release uses a small slice of real traffic the same way.

A canary release sends a small share of real traffic to the new version, compares it with the old version, and only continues if the new one is at least as healthy.

A typical schedule for Tiffin:

step 1:   5% of traffic to v2 for 15 minutes
step 2:  25% for 15 minutes
step 3:  50% for 15 minutes
step 4: 100%
at every step: if v2's error rate or p99 latency is clearly worse than v1's, send all traffic back to v1

Rolling and blue-green tell you if v2 starts. Canary tells you if v2 works, with real customers, while limiting the damage to a few percent of them. Modern deploy tools automate the comparison, so a bad canary rolls itself back while the team is still reading the alert.

Canaries need enough traffic

Here's a trap. Say v2 has a bug that breaks 2% of checkouts. At 4pm Tiffin does about 2 checkouts a second. 5% of that for 15 minutes is:

2 checkouts/s x 0.05 x 900 s = 90 checkouts on v2
2% of 90 = about 2 failures

Two failures is easy to miss in normal noise. The canary passes, the bug goes to 100%, and you find it at dinner. So either run the canary longer, run it on a larger slice, or watch business metrics like "orders completed per minute", not just HTTP errors. Small canaries catch big bugs. They're bad at catching small ones.

Choosing a strategy

These aren't exclusive. A common setup is canary first, then a rolling rollout for the rest.

Tiffin's rules end up as:

1. Every server shuts down gracefully.
2. Schema changes follow expand and contract. Never a breaking change in one release.
3. Each release goes to a 5% canary for 15 minutes, comparing errors, latency and orders per minute. Then it rolls out to the rest.
4. No deploys between 7pm and 10pm, unless it's fixing an incident. Dinner is not the time to learn something new.