Scaling out

The bigger server bought Tiffin some time, but two problems remain. Traffic keeps growing, and the whole app still lives on one machine. When that machine reboots, every customer, restaurant and rider is cut off.

The other way to grow is to run the same app on several smaller servers and spread the requests across them. This is called scaling out, or horizontal scaling. It's how almost every large web app runs, and it's the reason the rest of this module exists. Adding server two is easy. Making server two behave exactly like server one is where the work is.

Scaling out means handling more load by adding more machines that each do the same job, with something in front to share the work between them.

What changes

Here's Tiffin before and after:

%% caption: After. Three identical servers behind a load balancer, all sharing one database.
flowchart TD
  CA(["Customer app"]) --> LB["Load balancer<br/>spreads requests"]:::hot
  LB --> S1["Server 1"] & S2["Server 2"] & S3["Server 3"]
  S1 & S2 & S3 --> DB[("Database")]:::store

Two new things appear. The load balancer is a single address that customers talk to, and it forwards each request to one of the servers. It gets its own lesson next. The servers themselves are identical copies: same code, same config, any of them can answer any request. Making that last part true is the lesson after.

Notice what didn't change: there's still one database. Every server sends its queries to the same place. That's deliberate. App servers are easy to copy because they hold no data of their own. Databases are hard to copy because they do. Module 5 picks up there.

What you gain

Capacity without a ceiling

Need more? Add a server. Ten servers with 8 cores each give you about the same capacity as one 80-core machine, and you can keep going to 50 or 500. The ceiling moves from "the biggest machine money can buy" to "whatever your database and load balancer can take", which is a much later problem.

Surviving failure

This is usually the bigger win. With three servers, if one dies, the load balancer stops sending it traffic and the other two carry on. Customers might see one failed request. They won't see an outage.

But only if the remaining servers can carry the load. That's where N+1 comes from: work out how many servers (N) you need at peak, then run one more so that losing any single server still leaves enough.

Here's the maths for Tiffin. Each 8-core server has a knee at about 1,400 req/s, and you want peak to stay under 70% of that, so each server should carry at most about 980 req/s.

Peak traffic:            3,600 req/s (projected, six months out)
Per server at 70%:       980 req/s
Servers needed (N):      3,600 / 980 = 3.7, round up to 4
With one spare (N+1):    5 servers

If one of the five dies at 8pm, the other four take 900 req/s each. Busier, but still well under the knee.

Some teams go further and spread servers across two or three availability zones, which are separate data centres in the same region. Then a power cut in one building doesn't take all your servers. With three zones, you'd want to survive losing a whole zone, which pushes you to something like 6 servers, 2 per zone.

Following the traffic

Tiffin's traffic at 4pm is a fifth of its traffic at 8pm. One big machine is sized for 8pm all day. With many small servers, you can run 2 in the afternoon and 6 at dinner.

Clouds automate this with autoscaling: you set a rule like "keep average CPU near 60%", and the platform adds or removes servers to match. There are two traps:

• New servers take time. Booting a machine, pulling the app, warming it up and passing health checks often takes 2 to 5 minutes. Tiffin's traffic can double in 15 minutes around 7:45pm. If you only scale when CPU is already high, the help arrives late.
• So scale on a schedule too. Tiffin's dinner rush happens every day. A scheduled rule that raises the minimum to 6 servers at 7pm and lowers it at 10:30pm handles the predictable part. CPU-based autoscaling handles surprises on top.

Easier, safer changes

With several servers, you can deploy new code to one of them first and watch it before touching the rest, or patch the OS one server at a time with no downtime. That's the subject of the deploys lesson.

What breaks on server two

Here's the honest part. An app that worked perfectly on one server often misbehaves in strange ways on two. The bugs come from anything the app kept on the machine itself, assuming it would always be the same machine.

None of these are exotic. Every one of them will show up in Tiffin the week you add server two. That's not a reason to avoid scaling out. It's a reason to do it deliberately, knowing the list.

The database feels it too

Each app server opens its own connections to the database. One server with a pool of 20 connections is 20 connections. Six servers with 4 processes each and 20 connections per process is 480, and Postgres is configured for 100 by default. Scaling out the app tier moves pressure onto the database in ways that aren't obvious at first. Module 5 covers this, starting with connection pooling.

Up or out?

You don't really choose one forever. Most systems do both: they pick a sensible machine size and then run several of them.

For Tiffin, the plan is: go back to 8-core machines as the standard size, because several small servers follow the daily curve better than one big one. Run at least 2 servers at all times so a single failure isn't an outage, and scale to 5 or 6 at dinner.