Scaling up

Tiffin's server hits its knee at 1,400 requests a second, and Friday dinner already needs 1,800. The fastest fix is also the most boring one: stop the server, move the app onto a machine with more CPU and memory, and start it again.

This is called scaling up, or vertical scaling. It gets laughed at in design discussions because it isn't clever. That's exactly why you should reach for it first. It takes an afternoon, it changes no code, and it goes much further than most people expect.

Scaling up means handling more load by giving one machine more resources: more CPU cores, more memory, faster disks, a faster network.

How big can one machine get?

Much bigger than the laptop you're reading this on. On a cloud provider, changing the machine size is a setting. Here's a rough ladder of general-purpose cloud machines:

The right column assumes the app uses all its cores well, and that throughput grows slightly less than linearly, which is normal. The biggest cloud machines today go past 400 vCPUs and many terabytes of memory, though those are built mostly for huge databases.

Within a machine family, cloud pricing is roughly linear: twice the cores costs about twice as much per hour. So scaling up isn't a cost trap at the sizes most apps need. A 16-core machine costs about the same as two 8-core ones.

Plenty of successful products ran this way for years. Stack Overflow famously served a top-50 website from a small number of web servers and a pair of very large database machines, long after most people assumed they'd need hundreds of servers.

Why scale up first

The case for the bigger machine is really a case about everything you don't have to do:

• No code changes. Sessions in memory, uploaded files on local disk, a cron job inside the app: all of it keeps working, because there is still only one server. The next lessons show how much of that breaks on server two.
• No new parts. No load balancer to configure, no new failure modes, nothing new to monitor.
• Fast. Resize the machine, restart, done. Most clouds do it in a few minutes.
• Easy to undo. If it doesn't help, you've learnt that your bottleneck is somewhere else, and you can size back down.

For Tiffin, going from 8 to 16 cores nearly doubles capacity, to around 2,600 req/s. With Friday peaks at 1,800, that's 69% of the knee, just inside the healthy range. Traffic is growing about 10% a month, so this buys a few months. The second exercise below works out exactly how many, and the answer is fewer than you'd hope.

That's real time. It's time to fix caching, clean up the slow menu code, and get the app ready to run on several servers without panic.

Make sure the app can use the cores

There's a catch that bites a lot of teams. A bigger machine only helps if your program can use it.

Say Tiffin's backend is Node.js running as a single process. On the 8-core machine it was already using one core flat out while seven sat idle. Moving to 16 cores would change nothing. Running 8 processes on the current machine might give you almost 8 times the capacity for free.

So before you resize, look at the per-core CPU graph from the first lesson of this module. If one core is pinned and the rest are bored, the fix is configuration, not hardware.

Even with many processes, gains are rarely perfectly linear. Shared locks, a single shared cache in memory, garbage collection and memory bandwidth all mean that 16 cores usually give you something like 1.7 to 1.9 times what 8 gave, not exactly 2. That's why the table says about 2,600 for 16 cores, not 2,800.

Where scaling up stops

Scaling up has real limits. None of them hit on day one, but you should know them.

There's a ceiling

At some point there is no bigger machine. Long before that, the biggest machines get pricier per core, harder to get in every region, and slow to start. If your growth curve points past the biggest box, you need another plan.

One machine is one point of failure

This is the limit that usually matters first. With one server, any of these takes Tiffin completely offline:

• the hardware fails, or the cloud provider retires the host,
• a bad deploy crashes the process,
• the OS needs a security patch and a reboot,
• someone fills the disk with logs.

A bigger machine doesn't make any of this less likely. If anything it makes each failure more expensive, because more customers are on it. A restaurant-facing product that goes dark for 20 minutes at 8pm loses real orders and real trust.

Resizing usually means a restart

On most clouds, changing the machine size means stopping it. A few minutes of downtime at 4am is fine for Tiffin today. It won't be fine forever.

Bigger machines don't follow traffic

Tiffin's traffic is spiky. It's quiet at 4pm, busy at 1pm, and frantic at 8pm. One big machine is sized for the 8pm peak and sits mostly idle the other 21 hours of the day. With several smaller machines, you can add some for dinner and remove them after. That's the subject of the next lesson.

The decision for Tiffin

Putting it together:

The team does three things this week:

1. Confirm the app uses all cores (it does: the Node.js app was already running one process per core).
2. Resize the server from 8 to 16 cores at 4am on a Tuesday, when there are almost no orders.
3. Start preparing to run on more than one server, because a single machine is still a single point of failure, and the next growth jump will come.