Finding the limit

Two years after launch, Tiffin is doing well. Orders have grown from 50,000 a day to 200,000. Menus now come from Redis (Module 3), but the app itself still runs on one server with 8 CPU cores and 16 GB of memory, talking to one database on a separate machine.

Last Friday at 8:10pm, the app slowed to a crawl. Menus took 4 seconds to load, some checkouts failed, and the support team got 300 angry messages in 20 minutes. By 9pm everything was fine again.

Before you buy a bigger machine or add more servers, you need two answers. How much traffic can this server actually take? And what runs out first? Guessing here is expensive. If the database is the real problem, ten more app servers won't help at all. This lesson is about finding out.

The four vitals

A doctor doesn't start with surgery. They check pulse, blood pressure, temperature and breathing. A server has its own four vitals:

Google's SRE book calls these the four golden signals, and they are the first dashboard you should build for any service. Each one alone can mislead. High CPU with fast responses is fine. Low CPU with slow responses means the server is waiting on something else. You need all four together.

Averages lie, percentiles don't

Say 99 menu requests take 50 ms and one takes 5 seconds. The average is about 100 ms, which looks healthy. But one customer in a hundred waited 5 seconds.

So measure latency in percentiles. The p50 (median) is what a typical request sees. The p99 is the time that 99% of requests beat, so 1 in 100 is slower.

The p99 matters more than it sounds. Opening the Tiffin home screen fires about 30 requests. If each one has a 1% chance of being slow, the chance that at least one of them is slow is:

1 - 0.99^30 = 1 - 0.74 = 0.26

About one home screen in four hits a p99 request. Your "rare" slow case is something most customers see every week.

Load testing: break it on purpose

Waiting for next Friday to learn your limit is a bad plan. Instead you run a load test: you send fake but realistic traffic at a copy of the system and keep raising it until something gives.

Here's a small test written for k6, a popular open-source load testing tool:

import http from 'k6/http';
import { sleep } from 'k6';

export const options = {
  stages: [
    { duration: '5m', target: 400 },   // warm up
    { duration: '10m', target: 1600 }, // climb slowly
    { duration: '10m', target: 3200 }, // push past the limit
    { duration: '5m', target: 0 },     // cool down
  ],
  thresholds: { http_req_duration: ['p(99)<800'] },
};

export default function () {
  const r = Math.random();
  if (r < 0.6) http.get(`${__ENV.BASE}/restaurants/42/menu`);
  else if (r < 0.85) http.get(`${__ENV.BASE}/restaurants?area=koramangala`);
  else if (r < 0.95) http.get(`${__ENV.BASE}/orders/981/status`);
  else http.post(`${__ENV.BASE}/orders`, JSON.stringify({ cart_id: 77 }));
  sleep(1);
}

Each target is a number of virtual users: simulated customers, each running the function at the bottom in a loop, so each one sends about one request a second, as long as the server answers quickly (the next lesson shows why that caveat matters). The test climbs from 400 of them to 3,200 over 25 minutes.

A few rules make the difference between a useful test and a misleading one:

• Use a realistic mix. Real Tiffin traffic is mostly menu reads with a few order writes. A test that only hits the cheapest endpoint will report a limit you will never see in production.
• Use production-sized data. A query on a 1,000-row test table is instant. The same query on 40 million orders may not be.
• Ramp slowly. You want to see where things bend, not just that they broke.
• Watch the load generator too. If your laptop's CPU is at 100%, you're measuring your laptop. Run the test from a separate machine that is big enough.
• Test a copy, not production, unless you really know what you are doing. A load test that places 50,000 fake orders into real restaurant tablets is a memorable mistake.

Reading the results

Here's what the test on a copy of Tiffin's server showed:

Look at how the numbers behave. From 400 to 1,200 req/s, latency barely moves while CPU climbs steadily. Then, between 1,400 and 1,800, latency explodes even though traffic only rose by a quarter. That bend in the curve is called the knee.

A server's real capacity is not the load where it falls over. It's the load just before the knee, where latency is still acceptable.

For Tiffin, that's about 1,400 req/s. Friday's dinner peak was 1,800. That's the whole outage, explained in one table.

(Module 2 estimated that an 8-core server could do about 1,600 simple requests a second. The test roughly agrees: CPU is nearly maxed out at 1,800. But the usable capacity, before latency explodes, is lower than the raw estimate. That's why you measure.)

Why latency explodes near the limit

Think of a supermarket checkout. If the cashier is busy half the time, you rarely wait. If they're busy 95% of the time, a queue builds the moment two people arrive close together, and it never fully clears.

Servers behave the same way. A simple queueing model says that if a request takes time S to serve and the server is busy a fraction u of the time, the average response time is roughly:

response time ≈ S / (1 - u)

u = 50%  ->  2 x S
u = 80%  ->  5 x S
u = 90%  -> 10 x S
u = 95%  -> 20 x S

Real servers are messier than the model, but the shape is right. The last 10% of capacity is where all the pain lives. That's why teams plan to run at around 60-70% of capacity at peak, and why "the CPU is only at 85%" is not good news.

What runs out first?

CPU is only one resource. Here's how to read the common patterns:

That last row is the tricky one, and it's worth a closer look.

Little's law

There's a simple rule that links the numbers you see:

requests in flight = requests per second x seconds per request

At 1,600 req/s and 50 ms per request, Tiffin has 1,600 x 0.05 = 80 requests in progress at any moment. Easy.

Now say the payment provider slows down and checkout calls take 2 seconds instead of 200 ms. Even if only 5% of traffic is checkout, that's 80 req/s x 2 s = 160 requests stuck waiting on payments, up from 16. If the server can only work on 100 requests at once (because of its worker-thread limit, or a cap on open connections), every slot ends up waiting on payments, and menu requests queue behind them. The CPU sits at 30% while customers stare at spinners.

So when latency rises without CPU rising, look for the thing everyone is waiting on. Very often, as you'll see in Module 5, it's the database.

Back to Tiffin

The load test and the dashboards tell a clear story:

• The app server is CPU-bound. Profiling shows most time goes on turning data into JSON responses and checking login tokens.
• Its knee is around 1,400 req/s. Friday's peak was 1,800 req/s.
• The database was at 40% CPU during the test, so it's not the limit yet.

Caching (Module 3) took the menu work off the database, but the app server still does real work for every request, and traffic is still growing about 10% a month. You need more app capacity either way. There are two ways to get it: a bigger server, or more servers. First, the next lesson runs a real load test and reads its output line by line. The two after it cover each way to grow.