Lab: From one city to ten

Tiffin's board has approved the plan: over the next two years, expand from one city to ten. You're the engineer asked to answer one question before the first new city launches: what has to change, in what order, and how will we know when?

Nobody wants a rewrite. Nobody wants an outage in month eight either. Your job is a growth plan with numbers and triggers, not a dream architecture.

The brief

This lab rewinds the clock, and its numbers are deliberately its own. Forget the multi-city, 200-million-row Tiffin from earlier in this module, and the 200,000-orders-a-day city of Modules 4 to 8: here Tiffin is a smaller, made-up version of itself, one city (Pune) at 50,000 orders a day with everything from Part 1 already in place, so you get to plan the growth before it hurts. Two things do carry over, so the arithmetic matches the rest of the course: requests per order (Modules 4 to 8 saw about 3,600 requests a second at 11 orders a second, about 330 per order at the peak) and Module 4's measured server capacity. Here's where this Tiffin stands today:

The plan for two years from now:

• 10 cities, including two very large ones.
• 400,000 orders a day in total.
• Riders ping their location every 4 seconds while online, as they do today.
• About 60 engineers.

Work through each step yourself before reading the reasoning.

Step 1: Project the numbers

Now put the numbers side by side, because the comparison is the point:

                       Today      In 2 years   Growth
Peak orders/s             3           22          7x
Peak requests/s       1,000        7,000          7x
Peak order writes/s      45          330          7x
Peak ping writes/s      500        3,750          7x

Everything grows about 7 times. But the absolute numbers tell you where the trouble is. 330 order writes a second is easy for one Postgres primary. 3,750 ping writes a second, more than ten times the order writes and every one of them an insert into an indexed table, is the thing that will actually hurt. The rider pings, not the orders, are Tiffin's biggest scaling problem.

Step 2: The app servers

This is the reward for the work in Module 4. Because no server holds state, growth in the app tier is a number in a config file. The interesting decisions are all about data.

Step 3: Get the pings out of Postgres

The pings are two different things wearing one name:

1. "Where is this rider right now?" Needed for live tracking and for matching riders to orders. Only the latest value matters, and it changes every 4 seconds.
2. "Where was this rider during order 81?" Needed for disputes and support. Read rarely. Kept for 30 days.

Today the latest location is already in Redis, but every ping is also inserted into the rider_locations history table in Postgres: 500 inserts a second. At 3,750, it would dominate the primary's write load, its write-ahead log, its replicas and its backups.

One ping now takes two cheap paths, and neither touches the primary:

%% caption: The latest position goes to Redis right away. History is batched into object storage.
sequenceDiagram
  autonumber
  participant RA as Rider app
  participant T as Tracking server
  participant R as Redis
  participant Q as Queue
  participant W as Batch worker
  RA-)T: ping, every 4 s
  T->>R: set latest location
  T->>Q: add to the history batch
  Note over Q: thousands of pings build up
  W->>Q: take a batch
  Note over W: one file per city and hour, into object storage

Do this first, before the second city launches. It's the change with the biggest effect, and it's easier to do while the numbers are small.

Step 4: One primary, or shards?

With pings gone, here's the primary's peak write load in two years:

Order writes      22 orders/s x 15 rows   =  330/s
Carts, ratings, payouts, sessions          ~  500/s
                                           ~  850 writes/s

Step 5: Data size

Order data grows with orders. At 400,000 orders a day and roughly 3 KB per order across orders, order_items, order_events and their indexes:

400,000 x 3 KB = 1.2 GB a day, about 440 GB a year

The database is 350 GB today. Without action it'll be well past 1 TB in two years, with backups and restores growing to match.

Step 6: Search and teams

Search. Ten cities means roughly 15,000 restaurants by the end of the plan, but today this Tiffin has one city's worth. Searches always include a city, so each one looks at a few thousand restaurants at most. Start with Postgres full-text and trigram indexes. Trigger to move to a dedicated search engine: "search p95 above 300 ms at dinner, or product asks for typo tolerance, autocomplete and facets together." With ten cities and search as the first thing every customer does, expect this within the first year. The search lesson's Tiffin is already past that point: 15,000 restaurants, and customers who expect typo tolerance and results as they type.

Teams. Going from 15 to 60 engineers is when one codebase starts to hurt. Start the modular monolith in the first few months, before the team passes about 25: owners per folder, public interfaces, one owner per table, boundaries enforced in the build. Search, if it gets its own engine, is a natural first separate service. Trigger for more: "teams regularly wait on each other's deploys."

The plan, in order

%% caption: Tiffin in ten cities. Still one primary. The biggest change is moving pings and old orders out of it.
flowchart TD
  CA([Customer app]) --> LB[Load balancer] --> AS["App servers<br/>autoscaled, modular"]:::hot
  RA([Rider app]) -- "pings, every 4 s" --> TS[Tracking servers]
  AS --> PG[("Postgres primary<br/>orders, monthly partitions")]:::store
  PG --> RR[(Read replicas)]:::store
  AS --> RD[("Redis<br/>cache, latest rider locations")]
  AS -.-> SI[("Search index<br/>when triggered")]
  TS --> RD
  TS --> Q[[Queue]] --> PA[("Ping archive<br/>object storage, 30 days")]:::hot
  PG -. "archive job, nightly" .-> OS[("Object storage<br/>orders over 12 months")]:::store

Build it on AWS

Tiffin runs in Mumbai (ap-south-1), and ten Indian cities don't need a second Region: every city is a short network hop from Mumbai. So the plan above maps onto the same AWS account, with three AWS-specific decisions: whether the database stays on RDS, how pings reach S3 without becoming a bill, and how archiving actually shrinks the database. Prices are us-east-1 list prices as of late 2026, fine for comparing options; Mumbai's differ, so check the AWS pricing calculator before quoting them.

The architecture

%% caption: Tiffin in ten cities on AWS. Still one writer. Pings and old orders go to S3 instead of the database.
flowchart TD
  CA([Customer app]) --> ALB[Application Load Balancer] --> APP["App service<br/>ECS on Fargate, autoscaled"]:::hot
  RA([Rider app]) -- "pings, every 4 s" --> TS["Tracking service<br/>ECS on Fargate"]
  APP --> AU[("Aurora PostgreSQL<br/>writer and 2 readers")]:::store
  APP --> VK[("ElastiCache Serverless for Valkey<br/>cache, latest locations")]
  TS --> VK
  TS -- "batched records" --> FH["Firehose<br/>one stream per city"]:::hot
  FH --> S3[("S3: pings, 30 days<br/>and archived orders")]:::store
  AU -. "nightly archive task" .-> S3

Argument 1: stay on RDS, or move to Aurora?

Until now Tiffin has run RDS for PostgreSQL: a Multi-AZ primary (with a standby you can't read from) and two read replicas, each with its own copy of the data on its own disk. This lab rewinds to a smaller Tiffin than Module 5's (one city at 50,000 orders a day), so the instances here are far smaller than the replicas lab's, but the decision is the same at either size. In two years that's about 600 GB of data, provisioned at 1 TB to leave room. Price both on the same instance size, r7g.2xlarge:

RDS for PostgreSQL
  Multi-AZ primary            $1.913/h x 730 h          ~ $1,397
  2 read replicas             2 x $0.956/h x 730 h       ~ $1,396
  Storage (gp3): Multi-AZ 1 TB at $0.23 + 2 x 1 TB at $0.115    ~ $460
                                                          ~ $3,250 a month

Aurora PostgreSQL (Standard)
  Writer + 2 readers          3 x $1.106/h x 730 h       ~ $2,422
  Storage: 600 GB used, stored once for the cluster, $0.10/GB    ~ $60
  I/O: $0.20 per million requests                         ~ depends
                                                          ~ $2,480 + I/O

Aurora is cheaper before I/O for a structural reason. Its readers are the failover targets, so there's no idle standby to pay for. And all three instances share one cluster volume, which keeps copies of the data across three Availability Zones, so you pay for 600 GB once, not three provisioned terabytes. The same design gives the operational wins Tiffin needs over the next two years:

• A new reader takes minutes, not hours. It attaches to the existing volume instead of copying 600 GB. That matters on the night a new city launches and reads double.
• Storage grows on its own, so nobody gets paged for a full disk.
• Failover is to a reader that's already warm, and AWS documents it as typically under a minute.

The catch is the I/O line. On Aurora Standard you pay per million I/O requests; on I/O-Optimized the instances cost more ($1.437 an hour for this size) and I/O is free. AWS's own guidance: once I/O is 25% or more of the Aurora bill, I/O-Optimized is cheaper. Here that's about $800 a month of I/O. Start on Standard, read the first month's bill, and switch if needed. You can move from Standard to I/O-Optimized once every 30 days.

The migration doesn't need downtime for copying. Create an Aurora read replica of the RDS instance, let it catch up until its lag is zero, then promote it to a standalone Aurora cluster and move the app's connection string. AWS recommends turning off automatic minor version upgrades on the RDS instance early, so it doesn't drift to a version Aurora doesn't support yet. Do it before city 2, alongside the other "while it's small" changes.

Argument 2: getting pings to S3 without paying for air

The plan says: batch pings and write them to cheap storage. On AWS, the least code is Amazon Data Firehose with Direct PUT. The tracking service sends records, Firehose buffers them for a few minutes and writes files to S3. Use one Firehose stream per city, each writing under its own S3 prefix, and Firehose adds the date and hour to every path. A dispute lookup becomes an Athena query over one city's hour.

The trap is how Firehose bills: per GB ingested, with each record rounded up to the nearest 5 KB. A ping is about 60 bytes.

One record per ping:    65 million records/day x 5 KB = 325 GB/day billed
                        ~ 9.75 TB/month x $0.029/GB                  ~ $283 a month
Batched records:        each task sends one record a second holding
                        all its pings: ~4 GB/day of real data         ~ $3.50 a month

Neither number will sink Tiffin, but one is 80 times the other for the same data, and it scales with every city you add. Batch in the tracking service, which is buffering anyway. Then add an S3 lifecycle rule to expire ping files after 30 days, so the "keep 30 days" promise is enforced by the bucket, not by a script someone has to remember.

Why not Kinesis Data Streams? It's the right tool when several consumers need the same stream, or need to replay it. Here there's one consumer, S3, so Firehose is the stream and the writer in one.

Argument 3: archiving that actually shrinks the bill

The nightly archive job is an ECS task. It detaches the partition that turned 12 months old, writes it to S3 as Parquet (one file per month, often 5 to 10 times smaller than the rows), checks the row counts match, and drops the partition. On Aurora, dropping a partition returns its storage: on current Aurora versions the cluster volume shrinks when data is removed, so the bill follows the archive down. On older Aurora versions, freed space was reused but never shrank, so check which version you're on.

"Load older orders" becomes an Athena query against the Parquet files. It takes a few seconds and costs a tiny fraction of a cent per lookup, which is fine for a button a few customers press a day.

What it costs

At the two-year target (400,000 orders a day across ten cities):

That's about 29 cents per thousand orders, before Aurora I/O. Nearly three quarters of it is the database, which is why the plan works so hard to keep pings and old orders out of it.

The same design on GCP and Azure

The shape is the same on every cloud: one writer, readers that share or stream from it, and a cheap pipe from the hot path into object storage.