Module 10 / Designing end to end

Module 10 · lesson 3 of 5 · Read 8 min

Common mistakes

Boxes without reasons, numbers without units, and designing for Google on day one.

Most bad designs aren't bad because someone didn't know about consistent hashing. They're bad because of a handful of habits that show up again and again, in design docs, in architecture reviews and in interviews, from juniors and seniors alike.

This lesson is a field guide to those habits. Each one has a quick way to spot it and a quick way to fix it. Once you've seen them, you'll catch them in your own work before anyone else does.

Boxes without reasons

What it looks like. A diagram with a load balancer, an API gateway, Kafka, Redis, Elasticsearch, three microservices and a CDN, for an app with 2,000 users. Ask "why is Kafka there?" and the answer is "for scalability" or "for decoupling".

Why it's a problem. Every box costs money, time and a new way to fail at 2am (Module 1). A box with no reason has all of the cost and none of the benefit. It also hides the real design: if everything is there "just in case", nobody can tell which parts are load-bearing.

The fix. Point at each box and finish this sentence: "This is here because [requirement or number]." If you can't, take it out. You can always add it when the reason arrives.

Customer app
→
Server
→
Database
For Tiffin's first city, this is the whole design. Every additional box in this course arrived because a specific problem asked for it.

Numbers without units

What it looks like. "We'll have about 10,000." Ten thousand what? Users in total? Users online at once? Requests per second? Per day? Each of those leads to a completely different design.

"10,000" meaning... What it implies
Registered users Tiny. A laptop could run it
Requests per day ~0.1 per second. Trivial
Requests per second Several servers, caching, careful database work
Writes per second Getting close to one database's limit
Open WebSocket connections Needs connection servers and pub/sub (Module 6)

The fix. Every number gets a unit and, ideally, a source. "About 4,600 requests per second at dinner peak: 250,000 orders a day, 60% of them between 7pm and 10pm, and about 330 requests for every order placed, counting customers who browse and don't order, kitchen tablets and riders." Now everyone can check it.

A close relative: mixing up average and peak. At 250,000 orders a day, Tiffin averages about 3 orders a second, but between 7pm and 10pm it runs at closer to 14. Designing for the average means falling over every evening.

Designing for Google on day one

What it looks like. A startup with no customers designs for a billion users: global multi-region, sharded databases, event sourcing, a service mesh. The design is impressive and the product ships nine months late, if at all.

Why it's a problem. You pay every cost of scale up front, while you have none of the benefits. You also make the system harder to change at exactly the moment you know least about what it needs to be. Most products change shape several times before they find users. A design that can't change easily is a liability.

The fix. Design for the numbers you have, plus about 10 times. Then make sure nothing you do now blocks the next step: stateless servers, a sensible primary key, a city_id on every table. Module 9's lab is this exact skill: growing in steps, with triggers.

How much is enough?

A founder says: "Our clinic booking app will be used by every clinic in India, so we need it to handle millions of bookings a second." Today they have 40 clinics. What would you design for, and what would you say?

Show one answer

Do the maths first. India has on the order of a million clinics and hospitals. Even if every one used the app and each took 100 bookings a day, that's 100 million bookings a day, or about 1,200 a second on average and maybe 4,000 at peak. Not millions a second, even in the dream scenario.

Today, 40 clinics x 100 bookings is 4,000 a day, a fraction of one request a second. Design for 10x to 100x that: one server, one database, backups, stateless servers so you can add more. Tell the founder the design can grow in clear steps when they get there, and that spending months on scale now means fewer months building what clinics actually want.

Skipping the requirements

What it looks like. Two minutes in, someone is drawing. Twenty minutes later, it turns out the system needed to work offline, or keep data in one country, or handle 10 times more writes than reads, and half the drawing is wrong.

The fix. Spend the first few minutes asking. What does it need to do? For whom? How many? How fast? What must never happen? Write the answers down where everyone can see them, and refer back to them.

The happy path only

What it looks like. The design explains exactly how an order flows when everything works. Nobody has asked what happens when the payment provider times out, the worker crashes halfway through, the cache is empty or the rider's phone loses signal.

Why it's a problem. In production, something is always failing a little. The parts of a system that hurt are almost never the happy path.

The fix. For every arrow in the diagram, ask: what if this is slow? What if it fails? What if it happens twice? You've met the answers throughout Part 1: timeouts, retries, dead letter queues, idempotent jobs, failover, rate limits. A design that doesn't use any of them probably hasn't looked at failure yet.

Buzzwords as answers

What it looks like. "We'll make it eventually consistent." "We'll use CQRS." "We'll add a circuit breaker." Said as if the word solves the problem.

The fix. Replace the word with what actually happens. "After a customer updates their address, the order page may show the old address for up to a second, because it reads from a replica. Checkout reads from the primary, so delivery always uses the new one." If you can't say what happens, you don't yet know whether it works.

A slogan is the same mistake in a different form. "It's distributed, so it's reliable" sounds right and isn't, by itself. Five services each 99.9% available, all needed for one request, give you about 99.5%, worse than any one of them. Distribution adds failure modes. Reliability comes from redundancy and from handling failure on purpose, which Module 12 and Module 18 cover.

Forgetting the data

What it looks like. Lots of boxes, no tables. Or a database box labelled "DB" with no idea what's in it or how it's queried.

Why it's a problem. Most performance and correctness problems live in the data: a missing index, a query that scans everything, a table nobody can partition, two places that both think they own the same field.

The fix. Always sketch the main tables or documents, their keys, and the two or three most important queries. Check that each query can use an index. Module 2's data modelling and indexes lessons apply to every design, at every size.

Treating a copy as the truth

What it looks like. Checkout reads the price from the cache. The search index decides whether a restaurant is open. A read replica is used to check whether a coupon has been used.

The fix. For every piece of data, know which store is the source of truth and which are copies (caches, replicas, search indexes). Copies are fine for showing things. Decisions that matter, like charging money, accepting an order or spending a coupon, read from the source of truth.

Review a design

A colleague's design for Tiffin's rider ratings: "Riders get rated after each delivery. We'll store ratings in Cassandra for scale, use Kafka to stream them to a Flink job that computes averages in real time, and serve averages from Redis." Tiffin does 250,000 deliveries a day and about 20% are rated. Which mistakes do you see?

Show one answer

Boxes without reasons: 50,000 ratings a day is under one write a second on average, and about 3 a second at the dinner peak. Cassandra, Kafka and Flink are each a cluster to run, for a load a single Postgres table handles without noticing.

Numbers without units: "for scale" has no number attached. The real number says the opposite.

Requirement skipped: does a rider's average need to update in real time? Probably not. A background job or even a nightly recompute is likely fine.

A better version: a rider_ratings table in the existing database, a summary updated by a background job, and the average cached with the rider's profile. Same features, three fewer systems.

What to remember

  1. Every box needs a reason you can say in one sentence. No reason, no box.
  2. Every number needs a unit and a source. Design for the peak, not the average.
  3. Design for today's numbers plus about 10x, and make sure nothing blocks the next step.
  4. Ask about requirements before drawing anything.
  5. For every arrow, ask what happens if it's slow, fails or happens twice.
  6. Replace buzzwords with a description of what actually happens.
  7. Sketch the data and its queries, and know which store is the source of truth.