Calling a model is one line of code. Being the company on the other end of that line is a different job. Every message a user sends becomes a few seconds of work on an accelerator that costs dollars an hour, the answer has to stream out one token at a time, and the hardware is so scarce that you can't just add servers when a launch goes viral.
Module 24 treated the model as a remote dependency: LLMs for system designers covered tokens, time to first token and cost from the caller's side, and Caching and cost covered how to spend less on it. This lab is the provider's side. We'll design the system that serves a chat assistant to millions of people from a fleet of GPUs.
All hardware and model numbers in this lab are illustrative. Real figures vary a lot between models, accelerators and serving software. The shape of the reasoning is what carries over.
The brief
You're building the serving platform for a chat assistant like ChatGPT. A user types a message in a conversation, and the answer streams back word by word.
Who uses it?
- Free users, a large share of traffic, with tight usage limits.
- Paid users, who expect fast answers even at the busiest hour and pay for higher limits.
- Internal batch jobs, like summarising conversation titles or running evals, which can wait.
In scope: sending a message and streaming the answer, conversation history, choosing between a large and a small model, usage limits per plan, stopping and resuming a stream, safety checks on input and output, and capacity and cost.
Out of scope: training the model, retrieval over documents (see the RAG lab), tool use and agents (see Agent loops), image input, and billing and payments.
Requirements
Functional
- A user sends a message in a conversation and sees the answer stream back as it's generated.
- The user can stop an answer midway, and keep what was generated so far.
- If the connection drops, the app reconnects and the answer carries on from where it was.
- Users see and reopen their past conversations.
- Each plan has usage limits, in tokens per minute and per day.
- Paid users can pick the large model. Free users get the large model when there's room and the small model when there isn't.
Non-functional
- Fast start: time to first token under 1 second at p95 for paid users, under 3 seconds for free users, at normal peak.
- Readable speed: at least 20 tokens a second per stream. People read at roughly 5-10 tokens a second, so faster than that feels instant.
- Fair: one heavy user, or one huge prompt, can't slow everyone else down.
- Available: 99.9% of messages get an answer. When capacity runs short, free users degrade before paid users fail.
- Durable: a finished answer is never lost from the conversation history.
- Affordable: cost per million tokens is tracked and kept within a target.
Notice the shape. The interesting resource isn't requests or bytes. It's accelerator memory and accelerator time, and almost every decision in this design is about sharing them well.