In the first lesson of this module, Tiffin's support assistant came out at about $9,600 a day on a large model: 800,000 calls, each with 3,000 input tokens and 200 output tokens. Nobody approves $290,000 a month for a chat widget without asking hard questions.
The good news is that LLM costs respond very well to ordinary engineering. Most of the bill is the same tokens sent again and again, or a big model doing a small model's job. This lesson covers the three biggest levers - prompt caching, response caching and model routing - and the maths to check each one. You've met caching before (Module 3 covers cache-aside, TTLs and invalidation). The same ideas apply here, with one new twist: a cache that matches on meaning instead of exact keys.
All prices in this lesson are illustrative. Check your provider's current rates before you rely on any number.
Where the money goes
The whole cost model fits on one line:
cost per day = calls x (input tokens x input price + output tokens x output price)
Every lever attacks one of those terms:
| Lever | Term it reduces |
|---|---|
| Prompt caching | Input price, for the repeated part of the prompt |
| Response caching | Calls, by skipping the model entirely |
| Model routing | Both prices, by using a cheaper model |
| Shorter prompts, fewer retrieved chunks | Input tokens |
Shorter answers, a max_output_tokens cap |
Output tokens |
Before optimising, measure. Log the input and output token counts for every call, tagged by feature and prompt version. You'll usually find one feature, or one bloated prompt, is most of the bill.