A model knows a lot about the world, up to the day its training data was collected. It knows nothing about Tiffin's refund policy, last week's change to the cancellation window, or the help article about allergy labels. Ask it anyway and it will often make up a plausible answer. That's worse than no answer.
The fix is simple to say: before you ask the model, find the relevant pieces of your own documents and put them in the prompt. This is retrieval-augmented generation, or RAG. The model stops answering from memory and starts answering from text you handed it.
The idea fits in a sentence. Doing it well is mostly a search problem, and that's what this lesson is about. If you're arriving here directly: models read and bill in tokens, a page is roughly 500-800 tokens, and every token you add to a prompt costs money and latency.
Why not paste everything in?
Tiffin's help center has 400 articles averaging 1,200 tokens each. That's 480,000 tokens. Some models could fit it in their context window. But at an illustrative $3 per million input tokens, that's about $1.44 per question, plus several seconds of extra latency, and the answer quality drops when the relevant paragraph is buried in the middle of half a million tokens.
RAG picks the 5 most relevant passages of about 400 tokens each. That's 2,000 tokens, roughly 240 times less. The question becomes: how do you find those 5 passages, fast, out of thousands or millions?
Retrieval-augmented generation means searching your own data for the passages relevant to a question, and giving only those passages to the model as the material to answer from.