RAG in one page
The model sounded sure. It was wrong.
Ask a chatbot about your own insurance policy and it will guess. RAG stops the guessing: find the right page first, then answer from it, and show where the answer came from. Here is the whole idea, step by step, on real documents.
Try it first: search our documents
This runs in your browser on the 34 chunks the course measures. Pick a question and see which search finds the answer.
- Loading 34 chunks…
Step 1 of 6 · Load
Collect what the model does not know
A model only knows what it was trained on. Your price list, your manual and last week's outage are not in there. So the first step is to gather your own documents and keep track of which version is current.
Measured · Our test set: 25 documents from a telecom, a bank and an insurer.
Where it breaks: An old version left in the index will be found and quoted as if it were current.
Part 2: getting documents in →Step 2 of 6 · Chunk
Cut documents into pieces
Whole documents are too long to search well and too expensive to send. So we cut them into chunks of a few hundred words, ideally along headings and paragraphs, so each piece makes sense on its own.
Measured · Cutting by structure found the answer for 0.936 of questions; small fixed cuts only 0.766.
Where it breaks: A cut through the middle of an answer cannot be fixed later by a better prompt.
Part 3: chunking →Step 3 of 6 · Embed
Turn each chunk into numbers
An embedding model turns text into a list of numbers, so texts about the same thing end up close together. That lets a computer find a chunk that means the same as the question, even with different words.
Measured · Each chunk becomes 384 numbers. “Reset the router” and “Do not reset the router” still score 0.939 alike.
Where it breaks: Embeddings capture the topic, not every word: negation, exact codes and numbers slip through.
Part 4: embeddings →Step 4 of 6 · Search
Find the right pieces
When a question arrives, two searches run: one on shared words, one on meaning. Each is good at what the other misses. Merging their rankings gives a list that rarely loses the answer.
Measured · Combined search put the answer in the top 3 for all 47 test questions.
Where it breaks: Search by meaning alone missed 3 of 47, including the storm question above.
Part 5: retrieval →Step 5 of 6 · Pack
Build the prompt
The best few chunks go into the prompt, numbered as sources, with one instruction: answer only from these, and cite them. Fewer, better chunks beat many: every extra chunk costs money and gives the model more to wade through.
Measured · The top 3 chunks held every answer in a median of 1,078 tokens. Top 5 cost 60% more and added nothing.
Where it breaks: Too many chunks, or the best one buried in the middle, and the model can miss what is there.
Part 6: context →Step 6 of 6 · Answer
Answer with sources, or say “I don't know”
The model writes the answer from the sources and points to them. Then a cheap check compares the numbers and codes in the answer with the sources, and if nothing relevant was found, the right answer is “I don't know”.
Measured · A simple number check caught 28 of 39 answers where we had changed one number.
Where it breaks: Sources do not force the model to use them. Checks and measurements do.
Part 6: answers you can trust →Where it really goes wrong
Three things we found by measuring
Our own bug, found by measuring
One chunk was 520 tokens, but the embedding model reads only 512. The end of the chunk, holding two answers, was silently dropped. No error, just two questions that could never be found.
See how we fixed it →
A confident score means little
A question the documents cannot answer scored 0.84, higher than many real answers. A score threshold alone cannot tell them apart.
See how we fixed it →
Documents can attack you
We planted one document with hidden instructions. It reached the top 3 for 3 of 62 questions, one of them unrelated to its topic.
See how we fixed it →
You made it
You can already explain RAG to someone else.
At the top of this page RAG was a buzzword. Now you know the six steps, and you know where each one fails. Knowing where it fails is what real systems need.
- Why a model needs your documents, and what goes wrong without them
- Why documents are cut into chunks, and how a bad cut loses an answer
- What an embedding is, and what it cannot see
- Why keyword search and search by meaning work best together
- How a prompt is built from sources, and why fewer is often better
- How to check an answer instead of trusting it
Where to go next
54 pages and 8 labs, all measured on the same documents. Pick the path that fits you; you can switch any time.
New to all this
Start at Part 1 and read in order. Each page has a short story, the details, and a quiz at the end.
Begin Part 1Building a RAG system
Jump to the step that hurts. If the right chunk never reaches the prompt, no model can fix it, so Part 5 on retrieval is a good first stop.
Go to retrievalPreparing for interviews
Every page ends with the lines to say in an interview, backed by a measured number. Start with the architecture choices.
Choose an architecture