Chapter 1 · concept 1 of 4 · show all
2Tokens, cost and your first RAG
4Chunking
- 4.1Why we chunk, and the main chunking strategies
- 4.2Structure aware chunking and tables
- 4.3Chunk size, overlap and token based sizing
- 4.4Contextual chunking and late chunking
- 4.5Parent child and multi representation indexing
- 4.6Chunk metadata, provenance and the chunking debugging playbook
- Chapter overview
7Finding the right chunks
8When retrieval fails
Why models need outside knowledge, and what hallucination means
What a language model knows and does not know, why it answers confidently anyway, and why giving it your documents is the fix.
A real question
Why does the AI give a confident answer about our own product, and get it wrong?
It has never seen your documents. When a model does not know something, it does not say so. It writes the answer that sounds most likely.
The problem
Why this matters
A big, modern model knows everything, including our prices and policies.
It knows public text up to a cutoff date. Your private, new or changed facts were never in it, so it guesses.
Why not the simpler way
The simple way is to ask a strong model directly. It answers quickly and sounds sure. For public, stable facts that often works. For your company's tariffs, policies and manuals it cannot work: those facts were never in its training data, or they changed after the model was trained. The model does not stop and say so. It writes the most plausible answer it can.
The idea
The idea in plain words
A language model learned patterns from a large amount of text, up to a cutoff date. It does not look anything up. When it lacks a fact, it still predicts fluent text, and that text can be wrong. A fluent answer that the facts do not support is called a hallucination.
Think of a well read new colleague on day one. They write well and know a lot in general, but not your price list. If you hand them the price list, they answer correctly and can point to the line. If you do not, a polite guess is the best they can do.
How it works
Step by step
1It learned from public text, once
A model is trained on a huge amount of public text, up to a date. After that, what it knows is fixed.
2Ask about your company, and it guesses
Your tariff sheet was never in that text. The model still answers, with a price that only sounds right.
3Hand it the page, and it reads
Put the real tariff sheet in the prompt. Now it answers from the page and can point to it.
In detail
A model's knowledge lives in its weights, fixed at training time. Three kinds of facts are missing: private facts (your documents), recent facts (anything after the cutoff) and changing facts (the tariff that was replaced last month). Fine tuning can teach style and some facts, but it is slow to update and gives no source to cite.
Retrieval augmented generation takes the other route: find the relevant passages at question time and put them in the prompt. The model then answers from text it can see, and the answer can cite where it came from. When nothing relevant is found, the right behaviour is to say so.
The code
Try it in Python
# Ask the same question with and without your documents (needs an API key).
import anthropic
client = anthropic.Anthropic()
question = "What does error E4012 mean on the Polder FiberBox?"
source = "| E4012 | Mesh pairing failed | Mesh extender on old firmware |"
for prompt in (question, f"Answer only from this source:\n{source}\n\nQuestion: {question}"):
msg = client.messages.create(model="claude-haiku-5-5", max_tokens=300,
messages=[{"role": "user", "content": prompt}])
# The reply can start with a thinking block, so print only the text blocks.
print("".join(b.text for b in msg.content if b.type == "text"), "\n---")
See it
What we measured
Measured · golden questions by type
The Polder golden set, version 1. Every fact in the corpus is fictional, so no model can know it from training.
Not measured yet. Planned: ask the 47 answerable questions with and without retrieved context, and count correct answers and invented answers. Needs a model call per question; planned with the precomputed answers.
Try it on a real example
A customer asks: “What does PT-MOB-25GB cost per month today?”
PT-MOB-25GB is a plan of Polder Telecom, a company invented for this course. It exists in no public text, so no model was ever trained on it.
Whatever price the model writes is a guess. It will not warn you; the sentence will look just as confident as a right answer.
No source, no way to check, and nothing that stops a confident wrong number.
Choosing
Use it when, avoid it when
- Your facts are private, recent or change often
- Answers must cite a source someone can check
- General knowledge questions a model already answers well
- Tasks that need a new skill or style rather than new facts: consider fine tuning
What it costs · Retrieval adds an index to build and keep current, and more input tokens per call. In return you can update facts by updating documents, not by retraining.
In production
Treat "I don't know based on these documents" as a correct answer and test for it. Include questions in your test set that the documents do not cover, and count how often the system abstains instead of inventing. Ask for citations in the answer and check that the cited passage actually supports it. A hallucination is easiest to catch when every claim must point to a source.
Watch out
Common mistakes
- Bigger models do not hallucinate. They know more, but they still cannot know facts that were never in their data.
- Hallucinations are rare bugs. They are the normal behaviour of a model asked about something it does not know.
- Telling the model "do not make things up" is enough. It helps, but only a source in the prompt gives it something true to say.
Check yourself
Remember this
Remember these
- A model only knows its training text, up to a cutoff date.
- Private, new and changed facts are missing, and the model guesses instead of saying so.
- A fluent answer that the facts do not support is called a hallucination.
- Giving the model the right document, with a citation, is the fix.
Say it in an interview
- 01A model knows only its training data up to a cutoff, and it never looks anything up.
- 02Private, recent and changing facts are missing, and the model guesses fluently instead.
- 03A hallucination is a fluent answer the facts do not support.
- 04RAG puts the relevant passages in the prompt so answers come from visible text with citations.
- 05I test refusals too: 5 of my 52 golden questions should not be answered.
Quick quiz