Chapter 2 · concept 2 of 2 · show all
2Tokens, cost and your first RAG
4Chunking
- 4.1Why we chunk, and the main chunking strategies
- 4.2Structure aware chunking and tables
- 4.3Chunk size, overlap and token based sizing
- 4.4Contextual chunking and late chunking
- 4.5Parent child and multi representation indexing
- 4.6Chunk metadata, provenance and the chunking debugging playbook
- Chapter overview
7Finding the right chunks
8When retrieval fails
The RAG lifecycle and your first RAG in miniature
The two halves of every RAG system, building the index and answering a question, shown end to end on a small corpus you can run in the browser.
A real question
What are the parts of a RAG system, and which one do I check when the answer is wrong?
Two halves. Offline you build an index of your documents; online you search it and answer. Check each half on its own.
The problem
Why this matters
RAG is one library call: give it documents and a question, and the answer comes out.
It is a pipeline of steps. When you can see each one, you can tell whether a wrong answer came from search or from the model.
Why not the simpler way
The simple way to try RAG is to install a framework, call one function and get an answer. It works, and you learn nothing about why it fails. When the answer is wrong you cannot tell if the document was never indexed, the right chunk was not retrieved, or the model ignored it. Seeing each step on its own is what makes RAG debuggable.
The idea
The idea in plain words
Every RAG system has two halves. Offline, you build an index: load the documents, cut them into chunks, turn each chunk into an embedding, and store both the vectors and a keyword index. Online, for each question, you search that index, put the best chunks into a prompt, and let the model answer from them.
It is a library with a good card catalogue. The catalogue is built before anyone walks in. When someone asks a question, the librarian looks it up, pulls three books, and answers from those pages.
How it works
Step by step
1Offline: build the index
Load the documents, cut them into chunks, turn each chunk into numbers, and store both the numbers and a keyword index.
2Online: search two ways
Keyword search matches the exact words. Vector search matches the meaning. Merging the two lists covers both.
3Answer from the top chunks
The best chunks go into the prompt, and the model answers from them with citations.
In detail
Retrieval in the lab is hybrid. BM25 scores chunks by shared words, so exact codes like E4012 match. Vector search scores chunks by meaning, so "my internet keeps dropping" can match a section titled "unstable connection". Reciprocal rank fusion merges the two lists by position, and the top 3 go into the prompt.
The prompt tells the model to answer only from the sources, cite them as [1], [2], and say it does not know when the sources do not cover the question.
The code
Try it in Python
# pip install rank_bm25 sentence-transformers
from rank_bm25 import BM25Okapi
from sentence_transformers import SentenceTransformer
chunks = ["E4012: mesh pairing failed. Update the extender to 2.8.1.", "..."]
bm25 = BM25Okapi([c.lower().split() for c in chunks])
model = SentenceTransformer("BAAI/bge-small-en-v1.5")
vecs = model.encode(chunks, normalize_embeddings=True)
PREFIX = "Represent this sentence for searching relevant passages: "
def retrieve(q: str, k: int = 3) -> list[str]:
by_kw = bm25.get_scores(q.lower().split()).argsort()[::-1][:10]
by_vec = (vecs @ model.encode(PREFIX + q, normalize_embeddings=True)).argsort()[::-1][:10]
score = {}
for ranked in (by_kw, by_vec): # reciprocal rank fusion, k = 60
for rank, i in enumerate(ranked):
score[i] = score.get(i, 0) + 1 / (60 + rank + 1)
return [chunks[i] for i in sorted(score, key=score.get, reverse=True)[:k]]
See it
What we measured
Measured · evidence recall@3
Polder corpus (25 documents), 47 golden questions, recursive 512 token chunks, bge-small-en-v1.5 vectors. A hit means a top 3 chunk holds the whole answer passage.
Measured 2026-11-03 with the Mini RAG Lab logic. On this corpus the questions reuse exact codes and names, so keyword search alone is best. Fusion still beats vector search alone. Expect a different order on paraphrased questions; measure on your own data.
Try it on a real example
A real question: “How much notice is given for a price change during a fixed term?”
The words “price change” and “fixed term” match exactly, so keyword search puts the answer first.
Choosing
Use it when, avoid it when
- Any question answering over your own documents that change or are too many to paste
- Hybrid retrieval as the default: keyword for exact codes, vectors for paraphrases
- A framework one-liner as your only view of the system: you cannot see which step failed
- Vector search alone when users type ids, codes or product names
What it costs · Offline: one embedding per chunk, repeated only for changed chunks. Online: one query embedding, two searches and one model call with about 1,100 input tokens per question here.
In production
Measure each half on its own. A golden set of questions with a known answer passage tells you whether retrieval found that passage, before any model is involved. Only then judge the answers. Most bad answers in RAG are retrieval misses, and you cannot fix those with a better prompt.
Keep the two halves loosely coupled. The index is rebuilt when documents change; the online path should not need a redeploy for that.
Watch out
Common mistakes
- Vector search is the modern way, so keyword search is obsolete. On this corpus BM25 alone found every answer passage.
- Fusion always wins. It was one question behind BM25 here. It is the safe default when you do not know your query mix.
- A wrong answer means the model is bad. Check retrieval first: was the answer passage in the prompt?
Check yourself
Remember this
Remember these
- Offline: load, chunk, embed and index. Online: search, pack the prompt, answer.
- Keyword search is strong on exact codes and terms; vector search on meaning.
- Merging both lists (hybrid search) is the safe default.
- Check retrieval on its own before you blame the model.
Say it in an interview
- 01RAG has an offline half that builds the index and an online half that answers.
- 02I measure retrieval on its own with evidence recall before judging answers.
- 03Hybrid retrieval: BM25 for exact codes, vectors for meaning, RRF to merge by rank.
- 04On my corpus BM25 found 47 of 47, vectors 44 and fusion 46, so I measure, not assume.
- 05Most bad RAG answers are retrieval misses, which a better prompt cannot fix.
Quick quiz