Chapter 1 · concept 4 of 4 · show all
2Tokens, cost and your first RAG
4Chunking
- 4.1Why we chunk, and the main chunking strategies
- 4.2Structure aware chunking and tables
- 4.3Chunk size, overlap and token based sizing
- 4.4Contextual chunking and late chunking
- 4.5Parent child and multi representation indexing
- 4.6Chunk metadata, provenance and the chunking debugging playbook
- Chapter overview
7Finding the right chunks
8When retrieval fails
Use cases and the quality, latency and cost triangle
Where RAG fits and where it does not, and how every setting trades answer quality against speed and cost, with top k as the measured example.
A real question
If more chunks give the model more to work with, why not send twenty?
Every extra chunk costs money and time on every question. Past a point it buys nothing. You want the cheapest setting that is good enough.
The problem
Why this matters
More context is always safer, so turn every setting up.
On our corpus, gains stopped after 3 to 5 chunks. Top 3 held every answer; top 5 cost 60% more and found nothing new.
Why not the simpler way
The easy approach is to turn every setting up: more chunks, a bigger model, a reranker, several search passes. Each one can help quality. Each one also adds cost and waiting time, on every question, forever. Without numbers you end up paying for improvements you cannot see.
The idea
The idea in plain words
Every RAG system sits inside a triangle of quality, latency and cost. Moving towards one corner usually moves you away from another. The job is not to maximise quality but to find the cheapest, fastest setting that is good enough for the use case.
It is like choosing a delivery option. Same day, cheap and careful packaging rarely all come together. You decide which matters most for this parcel.
How it works
Step by step
1Write a target
For example: the answer passage found for 95% of test questions, under 2 seconds, under one cent a question.
2Move one lever at a time
Number of chunks, chunk size, the model, reranking. Change one, measure all three corners.
3Keep the cheapest that passes
Stop when the target is met. Anything more is cost without benefit.
In detail
RAG fits when answers live in documents: support over manuals, policy and compliance questions, internal knowledge, contract lookups. It fits badly when the answer needs a calculation over all records (use a database query), when there are no documents (brainstorming), or when one fixed answer is enough (a FAQ page).
The levers that move the triangle are chunk size, top k, the model, reranking and extra retrieval passes. Top k is the simplest to see: each extra chunk can hold the answer, and each one is paid for in tokens and read by the model.
The code
Try it in Python
def sweep_top_k(golden, retrieve, count_tokens, ks=(1, 2, 3, 5, 8)):
"""Print evidence recall and median prompt tokens for each top k."""
for k in ks:
hits, sizes = 0, []
for q in golden:
chunks = retrieve(q["question"], k)
hits += any(q["evidence"] in c for c in chunks)
sizes.append(sum(count_tokens(c) for c in chunks))
sizes.sort()
print(f"k={k} recall={hits / len(golden):.3f} median_tokens={sizes[len(sizes) // 2]}")
See it
What we measured
Measured · answer passage in context, and prompt size
Polder corpus, 47 golden questions, recursive 512 chunks, hybrid retrieval with RRF. A hit means one of the top k fused chunks holds the whole answer passage.
Measured 2026-11-04 with corpus/scripts/measure-parts-1-3.mts: each list's top 10 fused with RRF, first k kept, Mini RAG Lab prompt. Top 8 also reached 47 of 47 at 2,455 tokens. Going from top 1 to top 2 gave the biggest gain per token.
Try it on a real example
How many chunks should go into the prompt? (47 test questions)
Answer passage found: 33 of 47 · median prompt 379 tokens
Cheap, but 14 questions miss their answer.
Choosing
Use it when, avoid it when
- Questions answered by passages in documents
- A written target for quality, latency and cost before tuning
- Counts and sums over all records: query the database
- Turning every lever up without measuring what each one buys
What it costs · Input cost grows linearly with top k. Here, top 5 cost 55% more input than top 3 for one more answer passage out of 47.
In production
Write the target before tuning: for example "answer passage in context for 95% of golden questions, under 2 seconds, under $0.01 a question". Then change one lever at a time and keep the cheapest setting that meets all three. Re-check when traffic, corpus size or model prices change, because the best point moves.
Watch out
Common mistakes
- More context is always safer. Past top 5 here it added tokens and no hits.
- Quality is the only goal. A correct answer that takes 20 seconds or costs a euro may fail the use case.
- RAG fits every question about company data. Aggregate questions need a database.
Check yourself
Remember this
Remember these
- RAG fits answers that live in documents, not sums over a database.
- Every setting trades quality against latency and cost.
- Write the target before tuning, then change one lever at a time.
- Here, top 3 found all 47 answers; top 5 and top 8 added cost and nothing else.
Say it in an interview
- 01RAG is a trade between quality, latency and cost; I set a target for all three first.
- 02It fits answers that live in documents, not counts over records.
- 03On my corpus top 1 found 70% of answer passages, top 3 98% and top 5 100%.
- 04Top 5 cost 55% more input than top 3 for one more passage; top 8 bought nothing.
- 05I change one lever at a time and keep the cheapest setting that meets the target.
Quick quiz