Chapter 7 · concept 4 of 6 · show all
2Tokens, cost and your first RAG
4Chunking
- 4.1Why we chunk, and the main chunking strategies
- 4.2Structure aware chunking and tables
- 4.3Chunk size, overlap and token based sizing
- 4.4Contextual chunking and late chunking
- 4.5Parent child and multi representation indexing
- 4.6Chunk metadata, provenance and the chunking debugging playbook
- Chapter overview
7Finding the right chunks
8When retrieval fails
topK, no good match, and metadata filtering
How many chunks to pass on, what to do when none fit, and how filters on version, date and access remove wrong candidates before ranking. Measured recall and token cost per k.
A real question
Should the model get 3 chunks, 10 chunks, or none at all?
Enough to hold the answer and no more. Measure recall at each k, filter out wrong versions and forbidden documents first, and let the system say when nothing fits.
The problem
Why this matters
Send more chunks; the model will find the answer somewhere.
With hybrid search, the top 3 already held all 47 answers in 884 tokens. The top 10 held the same 47 in 2,704 tokens: three times the cost, plus more distractors for the model.
Why not the simpler way
k is often set once, to 5 or 10, and never revisited. Too small and answers are missed; too large and every question costs more, takes longer, and gives the model more wrong text to be confused by (RAG-35). Meanwhile, chunks from last year's price list or a restricted runbook compete for the same slots.
The idea
The idea in plain words
Three decisions sit between search and the prompt. How many chunks to keep (k). Which chunks are allowed at all (metadata filters). Whether any chunk is good enough (no good match).
Packing for a trip: decide how many bags you can carry, leave behind what is not allowed on the plane, and if nothing fits the trip, do not go.
How it works
Step by step
1Measure recall at each k
Count how often the answer is in the top 1, 3, 5 and 10. Pick the smallest k where the curve flattens.
2Filter first
Drop chunks from the wrong version, date, language or access role before ranking, so they cannot take a slot.
3Allow no good match
If nothing passes, pass nothing and say so. A low score cut-off and a prompt that allows “I don't know” work together.
In detail
k from data. With vector search alone, our recall at 1, 3, 5 and 10 was 0.638, 0.936, 0.979 and 1.000; with hybrid it was 33, 47, 47 and 47 of 47. A reranker (RAG-31) lets you retrieve 10 and pass 3.
Filters. Version validity (effective_from and effective_to), language, product line, and access roles are metadata on each chunk (RAG-18). In RAG-11, a validity date filter raised the right version at rank 1 from 2 of 5 to 4 of 5 questions. Pre-filtering is exact; filtering inside an ANN index can lower recall when the filter is very narrow (RAG-22).
No good match. A low similarity cut-off blocks off-topic questions (RAG-23). For near-topic questions, the model must be allowed to answer "I don't know based on these documents" (RAG-34).
The code
Try it in Python
from datetime import date
def candidates(chunks, user_roles: set[str], on: date, lang: str | None = None):
for c in chunks:
m = c["meta"]
if m["allowed_roles"] and not user_roles & set(m["allowed_roles"]):
continue # access first, always
if m["effective_from"] > on or (m["effective_to"] and m["effective_to"] < on):
continue # wrong version for this date
if lang and m["language"] != lang:
continue
yield c
def retrieve(question, chunks, user, k=3, min_score=0.60):
pool = list(candidates(chunks, user.roles, date.today()))
ranked = hybrid_search(question, pool)
return [c for c in ranked[:k] if c.vector_score >= min_score] # may be empty: say so
See it
What we measured
Measured · answers in the top k, and context size
Polder corpus, 34 recursive 512 token chunks, 47 golden questions, hybrid search (BM25 + bge-small, RRF). Tokens counted with our tokenizer over the chunk texts.
Measured 2026-11-04 with corpus/scripts/measure-parts-4-7.mts; version filter measured 2026-11-03. Token counts are averages per question.
Try it on a real example
Hybrid search on 47 questions: what each k costs and finds
Too few: the right chunk is often second or third.
Choosing
Use it when, avoid it when
- k from a recall curve, often 3 to 5 after a reranker
- Filters on version, date, language and access before ranking
- Large k as a safety margin
- Access checks applied only after ranking or in the prompt
What it costs · Each extra chunk adds about 300 tokens per question on our corpus. Filters cost almost nothing.
In production
Log k, the filters applied and the number of candidates left after filtering for every query. A filter that leaves zero candidates is a bug or a missing document, and should be visible, not silently turned into an empty context. Access filters must use the user's identity from the session, never a value the user can type (RAG-46).
Watch out
Common mistakes
- More context is safer. It costs more and gives the model more to get wrong.
- Filters lower recall. Wrong-version chunks were never the answer.
- An empty result is an error. It is the honest answer when nothing fits.
Check yourself
Remember this
Remember these
- Choose k from a recall curve on your golden set.
- On our corpus k = 3 held all 47 answers with hybrid search.
- Filter by version, date and access before ranking.
- When nothing fits, pass nothing and let the system say so.
Say it in an interview
- 01I pick k from a recall curve; on my corpus hybrid search hit 47 of 47 at k = 3.
- 02Going from k = 3 to 10 tripled tokens, 884 to 2,704, for no new answers.
- 03I filter by version, date and access before ranking so wrong chunks never take a slot.
- 04A validity filter raised the right version first from 2 to 4 of 5 questions.
- 05When nothing passes, the system says it does not know instead of guessing.
Quick quiz