Chapter 6 · concept 2 of 3 · show all
2Tokens, cost and your first RAG
4Chunking
- 4.1Why we chunk, and the main chunking strategies
- 4.2Structure aware chunking and tables
- 4.3Chunk size, overlap and token based sizing
- 4.4Contextual chunking and late chunking
- 4.5Parent child and multi representation indexing
- 4.6Chunk metadata, provenance and the chunking debugging playbook
- Chapter overview
7Finding the right chunks
8When retrieval fails
Score calibration and thresholds
Can a similarity cut-off tell "no good match" from a real answer? We scored 47 answerable and 15 unanswerable questions; the ranges overlap, so a threshold is one guard, not the guard.
A real question
The best chunk scored 0.84. Does that mean the answer is in it?
Not necessarily. On our corpus, a question with no answer scored as high as the best real one. A score threshold can catch questions that are far off topic, but not near misses.
The problem
Why this matters
Pick a cut-off like 0.7: above it the chunk is relevant, below it the system says “I don't know”.
Answerable questions scored 0.57 to 0.84. Questions with no answer scored 0.43 to 0.84. Any cut-off either loses good questions or lets bad ones through.
Why not the simpler way
A RAG system that always answers will make something up when the documents have nothing. The obvious fix is a score cut-off: if the best chunk is not similar enough, say "I don't know". It works for "what is the capital of Australia?" and fails for "does my home insurance cover my car?", which is the kind of question real users ask.
The idea
The idea in plain words
Calibration means a score you can read as a probability: 0.8 means right 80% of the time. Cosine similarity is not calibrated. Its range depends on the model (our multilingual e5 scored an unrelated pair 0.780; bge-small 0.391), on the corpus and on how the question is phrased. BM25 scores are not even bounded: our top BM25 scores ranged from 5.1 to 24.9 across questions.
A thermometer that reads 0.7 tells you nothing until you know whether it is in Celsius, Fahrenheit or a scale its maker invented.
How it works
Step by step
- 1
Collect both kinds
Use your golden questions, plus questions you know the corpus cannot answer: off topic and near topic.
2Look at the top score
For each question, record the best chunk's score. Plot the two groups and see where they overlap.
3Low cut-off, more checks
Set the cut-off to block only clear misses. Leave near misses to a reranker and a prompt that allows “I don't know”.
In detail
Measure, then choose. For each question, take the top score. Sweep cut-offs and count two things: answerable questions kept, and unanswerable ones blocked. Pick the cut-off by which mistake costs more in your product.
Better signals than one raw score. A cross-encoder reranker (RAG-31) reads question and chunk together, so its score is a closer look at the same pair; calibrate it on your golden set the same way before trusting it. We did not measure reranker cut-offs here. The gap between the first and second score is another signal. The strongest guard is the answer model itself, told to answer only from the context and to say when it cannot (RAG-34).
Per-model, per-corpus. A cut-off tuned for one embedding model is wrong for the next. Re-tune after every model or corpus change.
The code
Try it in Python
def sweep(top_answerable: list[float], top_unanswerable: list[float]):
for t in [0.50, 0.55, 0.60, 0.65, 0.70, 0.75]:
kept = sum(s >= t for s in top_answerable)
blocked = sum(s < t for s in top_unanswerable)
print(f"cut-off {t:.2f}: kept {kept}/{len(top_answerable)}, blocked {blocked}/{len(top_unanswerable)}")
# Choose the cut-off from this table, by which mistake your product can afford.
See it
What we measured
Measured · top cosine score per question, and a cut-off sweep
Polder corpus, 34 recursive 512 token chunks, bge-small. 47 answerable golden questions; 15 unanswerable: 5 from the golden set and 10 we wrote (5 off topic, 5 near topic).
Measured 2026-11-04 with corpus/scripts/measure-parts-4-7.mts. One unanswerable question (the duty manager escalation) has an answer in a restricted document; for a public user it counts as unanswerable.
Try it on a real example
Top similarity for questions the corpus cannot answer
All three fall below every answerable question (lowest 0.57). A cut-off catches these.
Choosing
Use it when, avoid it when
- A low cut-off to drop clearly off-topic questions
- Reranker scores, calibrated on your golden set, for finer decisions
- One fixed cut-off copied from a tutorial or another model
- A cut-off as the only way to say I don't know
What it costs · A sweep on the golden set takes minutes. Recalibrate after every model or corpus change.
In production
Log the top score of every query. A drop in the median after a release is an early warning that the corpus or the model changed. Track how often the system abstains: rising abstention can mean a missing document, falling abstention can mean a cut-off that was loosened by mistake. Keep near-topic unanswerable questions in the golden set so every change is tested against them.
Watch out
Common mistakes
- 0.7 is a standard threshold. There is no standard; it depends on the model and corpus.
- High score, answer present. The highest score of all here belonged to an unanswerable question.
- BM25 scores can be thresholded the same way. They are not bounded and vary from query to query.
Check yourself
Remember this
Remember these
- Raw scores are relative to the model, the corpus and the query.
- Measure both answerable and unanswerable questions before picking a cut-off.
- A low cut-off (0.60 here) blocked clear misses and kept 46 of 47 good questions.
- Near-topic questions need a reranker or the answer model to say “I don't know”.
Say it in an interview
- 01Cosine scores are not calibrated; their range depends on model, corpus and query.
- 02On my corpus answerable questions scored 0.57 to 0.84 and unanswerable 0.43 to 0.84.
- 03A 0.60 cut-off kept 46 of 47 good questions and blocked only the off-topic ones.
- 04Near-topic questions need a reranker or the model to abstain, not a stricter cut-off.
- 05I keep unanswerable questions in the golden set so every change is tested on them.
Quick quiz