Chapter 10 · concept 6 of 6 · show all
2Tokens, cost and your first RAG
4Chunking
- 4.1Why we chunk, and the main chunking strategies
- 4.2Structure aware chunking and tables
- 4.3Chunk size, overlap and token based sizing
- 4.4Contextual chunking and late chunking
- 4.5Parent child and multi representation indexing
- 4.6Chunk metadata, provenance and the chunking debugging playbook
- Chapter overview
7Finding the right chunks
8When retrieval fails
Offline and online evals, regression tests and slices
Run the golden set in CI and block changes that lose answers; split results by case type so a weak slice cannot hide in the average; and watch live traffic for what the golden set misses.
A real question
A change raises the total score. Is it safe to ship?
Not yet. Check per question and per slice: did any question stop working, did any kind of question get worse? Then watch live traffic, because users ask things the golden set never imagined.
The problem
Why this matters
If the total went up, ship it.
Vector search scored 44 of 47 overall, which looks fine. By slice, conflict questions were 1 of 2 and plain answerable questions 15 of 17, while tables, versions and codes were perfect. The average hides where it breaks.
Why not the simpler way
A golden set used once is a report. Used on every change, it is a safety net. And the net has holes: real users ask in ways the set does not cover. Offline and online evaluation together catch most problems before or soon after users see them.
The idea
The idea in plain words
Offline evaluation runs fixed test cases on demand: fast, repeatable, cheap. Online evaluation observes real use: realistic, noisy, slower to read. Regression tests fail when something that worked stops working. Slices are subsets by case type, language, product or user role, reported separately.
Car makers run crash tests in the lab and also study real accidents. Each finds problems the other misses.
How it works
Step by step
1Golden set in CI
Run the retrieval checks on every pull request that touches chunking, embeddings, search or prompts. It takes seconds.
2Gate per question and slice
Fail the build if a question that passed now fails, or a slice drops below its bar. Print the diff.
- 3
Watch live traffic
Feedback buttons, sampled human reviews, abstention and flag rates, and new failure cases added back to the golden set.
In detail
CI job. Rebuild the index for the changed config, run the golden set, compare with the stored baseline per question. Retrieval checks need no model calls; answer checks can run nightly or before releases.
Bars. A minimum per slice (for example, every exact-code question in the top 3) and a rule that no passing question may start failing without an explicit accept.
Online signals. Thumbs down rate and reasons, abstention rate, verification flags (RAG-37), latency and cost per question, and a weekly human review of a random sample.
Closing the loop. A bad live answer becomes a golden item with its evidence (RAG-40).
The code
Try it in Python
import json, sys
def gate(baseline_path: str, current: dict[str, dict], k: int = 3, bars: dict[str, float] | None = None):
base = json.load(open(baseline_path))
hit = lambda r: r is not None and r <= k
broke = [q for q in base if hit(base[q]["rank"]) and not hit(current[q]["rank"])]
slices: dict[str, list[bool]] = {}
for q, r in current.items():
slices.setdefault(r["type"], []).append(hit(r["rank"]))
low = {s: sum(v) / len(v) for s, v in slices.items() if bars and sum(v) / len(v) < bars.get(s, 0)}
if broke or low:
print("broke:", broke, "slices below bar:", low)
sys.exit(1)
See it
What we measured
Measured · answer in the top 3 by slice
Polder corpus, 34 recursive 512 token chunks, 47 golden questions with an answer. Slices from the golden set's case_type.
Measured 2026-11-04 with corpus/scripts/measure-parts-4-7.mts. Slices with 1 to 4 questions are too small to trust alone; they flag where to look.
Try it on a real example
Vector search by slice, 47 golden questions, answer in the top 3
The slices we feared most were fine at 512 tokens.
Choosing
Use it when, avoid it when
- A CI gate on per-question regressions and slice bars
- Online feedback and sampling, fed back into the golden set
- Shipping on a higher total alone
- Online-only evaluation with no fixed set
What it costs · Retrieval checks in CI take seconds. Answer checks cost model calls; run them less often.
In production
Keep the baseline file in the repository next to the golden set, and update it only in a commit that explains why. Show the per-question diff in the pull request. For online evaluation, segment by the same slices as offline, so a drop in one product line is visible. Watch for drift: when the corpus changes, re-run the whole set, not just the affected questions.
Watch out
Common mistakes
- A higher total is a better system. One slice can still get worse.
- Small slices are noise. They are; they still point at where to look.
- Online feedback is enough. Most users do not click; sampling is needed.
Check yourself
Remember this
Remember these
- Offline: the golden set on every change, in CI.
- Gate on per-question regressions and slice bars, not on the total.
- Online: feedback, sampled reviews, abstention and flag rates.
- Every production failure becomes a new golden question.
Say it in an interview
- 01My golden set runs in CI; the gate fails on any question that stops passing.
- 02I report by slice: vector search was 44 of 47 overall but 1 of 2 on conflicts.
- 03Hybrid search fixed both weak slices, which the slice report showed directly.
- 04Online I track feedback, abstention, verification flags and sampled reviews.
- 05Every production failure becomes a new golden item with its evidence.
Quick quiz