Chapter 10 · concept 1 of 6 · show all

Why measure, and what to measure

Without a fixed test, every change is a guess. What to measure at each stage of a RAG system, and three changes from this course where the obvious number told the wrong story.

A real question

You changed the chunk size and the answers feel better. Are they?

You cannot tell by feel. A fixed set of questions with known answers, run before and after every change, turns “feels better” into a number, and shows which questions got worse.

The problem

Why this matters

The belief

Try a few questions by hand; if they look good, the change is good.

What really happens

In this course, reranking left recall@3 at 47 of 47 but moved 8 answers up to rank 1. Diversity re-ranking looked harmless and lost 3 answers. Only a fixed test, read per question, showed either.

Why not the simpler way

RAG systems have dozens of settings and every one interacts with the others. Teams that do not measure end up tuning by anecdote: the CEO's favourite question, the last bug report. Changes that fix one case silently break three others. Measurement is what lets you change things with confidence.

The idea

The idea in plain words

What to measure, by stage:

Retrieval: is the evidence in the top k (recall@k), how high (MRR, nDCG), how much noise (precision) (RAG-41).
Generation: is the answer correct (matches the golden answer), faithful (supported by the context), cited, and does it abstain when it should (RAG-42).
System: latency per stage, tokens and cost per question, error and abstention rates.

A doctor does not ask "do you feel better?" only. They also measure temperature and blood pressure, the same way each time.

How it works

Step by step

1
Fix the test
A golden set: questions with a known answer and the exact evidence sentence (RAG-40). The same set, every time.
2
Measure each stage
Retrieval: was the evidence found, at what rank? Generation: is the answer correct, grounded and cited? Plus cost and latency.
3
Run it on every change
Before and after, per question. A total hides questions that broke while others improved.

In detail

Offline evaluation runs the golden set on demand and in CI (RAG-44). Online evaluation watches real traffic: feedback, sampled reviews, abstention and flag rates.

Read per question. Store each question's ranks and verdicts for every run. A diff between runs lists questions that improved and questions that broke.

Noise. Rounding, sampling temperature and small sets move numbers. Re-run the baseline twice to see how much a "change" can move with nothing changed.

The code

Try it in Python

def compare(before: dict[str, int | None], after: dict[str, int | None], k: int = 3):
    """before/after: question id -> rank of the evidence chunk (None if not found)."""
    hit = lambda r: r is not None and r <= k
    fixed = [q for q in before if not hit(before[q]) and hit(after[q])]
    broke = [q for q in before if hit(before[q]) and not hit(after[q])]
    moved_up = [q for q in before if before[q] and after[q] and after[q] < before[q]]
    print(f"top-{k}: {sum(map(hit, before.values()))} -> {sum(map(hit, after.values()))}")
    print(f"fixed {fixed}  broke {broke}  moved up {len(moved_up)}")

See it

What we measured

Measured · changes from this course, by two different numbers

Polder corpus, 34 recursive 512 token chunks, 47 golden questions. Numbers from RAG-21, RAG-30 and RAG-31.

Reranking hybrid top 10: recall@34747

Reranking hybrid top 10: answer first3340

MMR: distinct files in top 53.74.1

MMR: answers in top 34441

bge-small re-run, nothing changed4443

Measured 2026-11-04 with corpus/scripts/measure-parts-4-7.mts.

Try it on a real example

Three changes from this course, judged by the number you might check first

Change

Hybrid top 10, reranked

recall@3: 47 → 47 · answers at rank 1: 33 → 40

The first number says “no effect”. The second says it was the biggest gain in Part 5.

Choosing

Use it when, avoid it when

Use it when

A golden set from day one, run on every change
Separate numbers for retrieval, generation, cost and latency

Avoid it when

Judging changes by a few hand-picked questions
Reading one total without the per-question diff

What it costs · Writing 30 to 50 questions takes a day. Retrieval runs take seconds; answer runs cost model calls.

In production

Start measuring on day one with 30 to 50 questions; a small honest set beats none. Make the run cheap and fast so people use it: our retrieval run takes seconds. Answer-level checks need model calls; run those before releases, and retrieval checks on every commit.

Watch out

Common mistakes

One score tells you if it is better. Recall@3 hid the biggest gain in Part 5.
Evaluation comes at the end. It is what makes every earlier step safe to change.
A one-question change is real. It can be rounding; re-run the baseline.

Check yourself

Remember this

Remember these

A fixed golden set turns opinions into comparisons.
Measure retrieval and generation separately, plus cost and latency.
Pick metrics that can show the change you expect: rank 1, not only top 3.
Compare per question; small sets have noise of a question or two.

Say it in an interview

01I run a fixed golden set before and after every change and compare per question.
02I measure retrieval, generation, cost and latency separately.
03Reranking left recall@3 unchanged but moved 8 answers to rank 1; the metric has to fit the change.
04Diversity improved a variety number and lost 3 answers; I watch the number users feel.
05With 47 questions, I treat one-question differences as possible noise.

Quick quiz

Three questions

Reranking the hybrid top 10 kept recall@3 at 47 of 47. Did it help?

Why measure retrieval and answers separately?

Q3 · from RAG-05

What are the stages of a RAG request?

Saved on this device. No account needed.