Chapter 4 · concept 6 of 6 · show all

Chunk metadata, provenance and the chunking debugging playbook

What to store with every chunk so you can cite, filter and update it, and a step by step way to find out why a question was missed.

A real question

A test question failed. Was it the chunk size, the search, or the document?

Check four things in order: is the answer in the corpus, is it whole in one chunk, was that chunk found, and was it ranked high enough. Metadata on every chunk makes each check quick.

The problem

Why this matters

The belief

An answer was missed, so change the chunk size.

What really happens

Across all 16 settings, cut answers only happened with small fixed chunks. Every other miss was the right chunk ranked too low, which a new size will not fix.

Why not the simpler way

The simple chunk is a string and a vector. It searches fine, but when something goes wrong you cannot answer basic questions: which document and version did this come from, which section, is it still valid, may this user see it? And when a golden question fails, you cannot tell whether the cut, the ranking or the document was to blame.

The idea

The idea in plain words

Every chunk should carry its provenance: where it came from and in which state. With that, you can cite, filter, update and debug. Debugging then becomes a checklist instead of a guess: is the answer in the corpus, was it kept whole by the chunker, was it retrieved, and was it ranked high enough?

A museum label tells you the artist, the year, the room and the loan status. Without it, a painting is just a picture.

How it works

Step by step

1
Is the answer in the corpus?
Search the raw documents for the answer text. If it is missing, the problem is ingestion.
2
Is it whole in one chunk?
If no chunk holds the whole answer, the cut split it: fix size, overlap or strategy.
3
What rank did it get?
Just below the top 3: try hybrid search or reranking. Far down: look at the wording of question and chunk.

In detail

Store with every chunk: document id, version, file or URL, section path, character or page span, chunk index, content hash, effective dates, access labels, language and the chunker settings that made it.

The playbook for a missed question:

In the corpus? Search the raw documents for the evidence text. If it is missing, it is an ingestion problem.
Whole in one chunk? If no chunk contains the evidence, the cut split it. Fix the chunker, size or overlap.
Retrieved at all? Find the chunk's rank. If it is far down, look at the question and chunk wording, then try hybrid search or a contextual header.
Ranked below k? If it is just below the cut-off, try reranking or a slightly larger k, and measure the cost.

The code

Try it in Python

def diagnose(q, corpus_text, chunks, ranked_ids, k=3):
    """Classify why a golden question missed, in playbook order."""
    if q["evidence"] not in corpus_text:
        return "not in corpus: ingestion problem"
    holders = [c["id"] for c in chunks if q["evidence"] in c["text"]]
    if not holders:
        return "split: no chunk holds the whole answer"
    rank = min(ranked_ids.index(h) for h in holders)
    return "hit" if rank < k else f"ranked low: best holder at rank {rank + 1}"

See it

What we measured

Measured · misses by cause, 47 golden questions

All 16 precomputed settings, vector search with bge-small-en-v1.5, top 3. Split: no chunk holds the whole evidence. Ranked low: a chunk holds it but ranks below 3.

Fixed 128, no overlap: 6 split, 5 ranked lowbaseline11 misses

Fixed 128, 15% overlap: 0 split, 9 ranked low119 misses

Recursive 512: 0 split, 3 ranked low113 misses

All 13 other settings-1 split in total

Measured 2026-11-03 with the Chunk Lab logic. Splits happened only in fixed windows of 128 and 256 tokens without overlap. Every other miss was a ranking problem, so after a sensible chunk size the next gains come from retrieval, not from cutting.

Try it on a real example

Diagnose a real miss: “How much notice is given for a price change during a fixed term?” (recursive 512, vector search)

Check

Tariff sheet 2026.1

Price changes during a fixed term are announced at least 30 days in advance by e-mail.

Yes. Ingestion is fine.

Choosing

Use it when, avoid it when

Use it when

Provenance on every chunk: id, version, span, hash, dates, access
The four-step playbook on every failed golden question

Avoid it when

Chunks stored as bare text and vectors
Changing chunk size to fix what is really a ranking miss

What it costs · A few hundred bytes of metadata per chunk, and logging per evaluation run. Small next to the time it saves on the first incident.

In production

Log the chunk ids and ranks for every golden question in every evaluation run, and keep them next to the score. When a number drops after a change, you can diff which questions moved and why. Use the content hash to detect which chunks really changed on a re-index, and the effective dates and access labels as filters at query time.

Watch out

Common mistakes

Misses mean the chunk size is wrong. Here, most misses were ranking misses.
Overlap is a safe default. It fixed splits and created more ranking misses.
Metadata is only for filters. It is also how you cite, update and debug.

Check yourself

Remember this

Remember these

Store provenance on every chunk: document, version, span, hash, dates, access.
Diagnose misses in order: in corpus, whole, found, ranked high enough.
Overlap on 128 token chunks turned 6 cut answers into 0, but ranking misses rose from 5 to 9.
Log chunk ids and ranks per test question, so you can compare runs.

Say it in an interview

01Every chunk carries provenance: document, version, span, hash, dates and access labels.
02My playbook for a miss: in the corpus, whole in a chunk, retrieved, ranked high enough.
03Across 16 settings, splits only happened in small fixed windows; the rest were ranking misses.
04Overlap on fixed 128 turned 6 splits into 0 but raised ranking misses from 5 to 9.
05I log chunk ids and ranks per golden question so I can diff runs after a change.

Quick quiz

Three questions

Across all 16 chunk settings, what caused most misses?

Adding 15% overlap to fixed 128 changed the misses how?

Q3 · from RAG-15

What does evidence recall check that document recall does not?

Saved on this device. No account needed.