Chapter 4 · concept 4 of 6 · show all

Contextual chunking and late chunking

Two ways to give a chunk the context it lost when it was cut out, a cheap header and a model-written summary, plus late chunking, with a measured test of the cheap one.

A real question

A chunk says “the fee is EUR 20.00”. Which product? Which year?

Once a chunk is cut out, it forgets where it came from. Contextual chunking puts that context back before the chunk is embedded.

The problem

Why this matters

The belief

Each chunk carries enough meaning on its own to be found.

What really happens

Small chunks often start mid-topic. Adding the document title and section before embedding lifted our 128 token chunks from 41 to 44 of 47 answers.

Why not the simpler way

A chunk cut out of a document forgets where it came from. "The fee is EUR 20.00 per month" does not say which plan, which product or which year. The simple fix is bigger chunks, but bigger chunks rank worse and cost more prompt tokens. The better fix is to put the missing context back into each chunk before it is embedded.

The idea

The idea in plain words

Contextual chunking adds a short description to each chunk before embedding: at the cheap end, the document title and section heading; at the expensive end, a sentence written by a model that explains where the chunk sits in the document. Late chunking takes another route: it embeds the whole document at once and only then splits the token vectors into chunks, so each chunk's vector already "saw" the rest of the document.

A photo from an album is easier to understand with the caption "Amsterdam, 2019, Anna's wedding" written on the back.

How it works

Step by step

1
A chunk is cut out
It keeps its words, but loses the title and heading that said what it is about.
2
Add a short header
Write “Document: … Section: …” in front of it. A costlier option is a sentence of context written by a model.
3
Embed with context, show the original
The header goes into the vector only. Users and the model still see the original chunk.

In detail

Header variant. Prepend "Document: Polder Telecom Consumer Tariff Sheet 2025.2. Section: Mobile plans." to the chunk text, embed that, but store and show the original chunk. It costs nothing at query time and very little at index time.

Model-written context. For each chunk, ask a model to write one or two sentences of context from the whole document, then embed context plus chunk. The vendor that described this reported 35% fewer failed retrievals with contextual embeddings, 49% when BM25 also used the context, and 67% with reranking on top (Anthropic, "Introducing Contextual Retrieval", vendor blog, September 2024). It costs one model call per chunk at index time; prompt caching of the document reduces that.

Late chunking. Run a long-context embedding model over the whole document, keep its per-token vectors, and average them over each chunk's span (Günther et al., "Late Chunking", arXiv 2409.04701, 2024). It needs a model whose input fits the document; our 512 token bge-small cannot do it for the three long documents.

The code

Try it in Python

import re

def with_header(chunk: str, title: str, body: str, start: int) -> str:
    """Text to embed: document title and nearest heading, then the chunk itself."""
    headings = re.findall(r"(?m)^#{1,6} (.+)$", body[: start + 1])
    section = headings[-1] if headings else ""
    return f"Document: {title}. Section: {section}.\n{chunk}"

# Embed with_header(...), but store and return the original chunk text.

See it

What we measured

Measured · evidence recall@3

Polder corpus, 47 golden questions, recursive chunks, vector search with bge-small-en-v1.5. Header: 'Document: <title>. Section: <nearest heading>.' added before embedding only.

Recursive 128, plainbaseline0.872 (41 of 47)

Recursive 128, with header0.8720.936 (44 of 47)

Recursive 512, plain0.9360.936 (44 of 47)

Recursive 512, with header0.9360.936 (44 of 47)

Measured 2026-11-04: 128 token rows with fresh embeddings, 512 token rows with corpus/scripts/measure-part-3.mts. The header helped small chunks and did nothing for large ones, which already carry most of their context. Before the 512 chunks were re-split to fit the embedder, the header cost one question (43 of 47). Model-written context and late chunking were not measured here.

Try it on a real example

The same 128 token chunk, as it gets embedded

Embed

FiberBox manual · recursive 128, chunk 18

To reset, press the recessed reset button on the back with a paperclip for 15 seconds, until all lights blink at once. … Do not reset a router that shows a solid red power light. …

Nothing in the chunk says which router or which manual this is.

Choosing

Use it when, avoid it when

Use it when

Small chunks from long documents: add a title and section header
Many similar clauses across documents: consider model-written context

Avoid it when

Headers on already large chunks without measuring: they gained nothing here
Showing the added context to users as if it were document text

What it costs · Header: no extra calls. Model-written context: one model call per chunk at index time. Late chunking: a long-context embedding model and a different indexing pipeline.

In production

Start with the header variant: it is nearly free and you can measure it in an afternoon. Move to model-written context when your chunks are short and your documents are long and repetitive, such as contracts or regulations with many similar clauses. Keep the added context out of what you show the user; it is for search, not for the answer.

Watch out

Common mistakes

Context always helps. At 512 tokens it cost one answer passage here.
Late chunking is a chunking strategy. It is an embedding strategy; you still choose where to cut.
Contextual retrieval needs a new database. It changes the text you embed, nothing else.

Check yourself

Remember this

Remember these

A cut-out chunk loses its document and section context.
A title and section header is nearly free and easy to measure.
Model-written context costs one call per chunk at index time.
Late chunking embeds the whole document first, then splits the vectors.

Say it in an interview

01A chunk loses its context when cut out; contextual chunking puts it back before embedding.
02The cheap version is a title and section header; the costly one is a model-written summary.
03On my corpus the header lifted 128 token chunks from 41 to 44 of 47 and left 512 unchanged at 44.
04Late chunking embeds the whole document first, then pools vectors per chunk.
05I embed the context but store and show only the original chunk.

Quick quiz

Three questions

Adding "Document title. Section." to each chunk before embedding did what on our corpus?

What does late chunking change?

Q3 · from RAG-15

Why did 15% overlap not help in the earlier chunk size test?

Saved on this device. No account needed.