Chapter 4 · concept 5 of 6 · show all
2Tokens, cost and your first RAG
4Chunking
- 4.1Why we chunk, and the main chunking strategies
- 4.2Structure aware chunking and tables
- 4.3Chunk size, overlap and token based sizing
- 4.4Contextual chunking and late chunking
- 4.5Parent child and multi representation indexing
- 4.6Chunk metadata, provenance and the chunking debugging playbook
- Chapter overview
7Finding the right chunks
8When retrieval fails
Parent child and multi representation indexing
Search with small precise pieces, answer with the larger section they belong to, and other ways to index one passage in more than one form.
A real question
Small chunks are found easily but carry too little. Large chunks carry context but are hard to find. Do I have to choose?
No. Search with small pieces, and give the model the whole section each piece belongs to.
The problem
Why this matters
You pick one chunk size and live with its trade-off.
Parent child indexing uses two sizes: small children for search, their parent section for the answer. Same recall as 512 token chunks here, with half the prompt.
Why not the simpler way
Chunk size forces a trade. Small chunks match a question precisely but carry little context, so the answer can be half there. Large chunks carry context but match loosely and fill the prompt. The simple approach picks one size and accepts the loss. Parent child indexing refuses the trade: it searches with one size and answers with another.
The idea
The idea in plain words
Split each section (the parent) into small pieces (the children). Embed and search only the children. When a child matches, return its parent to the model. More generally, multi representation indexing stores several forms of the same passage for search, such as a summary, likely questions it answers, or a table rendered as sentences, and maps each back to the original passage.
A library index card lists one keyword, but points you to the whole chapter.
How it works
Step by step
1Split each section into children
Sections are the parents. Each one is cut into small children, and every child remembers its parent.
2Search only the children
Small pieces match a question precisely.
3Send the parent
For each matching child, the model gets the full section, so the answer has its context.
In detail
- Chunk by structure to get parents (here: sections up to 1,024 tokens).
- Split each parent into children (here: recursive 128 tokens), and keep the parent id on each child.
- At query time, rank children, walk down the list, and collect distinct parents until you have k.
- Put the parents, not the children, into the prompt.
The order matters. Our first attempt made children with a separate chunker over the whole document; some crossed a section boundary, mapped to a parent that held only part of the answer, and recall dropped to 26 of 47. Splitting children from inside each parent fixed it.
The code
Try it in Python
def build(parents, split_child, embed):
children = []
for pid, parent in enumerate(parents):
for text in split_child(parent): # children come from inside the parent
children.append({"pid": pid, "text": text, "vec": embed(text)})
return children
def retrieve(q_vec, children, parents, k=3):
ranked = sorted(children, key=lambda c: -(c["vec"] @ q_vec))
picked = []
for c in ranked:
if c["pid"] not in picked:
picked.append(c["pid"])
if len(picked) == k:
break
return [parents[p] for p in picked]
See it
What we measured
Measured · evidence recall@3 and context tokens per question
Polder corpus, 47 golden questions, bge-small-en-v1.5. Parents: sections up to 1,024 tokens (91). Children: recursive 128 inside each parent (125). Context tokens: median of the 3 units sent to the model.
Measured 2026-11-03 with fresh embeddings; the recursive 512 row re-measured 2026-11-04 with corpus/scripts/measure-part-3.mts. Same recall as recursive 512 with 51% fewer context tokens. Multi representation variants were not measured.
Try it on a real example
47 test questions, three ways to fill the prompt
Measured on the Polder corpus
Good recall, large prompts.
Choosing
Use it when, avoid it when
- Documents with clear sections and questions that need the surrounding text
- When the prompt budget is tight but recall must stay
- Children made by a separate chunker over the whole document
- Unbounded parents: very long sections fill the prompt
What it costs · Two stores instead of one, and more vectors: 125 children here against 34 chunks for recursive 512. In return, less than half the prompt tokens per question.
In production
Store parents in a plain key value store and children in the vector index, linked by id. Cap parent size, because a 5,000 token section returned three times defeats the purpose. For multi representation, generate summaries or questions once at index time and keep the generation prompt in version control; when the prompt changes, regenerate.
Watch out
Common mistakes
- Small chunks are always more precise. On their own they found 42 of 47; their parents found 44.
- Parent child is only for long documents. It helps wherever sections are natural answer units.
- Any child-to-parent mapping works. A child that crosses a boundary breaks it.
Check yourself
Remember this
Remember these
- Search small, answer with the section the small piece came from.
- Children are split from inside each parent, never across a boundary.
- Here: 44 of 47 answers with 404 instead of 817 context tokens.
- Multi representation goes further: index summaries or questions, return the original.
Say it in an interview
- 01Parent child search ranks small children and returns their parent section to the model.
- 02Children must be split from inside each parent, or the mapping breaks.
- 03On my corpus it matched recursive 512 at 44 of 47 with 404 instead of 817 context tokens.
- 04My first version made children separately and dropped to 26 of 47, which taught me that.
- 05Multi representation indexes summaries or questions, but always returns the original passage.
Quick quiz