Chapter 6 · concept 3 of 3 · show all

Late interaction and multivector retrieval

Instead of one vector per chunk, keep many (per sentence, or per token as in ColBERT) and score by the best match. We tested the sentence version; it used 12 times the vectors for the same result.

A real question

A chunk talks about five things. Can one vector stand for all five?

One vector is an average, and the one sentence you need gets diluted. Multivector retrieval keeps a vector per sentence, or per token, and scores a chunk by its best match.

The problem

Why this matters

The belief

More vectors per chunk always find more answers.

What really happens

We stored one vector per sentence: 405 vectors instead of 33. Recall stayed at 44 of 47. It helps when chunks are long and mixed; ours are short and on one topic.

Why not the simpler way

A 512-token chunk of a policy might cover scope, exceptions, deadlines and contacts. Its single vector is a blend of all four. A question about the deadline has to compete with that blend. Smaller chunks are one fix (RAG-15); keeping several vectors per chunk is another.

The idea

The idea in plain words

Single-vector (bi-encoder) retrieval: one vector per chunk, one per query, one dot product. Multivector retrieval: many vectors per chunk, scored by how well the best pieces match. Late interaction is the token-level version: query and document are encoded separately (so documents can be indexed ahead of time), and they "interact" only at search time, token by token.

A book's back cover summarizes the whole book. Its index lets you find the one page that mentions your word.

How it works

Step by step

1
Split into pieces
Embed each sentence, or in ColBERT each token, separately. A chunk now has many vectors.
2
Score by best match
Compare the question with every piece and keep the best (MaxSim). ColBERT sums the best match for each query token.
3
Pay for it
Storage and compute grow with the number of pieces. Systems compress the vectors and search in two passes.

In detail

Sentence MaxSim (what we tested): split each chunk into sentences, embed each, and give the chunk the highest sentence score. Simple, and it works with any embedding model.

ColBERT (Khattab and Zaharia, arXiv 2004.12832): a model trained to output a vector per token. The score is the sum, over query tokens, of each token's best match in the document. ColBERTv2 and PLAID compress the token vectors and use a first pass over centroids to keep it fast.

Multi-representation is a cousin: index a summary, a few hypothetical questions or the title as extra vectors pointing to the same chunk (RAG-17).

The code

Try it in Python

import re

def sentences(chunk: str) -> list[str]:
    return [s for s in re.split(r"(?<=[.!?])\s+|\n+", chunk) if len(s.strip()) > 20]

pieces = [(i, s) for i, c in enumerate(chunks) for s in sentences(c)]
piece_vecs = model.encode([s for _, s in pieces], normalize_embeddings=True)

def maxsim_search(q_vec, k=3):
    best = {}
    for (i, _), v in zip(pieces, piece_vecs):
        best[i] = max(best.get(i, -1.0), float(v @ q_vec))
    return sorted(best, key=best.get, reverse=True)[:k]

See it

What we measured

Measured · evidence recall@3

Polder corpus, 34 recursive 512 token chunks, bge-small. Sentence split on . ! ? and line breaks, sentences under 20 characters dropped. Chunk score = best sentence cosine.

One vector per chunk33 vectors44 of 47

One vector per sentence (MaxSim)405 vectors44 of 47

Measured 2026-11-04 with corpus/scripts/measure-parts-4-7.mts. ColBERT itself was not measured.

Try it on a real example

Same 34 chunks, 47 questions, bge-small

Vectors

1.33 vectors · 44 of 47Each chunk is one average of its meaning.✓

The baseline.

Choosing

Use it when, avoid it when

Use it when

Long, mixed-topic chunks you cannot make smaller
Cases where a reranker is too slow and recall is still low

Avoid it when

Short, single-topic chunks: no dilution to fix
Adding it before a reranker, which is easier to try

What it costs · Storage and embedding work grow with the number of pieces: 12 times here for sentences, more for tokens.

In production

Token-level indexes are several times larger than single-vector ones, and need a store that supports them. Try multivector when chunks are long, cover several topics, or when single-vector search ranks the right document low and smaller chunks are not an option. Measure against a reranker first (RAG-31): a reranker also reads token by token, only over the top candidates, and needs no new index.

Watch out

Common mistakes

Late interaction means late chunking. Late chunking (RAG-16) pools token vectors into one vector per chunk; late interaction keeps them separate.
Multivector is always better. It matched single-vector exactly on our corpus.
It needs a new embedding model. The sentence version works with the model you already have; ColBERT needs its own.

Check yourself

Remember this

Remember these

One vector per chunk averages everything the chunk says.
Multivector keeps many vectors and scores by the best match.
ColBERT does this per token, with compression to keep it affordable.
On our short chunks, 12 times the vectors gave the same 44 of 47.

Say it in an interview

01A single vector averages a chunk; multivector keeps many and scores by the best match.
02ColBERT does it per token with MaxSim and compresses the vectors to stay affordable.
03On my corpus, one vector per sentence used 405 vectors instead of 33 for the same 44 of 47.
04It pays off for long mixed chunks; I would try a reranker first.
05Late interaction keeps token vectors; late chunking pools them.

Quick quiz

Three questions

How does a multivector score a chunk?

What happened when we stored one vector per sentence on our corpus?

Q3 · from RAG-22

Binary vectors plus rescoring is an example of what pattern?

Saved on this device. No account needed.