Chapter 5 · concept 4 of 4 · show all

Embedding model migration and index rebuilds

Changing the embedding model means re-embedding everything. We mixed an old index with a new query model of the same size; recall fell from 44 to 28 of 47 without a single error.

A real question

We found a better embedding model. Can we just start using it for new questions?

No. Every stored vector was made by the old model, and the new model's vectors live in a different space. You have to re-embed everything and switch queries and index together.

The problem

Why this matters

The belief

Two models that both output 384 numbers are compatible.

What really happens

We searched the bge-small index with all-MiniLM query vectors. Both have 384 dimensions. Recall fell from 44 to 28 of 47, and nothing raised an error.

Why not the simpler way

Embedding models improve every few months, and vendors retire old ones. Sooner or later you will migrate. The danger is that a half-done migration looks fine: queries return results, scores look normal, and the answers get quietly worse.

The idea

The idea in plain words

Each embedding model learns its own map of meaning. Two maps can have the same number of coordinates and still put "router" in completely different places. A migration re-draws every point on the new map before anyone navigates with it.

Two cities can both use street numbers from 1 to 400. A house number from one city does not find a house in the other.

How it works

Step by step

1
Build the new index alongside
Re-embed every chunk with the new model into a separate index. The old one keeps serving users.
2
Test both on the golden set
Same questions, same metric. Switch only if the new index wins clearly, and check slices like languages and codes.
3
Switch together, keep a way back
Point queries and index at the new model in one release. Keep the old index until you are sure.

In detail

What changes. Chunk vectors, query vectors, any score cut-offs (RAG-23), cached query results, and sometimes chunk size, if the new model reads more or fewer tokens.

Blue-green index. Keep two indexes, old (blue) and new (green). Re-embed into green in the background; new and changed documents go to both while it runs. Evaluate green on the golden set, then flip a single setting that moves queries and index together.

Guard rails. Store model, version, prefix and pooling in the index metadata. At query time, refuse to search if the query model does not match. Dimension mismatches fail loudly (bge-base has 768, our index 384); same-dimension mismatches fail silently, so the check must compare names.

Cost. Re-embedding is one model call per chunk. For a hosted API that is tokens times price; for a local model it is compute time. Batch it and resume from a checkpoint.

The code

Try it in Python

INDEX_META = {"model": "BAAI/bge-small-en-v1.5", "prefix": "Represent this sentence for searching relevant passages: ", "pooling": "cls"}

def search(question: str, query_model_name: str):
    if query_model_name != INDEX_META["model"]:
        raise RuntimeError(f"query model {query_model_name} does not match index model {INDEX_META['model']}")
    ...

# Migration: build a second index with the new model, run the golden set on both,
# then change INDEX_META and the query model in the same release.

See it

What we measured

Measured · evidence recall@3 by index and query model

Polder corpus, 34 recursive 512 token chunks, 47 golden questions. Index and query vectors from the named models, q8, each with its own prefix and pooling.

bge-small index, bge-small queries44 of 47 (stored vectors)

bge-small index, all-MiniLM queries (mixed)4428 of 47

e5-small index, e5-small queries4447 of 47

Measured 2026-11-04 with corpus/scripts/measure-parts-4-7.mts. The mixed run uses the same 384 dimensions on both sides, so no error is raised.

Try it on a real example

Same 34 chunks and 47 questions

Index + queries

1.bge-small index + bge-small queries · 44 of 47One model for both.✓

Today's system.

Choosing

Use it when, avoid it when

Use it when

A parallel index and a golden-set comparison for every model change
A model-name check on every query

Avoid it when

Re-embedding only new documents with a new model
Switching the query model before the index is rebuilt

What it costs · One embedding call per chunk, plus storage for two indexes during the switch.

In production

Schedule migrations like database migrations: a plan, a dry run on a copy, a measured comparison and a rollback. Run the golden set per slice, because a new model can win overall and lose on one language or on exact codes. If the corpus is large, estimate the re-embedding time and cost before you start, and keep the old index until the new one has served real traffic without regressions.

Watch out

Common mistakes

Same size, same space. 384 and 384 gave 28 of 47.
A new model only matters for new documents. Every old vector must be redone too.
Migrations fail loudly. Only dimension mismatches do; the rest is silent.

Check yourself

Remember this

Remember these

Changing the model invalidates every stored vector.
Same dimensions do not mean the same space: mixing gave 28 of 47.
Build alongside, test on the golden set, switch both sides at once.
Store the model name and recipe with every vector, and check it at query time.

Say it in an interview

01A model change invalidates every vector, because each model has its own space.
02On my corpus, mixing two 384-dimension models dropped recall from 44 to 28 of 47 with no error.
03I migrate blue-green: build alongside, compare on the golden set, switch both sides together.
04I store the model name and recipe in index metadata and refuse mismatched queries.
05I re-tune score cut-offs after the switch, since every model has its own score range.

Quick quiz

Three questions

Index from bge-small, queries from all-MiniLM, both 384 numbers. What happened?

What is the safe way to switch embedding models?

Q3 · from RAG-12

When a document changes, which chunks need new embeddings?

Saved on this device. No account needed.