Chapter 11 · concept 3 of 6 · show all

Index lifecycle operations

An index is not built once. New documents, deletes, permission changes, new chunking and new models all change it. Build versioned indexes, gate them on the golden set, swap by alias, keep a rollback.

A real question

The index works today. What happens next month?

Documents are added, edited and deleted, permissions change, and you will want new chunking or a new model. Each is an index operation. Treat the index like a release: versioned, tested, swapped, and reversible.

The problem

Why this matters

The belief

Build the index once and it stays correct.

What really happens

In RAG-12, updating one tariff sheet left 2 orphan chunks, including a plan that was no longer for sale. In RAG-25, a half-done model change dropped recall from 44 to 27 of 47. Indexes go wrong quietly unless every change goes through a process.

Why not the simpler way

Most RAG incidents in production are not model failures. They are stale chunks after an update, documents that were deleted at the source but not in the index, permissions that changed without a re-sync, or a model upgrade done halfway. All of these are lifecycle problems.

The idea

The idea in plain words

The index lifecycle covers how indexes are built, updated, tested, promoted and retired. Incremental updates handle document changes. Full rebuilds handle chunking and model changes. Blue-green swaps make promotion and rollback instant.

A shop restocks shelves every day, but a new store layout is set up overnight and checked before the doors open.

How it works

Step by step

1
Build a versioned index
Each build has an id and a manifest: corpus snapshot, chunking settings, embedding model, build date.
2
Gate on the golden set
Run retrieval checks against the new build. No question that passed may fail without an explicit decision (RAG-44).
3
Swap and keep
Point the live alias at the new build. Keep the previous one until the new one has served real traffic.

In detail

Manifest. { build_id, corpus_snapshot, chunker, chunk_size, embed_model, prefix, created_at, golden_result }, stored with the index.

Incremental path. Change events from the source (or a nightly diff) → re-parse → re-chunk → hash compare → embed new chunks, delete removed ones, update metadata for permission changes (RAG-12, RAG-46).

Full path. New build in parallel, golden gate, alias swap, old build kept N days.

Housekeeping. Remove chunks whose documents no longer exist; expire caches by document id (RAG-30); alert when the corpus size jumps or drops unexpectedly.

The code

Try it in Python

import json, time

def build(corpus, chunker, model, golden, baseline):
    manifest = {"build_id": f"idx-{int(time.time())}", "chunker": chunker.name,
                "embed_model": model.name, "docs": {d.id: d.hash for d in corpus}}
    index = make_index(corpus, chunker, model)
    result = run_golden(index, golden)
    broke = [q for q, ok in baseline.items() if ok and not result[q]]
    if broke:
        raise SystemExit(f"{manifest['build_id']} blocked, broke {broke}")
    manifest["golden"] = sum(result.values())
    save(index, manifest)
    set_alias("live", manifest["build_id"])     # the old build stays for rollback

See it

What we measured

Measured · index builds and updates on the lab corpus

Polder corpus, 34 recursive 512 token chunks, bge-small q8 on a laptop CPU. Update numbers from RAG-12, migration from RAG-25.

Embed all 34 chunks879 ms

Build the BM25 indexabout 1 ms

Index file with vectors130 KB

Orphan chunks after one update without deletes2

Recall@3 with half-migrated models44 of 4727 of 47

Measured 2026-11-04 with corpus/scripts/measure-part-8.mts; earlier numbers measured 2026-11-03 and 2026-11-04.

Try it on a real example

Three changes, and what each one requires

Change

Incremental

Re-chunk the changed document, compare content hashes, embed only new chunks, delete the old ones.

Cheap, but deletes are where it goes wrong: our tariff update left 2 orphans without them (RAG-12).

Choosing

Use it when, avoid it when

Use it when

Versioned builds with manifests
Golden-set gates and alias swaps
Incremental updates with deletes for document edits

Avoid it when

Rebuilding the live index in place
Updates that add chunks but never delete

What it costs · Storage for two indexes during a swap, and a golden run per build.

In production

Measure build time and cost per chunk, so you know when a full rebuild stops being cheap. Track freshness: the time from a source edit to the index reflecting it. Run a daily consistency check: every indexed document still exists at the source with the same hash and permissions.

Watch out

Common mistakes

Indexes only grow. Deletes and permission changes matter as much as additions.
Incremental is always better. For small corpora a full rebuild is simpler and safer.
A successful build is a good build. Only the golden gate says that.

Check yourself

Remember this

Remember these

Every corpus, chunking or model change is an index operation.
Version builds with a manifest of what went into them.
Gate each build on the golden set; swap by alias.
Deletes and permission changes must reach the index too.

Say it in an interview

01I treat the index as a build artifact: versioned, gated, swapped by alias, with rollback.
02Document edits go incremental with hash diffs and deletes; chunking and model changes rebuild fully.
03On my corpus a full rebuild takes under a second, so I rebuild fully; at scale I would plan it.
04Each build stores a manifest of corpus, chunker and model.
05A daily check confirms every indexed document still exists with the same hash and permissions.

Quick quiz

Three questions

Why swap indexes through an alias rather than rebuilding in place?

How long did a full rebuild of our lab index take?

Q3 · from RAG-25

What does mixing two embedding models in one index do?

Saved on this device. No account needed.