RAGEngineering Hub
Menu
Chapter 7 · concept 5 of 6 · show all
  1. 1What RAG is, and why
    1. 1.1Why models need outside knowledge, and what hallucination means
    2. 1.2What RAG is: anatomy of a request
    3. 1.3RAG vs fine tuning vs long context and prompt caching
    4. 1.4Use cases and the quality, latency and cost triangle
    5. Chapter overview
  2. 2Tokens, cost and your first RAG
    1. 2.1Tokens and context windows for RAG
    2. 2.2The RAG lifecycle and your first RAG in miniature
    3. Chapter overview
  3. 3Getting documents in
    1. 3.1Content sources and file formats
    2. 3.2Parsing, layout and OCR
    3. 3.3Tables, images and mixed content
    4. 3.4Cleaning, normalization and quality gates
    5. 3.5Metadata, document identity and versioning
    6. 3.6Updates, deletes, reindexing and ingestion pipelines
    7. Chapter overview
  4. 4Chunking
    1. 4.1Why we chunk, and the main chunking strategies
    2. 4.2Structure aware chunking and tables
    3. 4.3Chunk size, overlap and token based sizing
    4. 4.4Contextual chunking and late chunking
    5. 4.5Parent child and multi representation indexing
    6. 4.6Chunk metadata, provenance and the chunking debugging playbook
    7. Chapter overview
  5. 5Embeddings
    1. 5.1Embeddings and similarity from first principles
    2. 5.2What embeddings cannot do
    3. 5.3Choosing an embedding model: size, cost, language
    4. 5.4Embedding model migration and index rebuilds
    5. Chapter overview
  6. 6Vector stores and scores
    1. 6.1Vector databases, ANN indexes and choosing a store
    2. 6.2Score calibration and thresholds
    3. 6.3Late interaction and multivector retrieval
    4. Chapter overview
  7. 7Finding the right chunks
    1. 7.1Query understanding, rewriting and expansion
    2. 7.2Dense retrieval and keyword search with BM25
    3. 7.3Hybrid retrieval and rank fusion
    4. 7.4topK, no good match, and metadata filtering
    5. 7.5Routing, diversity, deduplication and caching
    6. 7.6Reranking: two stage, cross encoder and LLM rerankers
    7. Chapter overview
  8. 8When retrieval fails
    1. 8.1Retrieval failure analysis playbook
    2. Chapter overview
  9. 9Building the answer
    1. 9.1Context selection, packing and citations
    2. 9.2Grounding prompts and abstention
    3. 9.3Lost in the middle and context management
    4. 9.4Conflicts, staleness and freshness rules
    5. 9.5Hallucination checks and answer verification
    6. 9.6Conversational RAG, answer UX and RAG APIs
    7. Chapter overview
  10. 10Measuring
    1. 10.1Why measure, and what to measure
    2. 10.2Building a golden dataset
    3. 10.3Retrieval metrics: precision, recall, MRR, nDCG
    4. 10.4Faithfulness and answer correctness
    5. 10.5LLM as judge, and judging the judge
    6. 10.6Offline and online evals, regression tests and slices
    7. Chapter overview
  11. 11Safety and operations
    1. 11.1Prompt injection and poisoned documents
    2. 11.2Access control, privacy and governance
    3. 11.3Index lifecycle operations
    4. 11.4Observability and tracing
    5. 11.5Latency, cost, caching and model routing
    6. 11.6Reliability, degraded modes and launch readiness
    7. Chapter overview
  12. 12Scale and advanced RAG
    1. 12.1Scaling, multi tenancy and enterprise architecture
    2. 12.2Choosing an architecture: naive, advanced, agentic, graph
    3. 12.3Agentic RAG and RAG as an MCP tool
    4. 12.4GraphRAG, structured data and multimodal RAG
    5. Chapter overview

Routing, diversity, deduplication and caching

Four ways to shape the candidate list: send questions to the right index, spread results across sources, drop near copies, and reuse answers. Measured: diversity cost us 3 answers, and a semantic cache would confuse two flood questions.

A real question

Three of the top five chunks say the same thing. Is that a waste, and can we reuse answers?

Near copies do waste slots, and spreading results across sources can help a summary. But variety can also push the answer out, and reusing answers needs exact keys. Measure each.

The problem

Why this matters

More variety in the results is always better.
The belief

More variety in the results is always better.

MMR, a common diversity method, raised distinct files in the top 5 from 3.7 to 4.1. It also dropped answers in the top 3 from 44 to 41: for single-fact questions, the best chunk and its neighbour are often both needed.
What really happens

MMR, a common diversity method, raised distinct files in the top 5 from 3.7 to 4.1. It also dropped answers in the top 3 from 44 to 41: for single-fact questions, the best chunk and its neighbour are often both needed.

Why not the simpler way

Once retrieval works, the next problems are waste and cost. Versions of the same tariff sheet fill the top 5 with near copies. Popular questions are answered again and again at full price. One big index mixes telecom manuals with bank policies. Each problem has a standard fix, and each fix can quietly hurt quality.

The idea

The idea in plain words

Routing picks which index or tool to search. Diversity re-ranking, such as Maximal Marginal Relevance (MMR, Carbonell and Goldstein, SIGIR 1998), prefers candidates unlike the ones already picked. Deduplication removes near copies. Caching stores answers (or retrieval results) for reuse; a semantic cache reuses them for similar, not identical, questions.

A newspaper editor does not print three versions of the same story, sends sports questions to the sports desk, and reuses yesterday's weather only if it is still yesterday.

How it works

Step by step

  1. Send insurance questions to insurance documents. A keyword rule or a small classifier works; a wrong route loses the answer entirely.
    1

    Route to the right index

    Send insurance questions to insurance documents. A keyword rule or a small classifier works; a wrong route loses the answer entirely.

  2. Versions and repeated boilerplate create near copies. Keep one per group, or use diversity re-ranking where questions need breadth.
    2

    Drop near duplicates

    Versions and repeated boilerplate create near copies. Keep one per group, or use diversity re-ranking where questions need breadth.

  3. 3

    Cache with exact keys

    Reuse an answer only when the question matches and the key facts match: product, postcode, date, version, user role.

In detail

MMR. Pick the best candidate, then repeatedly pick the one with the highest λ · relevance − (1 − λ) · similarity to already picked. We used λ = 0.7 over the top 15.

Dedup. Hash normalized text for exact copies; use vector similarity above a high cut-off (0.95 here) or MinHash for near copies. Prefer the newest valid version (RAG-11).

Routing. Rules on keywords or metadata first; a classifier or a model call when rules are not enough. Always allow a fallback to search everything.

Caching. Exact-match caches are safe. Semantic caches need a strict similarity cut-off plus exact matching on codes, numbers and dates, and must be invalidated when the source documents change.

The code

Try it in Python

import hashlib, re

def cache_key(question: str, user_role: str, corpus_version: str) -> str:
    exact = sorted(re.findall(r"[A-Z]{1,4}-?\d[\w.-]*|\b\d{4}\b|\d+(?:[.,]\d+)?", question))
    norm = " ".join(question.lower().split())
    return hashlib.sha256(f"{norm}|{exact}|{user_role}|{corpus_version}".encode()).hexdigest()

# Exact-match cache: safe. A semantic cache must still compare the `exact` list,
# and corpus_version changes whenever a document is updated, which expires old entries.

See it

What we measured

Measured · diversity, duplicates and cache risk

Polder corpus, 34 recursive 512 token chunks, 47 golden questions, bge-small vector search. MMR with λ 0.7 over the top 15.

Distinct files in top 5plain 3.7MMR 4.1
Answers in top 3plain 44MMR 41
Chunk pairs with similarity above 0.956
Most similar golden pair with different answers0.893 (flood cover 1012 / 1011)

Measured 2026-11-04 with corpus/scripts/measure-parts-4-7.mts. Routing and a live cache were not measured; our corpus is one small index.

Try it on a real example

Two golden questions a semantic cache would treat as one

Show

Similarity 0.893

Is flood damage covered at an address in postcode 1012 in October 2026?
Is flood damage covered in postcode 1011 in May 2026?

The most similar pair in our golden set with different answers.

Choosing

Use it when, avoid it when

Use it when
  • Dedup of near copies and old versions
  • Exact-key caching with expiry on document change
  • MMR for broad questions such as summaries
Avoid it when
  • MMR on single-fact questions without measuring
  • Semantic caching on similarity alone

What it costs · Dedup and MMR: cheap vector math. Caching saves model calls but needs invalidation. Routing may add a classifier call.

In production

Measure cache hit rate and, more importantly, wrong-hit rate: sample cached answers and check them against a fresh run. Invalidate cache entries by document id when a document is updated or deleted (RAG-12). For routing, log the chosen route and how often the fallback fires; a route that is often wrong hides answers completely, which no reranker can fix.

Watch out

Common mistakes

  • Diversity always improves results. It cost 3 answers in our top 3.
  • Similar questions have the same answer. Our most similar differing pair scored 0.893.
  • A cache is free speed. A wrong hit is a wrong answer delivered quickly.

Check yourself

Remember this

Remember these

  • Route only when indexes are clearly separate, and log the route.
  • Near-duplicate chunks waste slots: we found 6 pairs above 0.95 similarity.
  • Diversity helps broad questions and hurt our single-fact ones: 44 to 41.
  • Cache on exact keys, and expire entries when documents change.

Say it in an interview

  1. 01I dedup near copies and keep the newest valid version.
  2. 02MMR raised source variety from 3.7 to 4.1 files but cost 3 answers in the top 3 on my corpus.
  3. 03Two flood questions scored 0.893 alike with different answers, so my cache keys on postcode and date.
  4. 04Caches expire when the source documents change.
  5. 05Routing gets a fallback to full search, and I log every route.

Quick quiz

Three questions

Question 1

Q1

What did MMR (a diversity re-ranking) do on our corpus?

Question 2

Q2

Why is a semantic cache risky for the two flood cover questions?

Question 3

Q3 · from RAG-29

When should metadata filters run?

Saved on this device. No account needed.