RAGEngineering Hub
Menu
Chapter 7 · concept 6 of 6 · show all
  1. 1What RAG is, and why
    1. 1.1Why models need outside knowledge, and what hallucination means
    2. 1.2What RAG is: anatomy of a request
    3. 1.3RAG vs fine tuning vs long context and prompt caching
    4. 1.4Use cases and the quality, latency and cost triangle
    5. Chapter overview
  2. 2Tokens, cost and your first RAG
    1. 2.1Tokens and context windows for RAG
    2. 2.2The RAG lifecycle and your first RAG in miniature
    3. Chapter overview
  3. 3Getting documents in
    1. 3.1Content sources and file formats
    2. 3.2Parsing, layout and OCR
    3. 3.3Tables, images and mixed content
    4. 3.4Cleaning, normalization and quality gates
    5. 3.5Metadata, document identity and versioning
    6. 3.6Updates, deletes, reindexing and ingestion pipelines
    7. Chapter overview
  4. 4Chunking
    1. 4.1Why we chunk, and the main chunking strategies
    2. 4.2Structure aware chunking and tables
    3. 4.3Chunk size, overlap and token based sizing
    4. 4.4Contextual chunking and late chunking
    5. 4.5Parent child and multi representation indexing
    6. 4.6Chunk metadata, provenance and the chunking debugging playbook
    7. Chapter overview
  5. 5Embeddings
    1. 5.1Embeddings and similarity from first principles
    2. 5.2What embeddings cannot do
    3. 5.3Choosing an embedding model: size, cost, language
    4. 5.4Embedding model migration and index rebuilds
    5. Chapter overview
  6. 6Vector stores and scores
    1. 6.1Vector databases, ANN indexes and choosing a store
    2. 6.2Score calibration and thresholds
    3. 6.3Late interaction and multivector retrieval
    4. Chapter overview
  7. 7Finding the right chunks
    1. 7.1Query understanding, rewriting and expansion
    2. 7.2Dense retrieval and keyword search with BM25
    3. 7.3Hybrid retrieval and rank fusion
    4. 7.4topK, no good match, and metadata filtering
    5. 7.5Routing, diversity, deduplication and caching
    6. 7.6Reranking: two stage, cross encoder and LLM rerankers
    7. Chapter overview
  8. 8When retrieval fails
    1. 8.1Retrieval failure analysis playbook
    2. Chapter overview
  9. 9Building the answer
    1. 9.1Context selection, packing and citations
    2. 9.2Grounding prompts and abstention
    3. 9.3Lost in the middle and context management
    4. 9.4Conflicts, staleness and freshness rules
    5. 9.5Hallucination checks and answer verification
    6. 9.6Conversational RAG, answer UX and RAG APIs
    7. Chapter overview
  10. 10Measuring
    1. 10.1Why measure, and what to measure
    2. 10.2Building a golden dataset
    3. 10.3Retrieval metrics: precision, recall, MRR, nDCG
    4. 10.4Faithfulness and answer correctness
    5. 10.5LLM as judge, and judging the judge
    6. 10.6Offline and online evals, regression tests and slices
    7. Chapter overview
  11. 11Safety and operations
    1. 11.1Prompt injection and poisoned documents
    2. 11.2Access control, privacy and governance
    3. 11.3Index lifecycle operations
    4. 11.4Observability and tracing
    5. 11.5Latency, cost, caching and model routing
    6. 11.6Reliability, degraded modes and launch readiness
    7. Chapter overview
  12. 12Scale and advanced RAG
    1. 12.1Scaling, multi tenancy and enterprise architecture
    2. 12.2Choosing an architecture: naive, advanced, agentic, graph
    3. 12.3Agentic RAG and RAG as an MCP tool
    4. 12.4GraphRAG, structured data and multimodal RAG
    5. Chapter overview

Reranking: two stage, cross encoder and LLM rerankers

A fast search finds 10 candidates, then a cross-encoder reads each one with the question and reorders them. On our corpus, answers ranked first rose from 30 to 40 of 47, at about 1.5 seconds per question on a CPU.

A real question

The answer is in the top 10, but at rank 4. How do we get it to rank 1?

Add a second, slower, more careful step. A reranker reads the question together with each of the 10 candidates and scores them again. Only the best few go to the model.

The problem

Why this matters

The first search already puts the candidates in the right order.
The belief

The first search already puts the candidates in the right order.

Vector search put the answer first for 30 of 47 questions. After a cross-encoder reranked its top 10, it was first for 40, and in the top 3 for all 47.
What really happens

Vector search put the answer first for 30 of 47 questions. After a cross-encoder reranked its top 10, it was first for 40, and in the top 3 for all 47.

Why not the simpler way

Embedding models encode the question and each chunk separately, then compare two summaries. That is what makes them fast enough to search everything, and also why they misjudge close calls: "notice for a price change" looks like any text about notices and terms. When the answer is in the top 10 but not the top 3, the fix is better ordering, not better search.

The idea

The idea in plain words

A bi-encoder (the embedding model) encodes question and chunk apart. A cross-encoder encodes them as one input, so every question word can attend to every chunk word. It is more accurate and much slower, so it only reads the short list. An LLM reranker asks a general model to score or order the candidates; it is flexible and the most expensive.

A recruiter screens 200 CVs in minutes by keywords. The hiring manager then reads the top 10 carefully.

How it works

Step by step

  1. BM25, vectors or both return the top 10 to 50. Fast over the whole corpus, rough in its order.
    1

    Cheap search finds candidates

    BM25, vectors or both return the top 10 to 50. Fast over the whole corpus, rough in its order.

  2. A cross-encoder takes “question + chunk” as one input and outputs a relevance score. It sees how the words relate.
    2

    Read each pair closely

    A cross-encoder takes “question + chunk” as one input and outputs a relevance score. It sees how the words relate.

  3. One model pass per candidate. On a laptop CPU, 10 candidates took about 1.5 seconds. Keep the candidate list short, or use a GPU or an API.
    3

    Pay in latency

    One model pass per candidate. On a laptop CPU, 10 candidates took about 1.5 seconds. Keep the candidate list short, or use a GPU or an API.

In detail

Pipeline. Retrieve N (10 to 50) with hybrid search, rerank, keep the top k (3 to 5) for the prompt (RAG-29).

Models. Open cross-encoders (bge-reranker, ms-marco MiniLM), hosted rerank APIs, or an LLM prompted to rank. We used bge-reranker-base, quantized, in Node on a CPU.

N matters. The reranker cannot fix what the retriever missed; it only reorders. Larger N raises the ceiling and the latency, linearly.

Scores. Reranker scores are not cosine scores. If you use them as a cut-off for "no good match", calibrate on your golden set (RAG-23).

Hybrid retrieval: the question is searched by keyword (BM25) and by meaning (vectors); reciprocal rank fusion merges the two ranked lists by position; the top 3 chunks go into the prompt.Question“E4012?”Keyword · BM25exact words, codesVector searchsimilar meaningFuse · RRF1 / (60 + rank)Top 3into the promptranked list Aranked list B
Fusion uses only each chunk's position in the two lists, so a BM25 score of 9.7 and a cosine of 0.67 never need to be compared.

The code

Try it in Python

from sentence_transformers import CrossEncoder

reranker = CrossEncoder("BAAI/bge-reranker-base")

def rerank(question: str, candidates: list[str], keep: int = 3) -> list[str]:
    scores = reranker.predict([(question, c) for c in candidates])   # one pass per pair
    order = sorted(range(len(candidates)), key=lambda i: -scores[i])
    return [candidates[i] for i in order[:keep]]

top10 = hybrid_search(question, k=10)
context = rerank(question, top10, keep=3)

See it

What we measured

Measured · answers at rank 1 and in the top 3, before and after reranking the top 10

Polder corpus, 34 recursive 512 token chunks, 47 golden questions. Reranker: Xenova/bge-reranker-base, q8, transformers.js on a laptop CPU, question + chunk text truncated to the 512 token limit. 14 of the question and answer-chunk pairs were over the limit, but in every one the evidence fell before the cut (corpus/scripts/check-truncation.mts).

Vector top 10: answer first3040
Vector top 10: answer in top 34447
Hybrid top 10: answer first3340
Hybrid top 10: answer in top 34747
Reranking time per question, 10 candidatesabout 1.5 s

Measured 2026-11-04 with corpus/scripts/measure-parts-4-7.mts. One question (g-34) moved from rank 1 to 2 after reranking either list; on the vector list g-08 (“When is a slow connection a line problem rather than Wi-Fi?”) also fell from 1 to 3. LLM rerankers were not measured.

Try it on a real example

“How much notice is given for a price change during a fixed term?”, vector top 10

Order
1.Claims handbook #1“We decide on your claim within 10 working days…”–
2.Endorsement E-12Flood cover from 1 April 2026.–
3.Bank annual report #2Credit risk and loan loss provisions.–
4.Tariff sheet 2026.1“Price changes during a fixed term are announced at least 30 days in advance by e-mail.”✓

The vector saw “notice” and “term” and found insurance and banking text first.

Choosing

Use it when, avoid it when

Use it when
  • A cross-encoder over the top 10 to 50 when the answer is found but ranked low
  • Reranker scores, calibrated, for no-good-match decisions
Avoid it when
  • Reranking hundreds of candidates per question
  • Expecting a reranker to recover answers the retriever missed

What it costs · One model pass per candidate: about 1.5 s for 10 on a laptop CPU, much less on a GPU. Hosted rerankers charge per search.

In production

Reranking is the largest added latency in retrieval (1.4 s for 10 pairs on CPU in Part 8). Its gain depends on the corpus, so measure it. Batch the pairs, run on a GPU or use a hosted reranker, and cap N. Measure the gain per slice: on our fused results, top-3 recall was already 47 of 47, so reranking only moved answers up within the top 3 (rank 1 from 33 to 40). One question got worse: “Is the fibre outage in postcode 3511 fixed?” went from rank 1 to 2. Check those regressions, not just the average.

Watch out

Common mistakes

  • Reranking improves recall. It only reorders; recall at N is fixed by the retriever.
  • A reranker never hurts. One of our questions dropped from rank 1 to 2.
  • LLM reranking is always best. It is the slowest and most expensive; measure it against a cross-encoder first.

Check yourself

Remember this

Remember these

  • Two stages: a fast retriever for recall, a reranker for order.
  • A cross-encoder reads question and chunk together.
  • On our corpus: answers first rose from 30 to 40 of 47.
  • Cost: about 1.5 s per question for 10 candidates on a CPU.

Say it in an interview

  1. 01I use two stages: hybrid search for the top 10, a cross-encoder to reorder.
  2. 02A cross-encoder reads question and chunk together, so it judges close calls better.
  3. 03On my corpus reranking raised answers at rank 1 from 30 to 40 of 47.
  4. 04It took about 1.5 seconds per question for 10 candidates on a CPU, so I cap N.
  5. 05I check per-question regressions: one answer moved from rank 1 to 2.
Try it yourself: Retrieval LabKeyword, vector and hybrid search side by side, with the rank of the answer in each.

Quick quiz

Three questions

Question 1

Q1

What does a cross-encoder do that an embedding model does not?

Question 2

Q2

On our corpus, what did reranking the vector top 10 change?

Question 3

Q3 · from RAG-28

Why does RRF not need score tuning?

Saved on this device. No account needed.