RAGEngineering Hub
Menu
Chapter 9 · concept 6 of 6 · show all
  1. 1What RAG is, and why
    1. 1.1Why models need outside knowledge, and what hallucination means
    2. 1.2What RAG is: anatomy of a request
    3. 1.3RAG vs fine tuning vs long context and prompt caching
    4. 1.4Use cases and the quality, latency and cost triangle
    5. Chapter overview
  2. 2Tokens, cost and your first RAG
    1. 2.1Tokens and context windows for RAG
    2. 2.2The RAG lifecycle and your first RAG in miniature
    3. Chapter overview
  3. 3Getting documents in
    1. 3.1Content sources and file formats
    2. 3.2Parsing, layout and OCR
    3. 3.3Tables, images and mixed content
    4. 3.4Cleaning, normalization and quality gates
    5. 3.5Metadata, document identity and versioning
    6. 3.6Updates, deletes, reindexing and ingestion pipelines
    7. Chapter overview
  4. 4Chunking
    1. 4.1Why we chunk, and the main chunking strategies
    2. 4.2Structure aware chunking and tables
    3. 4.3Chunk size, overlap and token based sizing
    4. 4.4Contextual chunking and late chunking
    5. 4.5Parent child and multi representation indexing
    6. 4.6Chunk metadata, provenance and the chunking debugging playbook
    7. Chapter overview
  5. 5Embeddings
    1. 5.1Embeddings and similarity from first principles
    2. 5.2What embeddings cannot do
    3. 5.3Choosing an embedding model: size, cost, language
    4. 5.4Embedding model migration and index rebuilds
    5. Chapter overview
  6. 6Vector stores and scores
    1. 6.1Vector databases, ANN indexes and choosing a store
    2. 6.2Score calibration and thresholds
    3. 6.3Late interaction and multivector retrieval
    4. Chapter overview
  7. 7Finding the right chunks
    1. 7.1Query understanding, rewriting and expansion
    2. 7.2Dense retrieval and keyword search with BM25
    3. 7.3Hybrid retrieval and rank fusion
    4. 7.4topK, no good match, and metadata filtering
    5. 7.5Routing, diversity, deduplication and caching
    6. 7.6Reranking: two stage, cross encoder and LLM rerankers
    7. Chapter overview
  8. 8When retrieval fails
    1. 8.1Retrieval failure analysis playbook
    2. Chapter overview
  9. 9Building the answer
    1. 9.1Context selection, packing and citations
    2. 9.2Grounding prompts and abstention
    3. 9.3Lost in the middle and context management
    4. 9.4Conflicts, staleness and freshness rules
    5. 9.5Hallucination checks and answer verification
    6. 9.6Conversational RAG, answer UX and RAG APIs
    7. Chapter overview
  10. 10Measuring
    1. 10.1Why measure, and what to measure
    2. 10.2Building a golden dataset
    3. 10.3Retrieval metrics: precision, recall, MRR, nDCG
    4. 10.4Faithfulness and answer correctness
    5. 10.5LLM as judge, and judging the judge
    6. 10.6Offline and online evals, regression tests and slices
    7. Chapter overview
  11. 11Safety and operations
    1. 11.1Prompt injection and poisoned documents
    2. 11.2Access control, privacy and governance
    3. 11.3Index lifecycle operations
    4. 11.4Observability and tracing
    5. 11.5Latency, cost, caching and model routing
    6. 11.6Reliability, degraded modes and launch readiness
    7. Chapter overview
  12. 12Scale and advanced RAG
    1. 12.1Scaling, multi tenancy and enterprise architecture
    2. 12.2Choosing an architecture: naive, advanced, agentic, graph
    3. 12.3Agentic RAG and RAG as an MCP tool
    4. 12.4GraphRAG, structured data and multimodal RAG
    5. Chapter overview

Conversational RAG, answer UX and RAG APIs

Follow-up questions, showing sources, collecting feedback and returning structured answers. We searched 7 follow-ups three ways: as typed, glued to the previous question, and rewritten.

A real question

A user asks “What about 1011 in May?”. What does the search engine see?

Only those five words, unless you rewrite them. A conversational RAG system turns follow-ups into stand-alone questions, shows its sources, accepts feedback and returns answers in a shape other programs can use.

The problem

Why this matters

Add the previous question to the follow-up, and search has the context it needs.
The belief

Add the previous question to the follow-up, and search has the context it needs.

On 7 follow-ups we wrote, gluing found the answer in the top 3 for 5, typed alone for 6, and a stand-alone rewrite for all 7. The old question's words pulled search back to the old topic.
What really happens

On 7 follow-ups we wrote, gluing found the answer in the top 3 for 5, typed alone for 6, and a stand-alone rewrite for all 7. The old question's words pulled search back to the old topic.

Why not the simpler way

Most RAG demos answer one question at a time. Real users have conversations: they ask, read, and follow up with "and for the washing machine?" or "what about last year?". Without handling that, the second question fails even though the system answered the first one well. The answer also has to reach the user, and other systems, in a usable form.

The idea

The idea in plain words

Conversational RAG keeps enough of the conversation to understand follow-ups, and searches with a rewritten, stand-alone question. Answer UX shows the answer with its sources, makes abstentions useful, and collects feedback. A RAG API returns a structured response other programs can rely on.

A good help desk agent repeats your question back in full before looking it up: "So you want to know how long the decision takes once your claim is complete?"

How it works

Step by step

  1. Turn “What about 1011 in May?” into “Is flood damage covered in postcode 1011 in May 2026?” using the last few turns.
    1

    Rewrite follow-ups

    Turn “What about 1011 in May?” into “Is flood damage covered in postcode 1011 in May 2026?” using the last few turns.

  2. Show citations as links to the exact document and version. A thumbs up or down with an optional reason feeds your evaluation.
    2

    Show sources, ask for feedback

    Show citations as links to the exact document and version. A thumbs up or down with an optional reason feeds your evaluation.

  3. Answer text, citations with ids, an abstained flag and a trace id. The interface and the logs both need them.
    3

    Return structure

    Answer text, citations with ids, an abstained flag and a trace id. The interface and the logs both need them.

In detail

Rewrite. A small model gets the last few turns and the new message and returns one stand-alone question (RAG-26). Keep codes, numbers and dates exactly. Skip the rewrite when the message is already stand-alone.

Memory. Keep recent turns verbatim, summarize older ones (RAG-35). Do not carry old retrieved chunks forward by default; search again with the rewritten question.

UX. Citations as clickable sources with document title and version; a clear abstention message with a next step; streaming text so users see progress; feedback buttons tied to the trace id.

API shape. { answer, citations: [{ id, doc, version, url }], abstained, trace_id, model, latency_ms }. Version the API so clients survive changes.

The code

Try it in Python

from dataclasses import dataclass, field

@dataclass
class Citation:
    id: int
    doc: str
    version: str
    url: str

@dataclass
class RagAnswer:
    answer: str
    citations: list[Citation] = field(default_factory=list)
    abstained: bool = False
    trace_id: str = ""
    rewritten_question: str = ""

def handle(message, history, rewrite, retrieve, generate, trace_id):
    question = rewrite(history[-4:], message) if history else message
    sources = retrieve(question)
    text, cited = generate(question, sources)
    return RagAnswer(text, cited, text.startswith("I don't know"), trace_id, question)

See it

What we measured

Measured · answer chunk in the top 3 for follow-up questions

Polder corpus, 34 recursive 512 token chunks, hybrid search. 7 follow-ups written by us, each after a golden question. Rewritten = the matching golden question, written by hand as the ideal rewrite.

Follow-up as typed6 of 7
Previous question + follow-up5 of 7
Stand-alone rewrite7 of 7
Hardest case (decision time)as typed rank 11rewritten rank 2

Measured 2026-11-04 with corpus/scripts/measure-parts-6-7.mts. Our follow-ups often repeat key words, so typing alone did well; model-written rewrites were not measured (not measured yet).

Try it on a real example

Follow-up after “Within how many days must I report home contents damage?”

Search with

Query

How long do you take to decide once it is complete?

Answer chunk at rank 11. “It” and “you” mean nothing to search.

Choosing

Use it when, avoid it when

Use it when
  • A stand-alone rewrite for follow-ups, logged
  • Structured answers with citations and an abstained flag
Avoid it when
  • Gluing previous turns onto the query
  • Returning only a text string from a RAG API

What it costs · One small model call per follow-up for the rewrite. Structure and feedback cost almost nothing.

In production

Log the original message, the rewrite, the retrieved ids and the answer under one trace id (RAG-48). Follow-up failures are usually rewrite failures, and the log shows it in seconds. Add follow-up conversations to the golden set as short scripts: two or three turns with the expected answer for the last one.

Watch out

Common mistakes

  • Chat history solves follow-ups. The search still needs one clear question.
  • More history in the query helps. Gluing found fewer answers than typing alone here.
  • Users read the sources. Many do not; make the most important source easy to open.

Check yourself

Remember this

Remember these

  • Rewrite follow-ups into stand-alone questions before search.
  • Gluing the previous turn on is not a rewrite: 5 of 7 versus 7 of 7 here.
  • Show sources as links and collect feedback with a reason.
  • Return structured answers: text, citations, abstained flag, trace id.

Say it in an interview

  1. 01I rewrite follow-ups into stand-alone questions and log both.
  2. 02On my 7 follow-ups, gluing the previous question found 5, a rewrite found 7.
  3. 03Answers return as structured data: text, citations, an abstained flag and a trace id.
  4. 04Feedback is tied to the trace id so bad answers can be replayed.
  5. 05Multi-turn conversations are part of my golden set.

Quick quiz

Three questions

Question 1

Q1

Gluing the previous question onto the follow-up did what on our 7 follow-ups?

Question 2

Q2

Why return an answer as structured data rather than a string?

Question 3

Q3 · from RAG-26

When should a follow-up be rewritten before search?

Saved on this device. No account needed.