RAGEngineering Hub
Menu
Chapter 12 · concept 3 of 4 · show all
  1. 1What RAG is, and why
    1. 1.1Why models need outside knowledge, and what hallucination means
    2. 1.2What RAG is: anatomy of a request
    3. 1.3RAG vs fine tuning vs long context and prompt caching
    4. 1.4Use cases and the quality, latency and cost triangle
    5. Chapter overview
  2. 2Tokens, cost and your first RAG
    1. 2.1Tokens and context windows for RAG
    2. 2.2The RAG lifecycle and your first RAG in miniature
    3. Chapter overview
  3. 3Getting documents in
    1. 3.1Content sources and file formats
    2. 3.2Parsing, layout and OCR
    3. 3.3Tables, images and mixed content
    4. 3.4Cleaning, normalization and quality gates
    5. 3.5Metadata, document identity and versioning
    6. 3.6Updates, deletes, reindexing and ingestion pipelines
    7. Chapter overview
  4. 4Chunking
    1. 4.1Why we chunk, and the main chunking strategies
    2. 4.2Structure aware chunking and tables
    3. 4.3Chunk size, overlap and token based sizing
    4. 4.4Contextual chunking and late chunking
    5. 4.5Parent child and multi representation indexing
    6. 4.6Chunk metadata, provenance and the chunking debugging playbook
    7. Chapter overview
  5. 5Embeddings
    1. 5.1Embeddings and similarity from first principles
    2. 5.2What embeddings cannot do
    3. 5.3Choosing an embedding model: size, cost, language
    4. 5.4Embedding model migration and index rebuilds
    5. Chapter overview
  6. 6Vector stores and scores
    1. 6.1Vector databases, ANN indexes and choosing a store
    2. 6.2Score calibration and thresholds
    3. 6.3Late interaction and multivector retrieval
    4. Chapter overview
  7. 7Finding the right chunks
    1. 7.1Query understanding, rewriting and expansion
    2. 7.2Dense retrieval and keyword search with BM25
    3. 7.3Hybrid retrieval and rank fusion
    4. 7.4topK, no good match, and metadata filtering
    5. 7.5Routing, diversity, deduplication and caching
    6. 7.6Reranking: two stage, cross encoder and LLM rerankers
    7. Chapter overview
  8. 8When retrieval fails
    1. 8.1Retrieval failure analysis playbook
    2. Chapter overview
  9. 9Building the answer
    1. 9.1Context selection, packing and citations
    2. 9.2Grounding prompts and abstention
    3. 9.3Lost in the middle and context management
    4. 9.4Conflicts, staleness and freshness rules
    5. 9.5Hallucination checks and answer verification
    6. 9.6Conversational RAG, answer UX and RAG APIs
    7. Chapter overview
  10. 10Measuring
    1. 10.1Why measure, and what to measure
    2. 10.2Building a golden dataset
    3. 10.3Retrieval metrics: precision, recall, MRR, nDCG
    4. 10.4Faithfulness and answer correctness
    5. 10.5LLM as judge, and judging the judge
    6. 10.6Offline and online evals, regression tests and slices
    7. Chapter overview
  11. 11Safety and operations
    1. 11.1Prompt injection and poisoned documents
    2. 11.2Access control, privacy and governance
    3. 11.3Index lifecycle operations
    4. 11.4Observability and tracing
    5. 11.5Latency, cost, caching and model routing
    6. 11.6Reliability, degraded modes and launch readiness
    7. Chapter overview
  12. 12Scale and advanced RAG
    1. 12.1Scaling, multi tenancy and enterprise architecture
    2. 12.2Choosing an architecture: naive, advanced, agentic, graph
    3. 12.3Agentic RAG and RAG as an MCP tool
    4. 12.4GraphRAG, structured data and multimodal RAG
    5. Chapter overview

Agentic RAG and RAG as an MCP tool

Instead of one fixed search, a model decides what to look up, reads, and searches again. Exposing retrieval as an MCP tool lets any agent use it. When the loop helps, what it costs, and how to keep it safe.

A real question

Some questions need two or three look-ups. Can the model do the searching itself?

Yes: give it a search tool and let it decide what to fetch, read the results and search again. That is agentic RAG. It handles multi-step questions, at the cost of more calls, more latency and new ways to fail.

The problem

Why this matters

The belief

Agents always beat a fixed pipeline.

What really happens

Our one multi-document question, “What caused incident INC-77?”, is already answered by one hybrid search (answer at rank 2). An agent would add model calls for no gain. Agents pay off when questions really need several dependent steps.

Why not the simpler way

A fixed pipeline makes one search with the user's words. Some questions need more: compare two plans, find the latest status, combine a policy with a customer's own data, or calculate. Agents handle these by planning and calling tools, which is also how many assistants now consume RAG.

The idea

The idea in plain words

Agentic RAG: a model in a loop, with tools such as search, document fetch, database queries or calculators, deciding each next step from what it has read. RAG as a tool: retrieval is offered as a function an agent can call. MCP (Model Context Protocol, an open standard published by Anthropic in 2024) defines how tools and data sources are described to models, so one retrieval server can serve many clients.

A researcher with a library card: rather than handing over one search result, you let them search, read and search again until they can answer.

How it works

Step by step

  1. 1

    Plan

    The model reads the question and decides what to look up first: “find the bulletins about 3511, then the latest one”.

  2. 2

    Retrieval as a tool

    search_documents and get_document become tools, ideally behind MCP so any agent can use them with the same filters and access rules.

  3. 3

    Cap and log

    Limit steps and tokens, log every tool call in the trace, and keep the same grounding and verification rules for the final answer.

In detail

Loop. System rules, tools, the question → the model calls a tool → results are added → repeat until it answers or hits a limit.

Tool design. Few, clear tools: search_documents(query, filters) returning ids, titles, versions and snippets; get_document(id, section) returning text. Access filters come from the session, never from tool arguments the model chooses (RAG-46).

Guard rails. Step and token limits, timeouts, no write actions without confirmation, and the injection rules for anything a tool returns (RAG-45).

Evaluation. Score the final answer as before, and also the path: number of steps, tool errors, and whether the needed documents were fetched.

The code

Try it in Python

TOOLS = [
    {"name": "search_documents",
     "description": "Search company documents. Returns ids, titles, versions, dates and snippets.",
     "input_schema": {"type": "object", "properties": {"query": {"type": "string"}}, "required": ["query"]}},
    {"name": "get_document",
     "description": "Fetch the text of one document section by id.",
     "input_schema": {"type": "object", "properties": {"id": {"type": "string"}}, "required": ["id"]}},
]

def run_tool(name, args, session):
    if name == "search_documents":                       # roles from the session, not from the model
        return hybrid_search(args["query"], roles=session.roles)[:5]
    if name == "get_document":
        return fetch_if_allowed(args["id"], roles=session.roles)

MAX_STEPS = 4    # stop and answer with what was found, or abstain

See it

What we measured

Measured · questions that might need several steps

Polder corpus, 34 recursive 512 token chunks, hybrid search, with and without reranking.

Multi-document questions in the golden set1 (INC-77 cause)
INC-77 cause: rank with one hybrid search2
3511 outage status: hybrid / reranked12
Agent loop on these questionsnot measured

Measured 2026-11-04 with corpus/scripts/measure-parts-4-7.mts. An agent needs model calls: not measured yet. Our golden set has few multi-step questions, so it cannot show where an agent pays off.

Try it on a real example

“Is the fibre outage in postcode 3511 fixed?” (asked on 2026-10-03)

Show

Network bulletins 1, 2, 4 (INC-77)

10-01 09:00: “cause under investigation” · 10-01 14:30: “caused by a damaged cable; expected restore 10-02 06:00” · 10-03 07:00: “resolved on 2026-10-02 at 05:40”

The same outage at three moments. Only the latest is the answer.

Choosing

Use it when, avoid it when

Use it when
  • Agents for multi-step questions, comparisons and tool use
  • Retrieval behind MCP with session-based access
  • Step, token and time limits
Avoid it when
  • Agents for simple look-ups
  • Tools that accept access roles from the model

What it costs · 2 to several model calls per question instead of 1, plus variable latency.

In production

Agent cost and latency vary per question: track p95 steps and tokens, and cap them. Cache tool results within a session. Keep a fixed pipeline for the common simple questions and route only complex ones to the agent (RAG-49). Log the whole loop in one trace (RAG-48), because failures are often a wrong tool choice early on.

Watch out

Common mistakes

  • Agents remove the need for good retrieval. They call the same retrieval, more often.
  • More steps mean better answers. They also mean more chances to go wrong.
  • MCP is a model. It is a protocol for connecting tools and data to models.

Check yourself

Remember this

Remember these

  • Agentic RAG lets the model search, read and search again.
  • Use it for multi-step questions and tools; not for plain look-ups.
  • Expose retrieval as an MCP tool with the same filters and access rules.
  • Cap steps, log every call, and keep grounding and verification.

Say it in an interview

  1. 01Agentic RAG lets the model plan, search, read and search again with tools.
  2. 02I use it for multi-step questions; plain look-ups stay on the fixed pipeline.
  3. 03Retrieval is exposed as an MCP tool, with access roles taken from the session.
  4. 04I cap steps and tokens and log every tool call in one trace.
  5. 05On my corpus the one multi-document question was already answered by one hybrid search.

Quick quiz

Three questions

Question 1

Q1

"Is the fibre outage in postcode 3511 fixed?" Why can one fixed search struggle?

Question 2

Q2

What is MCP?

Question 3

Q3 · from RAG-52

What should come before adding an agent?

Saved on this device. No account needed.