RAGEngineering Hub
Menu
Chapter 1 · concept 2 of 4 · show all
  1. 1What RAG is, and why
    1. 1.1Why models need outside knowledge, and what hallucination means
    2. 1.2What RAG is: anatomy of a request
    3. 1.3RAG vs fine tuning vs long context and prompt caching
    4. 1.4Use cases and the quality, latency and cost triangle
    5. Chapter overview
  2. 2Tokens, cost and your first RAG
    1. 2.1Tokens and context windows for RAG
    2. 2.2The RAG lifecycle and your first RAG in miniature
    3. Chapter overview
  3. 3Getting documents in
    1. 3.1Content sources and file formats
    2. 3.2Parsing, layout and OCR
    3. 3.3Tables, images and mixed content
    4. 3.4Cleaning, normalization and quality gates
    5. 3.5Metadata, document identity and versioning
    6. 3.6Updates, deletes, reindexing and ingestion pipelines
    7. Chapter overview
  4. 4Chunking
    1. 4.1Why we chunk, and the main chunking strategies
    2. 4.2Structure aware chunking and tables
    3. 4.3Chunk size, overlap and token based sizing
    4. 4.4Contextual chunking and late chunking
    5. 4.5Parent child and multi representation indexing
    6. 4.6Chunk metadata, provenance and the chunking debugging playbook
    7. Chapter overview
  5. 5Embeddings
    1. 5.1Embeddings and similarity from first principles
    2. 5.2What embeddings cannot do
    3. 5.3Choosing an embedding model: size, cost, language
    4. 5.4Embedding model migration and index rebuilds
    5. Chapter overview
  6. 6Vector stores and scores
    1. 6.1Vector databases, ANN indexes and choosing a store
    2. 6.2Score calibration and thresholds
    3. 6.3Late interaction and multivector retrieval
    4. Chapter overview
  7. 7Finding the right chunks
    1. 7.1Query understanding, rewriting and expansion
    2. 7.2Dense retrieval and keyword search with BM25
    3. 7.3Hybrid retrieval and rank fusion
    4. 7.4topK, no good match, and metadata filtering
    5. 7.5Routing, diversity, deduplication and caching
    6. 7.6Reranking: two stage, cross encoder and LLM rerankers
    7. Chapter overview
  8. 8When retrieval fails
    1. 8.1Retrieval failure analysis playbook
    2. Chapter overview
  9. 9Building the answer
    1. 9.1Context selection, packing and citations
    2. 9.2Grounding prompts and abstention
    3. 9.3Lost in the middle and context management
    4. 9.4Conflicts, staleness and freshness rules
    5. 9.5Hallucination checks and answer verification
    6. 9.6Conversational RAG, answer UX and RAG APIs
    7. Chapter overview
  10. 10Measuring
    1. 10.1Why measure, and what to measure
    2. 10.2Building a golden dataset
    3. 10.3Retrieval metrics: precision, recall, MRR, nDCG
    4. 10.4Faithfulness and answer correctness
    5. 10.5LLM as judge, and judging the judge
    6. 10.6Offline and online evals, regression tests and slices
    7. Chapter overview
  11. 11Safety and operations
    1. 11.1Prompt injection and poisoned documents
    2. 11.2Access control, privacy and governance
    3. 11.3Index lifecycle operations
    4. 11.4Observability and tracing
    5. 11.5Latency, cost, caching and model routing
    6. 11.6Reliability, degraded modes and launch readiness
    7. Chapter overview
  12. 12Scale and advanced RAG
    1. 12.1Scaling, multi tenancy and enterprise architecture
    2. 12.2Choosing an architecture: naive, advanced, agentic, graph
    3. 12.3Agentic RAG and RAG as an MCP tool
    4. 12.4GraphRAG, structured data and multimodal RAG
    5. Chapter overview

What RAG is: anatomy of a request

One question followed step by step through a RAG system, from the typed words to a cited answer, with what each step costs.

A real question

What actually happens between typing a question and getting an answer?

Four steps: the question becomes numbers, search finds the best pieces of your documents, those pieces are packed into a prompt, and the model answers from them.

The problem

Why this matters

Why not the simpler way

The simple picture of RAG is "search, then ask the model". That hides the steps where things go wrong. A wrong answer can come from the question being embedded badly, the search missing the passage, the prompt dropping it, or the model ignoring it. If you know each step, you know where to look.

The idea

The idea in plain words

Retrieval augmented generation answers a question in two moves: first retrieve the passages that are most likely to contain the answer, then generate an answer from those passages only, with citations.

It is an open book exam. The student still has to understand the question and write the answer, but the facts come from the pages they look up, and they can say which page.

How it works

Step by step

  1. The question goes through an embedding model and becomes a list of 384 numbers that capture its meaning.
    1

    Turn the question into numbers

    The question goes through an embedding model and becomes a list of 384 numbers that capture its meaning.

  2. Keyword search finds pieces with the same words, like E4012. Vector search finds pieces with a similar meaning. The two lists are merged.
    2

    Search your documents

    Keyword search finds pieces with the same words, like E4012. Vector search finds pieces with a similar meaning. The two lists are merged.

  3. The best three pieces are numbered as sources, under a short instruction, with the question at the end.
    3

    Pack the prompt

    The best three pieces are numbered as sources, under a short instruction, with the question at the end.

  4. It writes the answer from the sources only, with [1] or [2] next to each fact, or says it does not know.
    4

    The model answers and cites

    It writes the answer from the sources only, with [1] or [2] next to each fact, or says it does not know.

In detail

Follow one real question from the Mini RAG Lab, "What does error E4012 mean on the FiberBox?":

  1. Embed the question. The question goes through the same embedding model as the chunks, with the model's query prefix, and becomes a vector of 384 numbers.
  2. Search. BM25 finds chunks that share words such as E4012; vector search finds chunks with a similar meaning. The two lists are merged by rank.
  3. Pack the prompt. The top 3 chunks are numbered as sources and placed under a short instruction, with the question last.
  4. Generate. The model writes an answer that cites [1] or [2], or says it does not know.
  5. Return. The answer goes back with its sources, so the user can check the line it came from.
RAG pipeline: documents are loaded, chunked, embedded and indexed offline; at question time the index is searched, the best chunks are packed into the prompt, and the model writes a cited answer.OFFLINE · BUILD THE INDEXLoadPDF, HTML, MDChunk256–512 tokensEmbedtext → vectorIndexvectors + BM25ONLINE · ANSWER A QUESTIONQuestionuser asksRetrievetop k chunksPack promptwithin budgetGeneratecited answersearch the index
Quality is decided offline more often than online: a chunk that splits the answer cannot be rescued by a better prompt.

Try it · what the model actually sees

Loading 34 chunks…

The code

Try it in Python

def answer(question: str) -> dict:
    q_vec = embed(QUERY_PREFIX + question)            # 1. same model as the chunks
    hits = rrf(bm25.search(question), vectors.search(q_vec))[:3]   # 2. hybrid search
    prompt = build_prompt(question, hits)              # 3. numbered sources + question
    text = llm(prompt)                                 # 4. answer with [1], [2] citations
    log(question=question, hits=[h.id for h in hits], prompt_tokens=count(prompt), answer=text)
    return {"answer": text, "sources": [h.source for h in hits]}   # 5. return with sources

See it

What we measured

Measured · time and size per request

Mini RAG Lab logic on an Apple M4 laptop in Node: 34 chunks (recursive 512), bge-small-en-v1.5 q8, 47 golden questions.

Embed the questionbaseline2.9 ms
Hybrid search (BM25, vectors, RRF)2.9 ms0.03 ms
Prompt size, median-1,078 tokens

Measured 2026-11-03. Search is fast because the index is tiny; it grows with the corpus, which is what vector databases are for. Not measured yet. Planned: model call latency with a real key.

Try it on a real example

Follow one real question: “What does error E4012 mean on the FiberBox?”

Step
1.FiberBox manual, chunk 0 · fused 0.033The manual's introduction and first sections.–
2.FiberBox manual, chunk 2 · fused 0.032…| E4012 | Mesh pairing failed | Mesh extender on old firmware |…✓
3.Network bulletin 2, chunk 0 · fused 0.016A service bulletin about the network.–

The error table is found, in second place. Both keyword and vector search ranked it second.

Choosing

Use it when, avoid it when

Use it when
  • Questions over documents that the model has not seen
  • Any answer that must show where it came from
Avoid it when
  • Tasks with no documents behind them, such as brainstorming
  • Questions that need a calculation over all records: use a database query instead

What it costs · Per request: one small embedding, a search and one model call. The model call dominates both cost and latency.

In production

Log every step of every request: the question, the chunk ids retrieved with their scores, the final prompt size and the answer with its citations. When a user reports a bad answer, these logs tell you in a minute which step failed. Without them you can only guess, and guessing usually blames the model.

Retrieval is cheap; generation is the slow and costly step. Spend effort on getting the right 3 chunks rather than sending 20 and hoping.

Watch out

Common mistakes

  • RAG is one search call. It is five steps, and each one can fail on its own.
  • The model searches the documents. Your code searches; the model only sees what you put in the prompt.
  • More chunks give better answers. They add cost and can push the right passage into the middle where it is used less.

Check yourself

Remember this

Remember these

  • RAG is retrieve first, then generate from what was retrieved.
  • The question must use the same embedding model as the documents.
  • Search returns pieces (chunks), not whole documents.
  • The prompt asks for citations, so every answer can be checked.

What our numbers can and cannot show. Every measurement in this course runs on one small test set: 25 Polder documents cut into 34 chunks, and 47 questions with a known answer. A hit means a retrieved chunk contains the answer text. Many questions reuse the documents' own words, which helps keyword search, and only 2 need more than one document. On a set this small, several methods often tie, so a tie here does not mean they tie on your data. Treat the numbers as a worked example of how to measure, then measure on your own documents.

Say it in an interview

  1. 01RAG retrieves passages first, then generates an answer from them with citations.
  2. 02A request is: embed the question, search, pack the prompt, generate, return with sources.
  3. 03The question must use the same embedding model as the chunks, or the vectors do not match.
  4. 04On my laptop, embedding took 2.9 ms and search 0.03 ms; the model call dominates.
  5. 05I log chunk ids and scores per request so I can tell retrieval misses from model errors.

Quick quiz

Three questions

Question 1

Q1

In the measured request, which step takes the most time?

Question 2

Q2

Why is the question embedded with the same model as the chunks?

Question 3

Q3 · from RAG-01

A model answers fluently about a document it was never given. What is that called?

Saved on this device. No account needed.