RAGEngineering Hub
Menu
Chapter 6 · concept 1 of 3 · show all
  1. 1What RAG is, and why
    1. 1.1Why models need outside knowledge, and what hallucination means
    2. 1.2What RAG is: anatomy of a request
    3. 1.3RAG vs fine tuning vs long context and prompt caching
    4. 1.4Use cases and the quality, latency and cost triangle
    5. Chapter overview
  2. 2Tokens, cost and your first RAG
    1. 2.1Tokens and context windows for RAG
    2. 2.2The RAG lifecycle and your first RAG in miniature
    3. Chapter overview
  3. 3Getting documents in
    1. 3.1Content sources and file formats
    2. 3.2Parsing, layout and OCR
    3. 3.3Tables, images and mixed content
    4. 3.4Cleaning, normalization and quality gates
    5. 3.5Metadata, document identity and versioning
    6. 3.6Updates, deletes, reindexing and ingestion pipelines
    7. Chapter overview
  4. 4Chunking
    1. 4.1Why we chunk, and the main chunking strategies
    2. 4.2Structure aware chunking and tables
    3. 4.3Chunk size, overlap and token based sizing
    4. 4.4Contextual chunking and late chunking
    5. 4.5Parent child and multi representation indexing
    6. 4.6Chunk metadata, provenance and the chunking debugging playbook
    7. Chapter overview
  5. 5Embeddings
    1. 5.1Embeddings and similarity from first principles
    2. 5.2What embeddings cannot do
    3. 5.3Choosing an embedding model: size, cost, language
    4. 5.4Embedding model migration and index rebuilds
    5. Chapter overview
  6. 6Vector stores and scores
    1. 6.1Vector databases, ANN indexes and choosing a store
    2. 6.2Score calibration and thresholds
    3. 6.3Late interaction and multivector retrieval
    4. Chapter overview
  7. 7Finding the right chunks
    1. 7.1Query understanding, rewriting and expansion
    2. 7.2Dense retrieval and keyword search with BM25
    3. 7.3Hybrid retrieval and rank fusion
    4. 7.4topK, no good match, and metadata filtering
    5. 7.5Routing, diversity, deduplication and caching
    6. 7.6Reranking: two stage, cross encoder and LLM rerankers
    7. Chapter overview
  8. 8When retrieval fails
    1. 8.1Retrieval failure analysis playbook
    2. Chapter overview
  9. 9Building the answer
    1. 9.1Context selection, packing and citations
    2. 9.2Grounding prompts and abstention
    3. 9.3Lost in the middle and context management
    4. 9.4Conflicts, staleness and freshness rules
    5. 9.5Hallucination checks and answer verification
    6. 9.6Conversational RAG, answer UX and RAG APIs
    7. Chapter overview
  10. 10Measuring
    1. 10.1Why measure, and what to measure
    2. 10.2Building a golden dataset
    3. 10.3Retrieval metrics: precision, recall, MRR, nDCG
    4. 10.4Faithfulness and answer correctness
    5. 10.5LLM as judge, and judging the judge
    6. 10.6Offline and online evals, regression tests and slices
    7. Chapter overview
  11. 11Safety and operations
    1. 11.1Prompt injection and poisoned documents
    2. 11.2Access control, privacy and governance
    3. 11.3Index lifecycle operations
    4. 11.4Observability and tracing
    5. 11.5Latency, cost, caching and model routing
    6. 11.6Reliability, degraded modes and launch readiness
    7. Chapter overview
  12. 12Scale and advanced RAG
    1. 12.1Scaling, multi tenancy and enterprise architecture
    2. 12.2Choosing an architecture: naive, advanced, agentic, graph
    3. 12.3Agentic RAG and RAG as an MCP tool
    4. 12.4GraphRAG, structured data and multimodal RAG
    5. Chapter overview

Vector databases, ANN indexes and choosing a store

When exact search is enough, what approximate indexes like HNSW trade away, and how compression shrinks vectors 32 times, with measured recall on our corpus.

A real question

Do I need a vector database before I can build RAG?

Not at first. For thousands of chunks, comparing the question with every vector is fast and exact. Approximate indexes and databases matter when you reach millions of vectors, many users or frequent updates.

The problem

Why this matters

RAG starts with choosing and paying for a vector database.
The belief

RAG starts with choosing and paying for a vector database.

Our labs keep 34 chunk vectors in a JSON file. An exact search over all of them took about 16 microseconds. Storage choice is a scaling decision, not a starting one.
What really happens

Our labs keep 34 chunk vectors in a JSON file. An exact search over all of them took about 16 microseconds. Storage choice is a scaling decision, not a starting one.

Why not the simpler way

Tutorials often start by installing a vector database. That adds a service to run, a cost, and settings you do not yet understand, before you know whether retrieval even works. The real questions are how many vectors you have, how fast answers must be, how often data changes, and what else you need to filter on.

The idea

The idea in plain words

A vector store keeps vectors and finds the nearest ones to a query. Exact search compares with all of them. Approximate nearest neighbour (ANN) indexes organise the vectors so a search only visits a few. Compression stores each vector in fewer bytes.

Finding a friend's house: you could knock on every door in the city (exact), or ask for the neighbourhood, then the street, then the number (approximate, and almost always right).

How it works

Step by step

  1. Compare the question with every vector. Always correct. Cost grows with the number of vectors.
    1

    Exact search

    Compare the question with every vector. Always correct. Cost grows with the number of vectors.

  2. An index such as HNSW links each vector to its neighbours and walks that graph. It checks a small share of vectors and sometimes misses the true best.
    2

    Approximate search (ANN)

    An index such as HNSW links each vector to its neighbours and walks that graph. It checks a small share of vectors and sometimes misses the true best.

  3. Store small codes (int8 or 1 bit per number) to save memory, then recheck the top candidates with full vectors.
    3

    Compress, then rescore

    Store small codes (int8 or 1 bit per number) to save memory, then recheck the top candidates with full vectors.

In detail

HNSW (Malkov and Yashunin, arXiv 1603.09320) builds layers of a graph where each vector links to near neighbours. A search starts at the top layer and greedily moves closer. Settings like M (links per node) and ef (how many candidates to keep while searching) trade memory and time for recall.

IVF splits vectors into clusters and searches only the clusters nearest the query. Fewer clusters searched means faster and less exact.

Quantization. int8 rounds each number to 256 levels; binary keeps only its sign and compares with Hamming distance. Rescoring the top candidates with the original vectors recovers most of the loss.

What a store must also do. Metadata filters (version, language, access role), deletes and updates without a full rebuild (RAG-12), backups, and hybrid search with keywords (RAG-28).

The code

Try it in Python

import numpy as np

def to_binary(v: np.ndarray) -> np.ndarray:
    return np.packbits(v > 0, axis=-1)            # 384 floats -> 48 bytes

def search_binary_rescore(q, docs_f32, docs_bin, k=3, candidates=10):
    qb = to_binary(q)
    ham = np.unpackbits(docs_bin ^ qb, axis=-1).sum(axis=1)   # lower = closer
    cand = np.argsort(ham)[:candidates]
    exact = docs_f32[cand] @ q                                 # rescore with full vectors
    return cand[np.argsort(-exact)[:k]]

See it

What we measured

Measured · evidence recall@3 by storage format

Polder corpus, recursive 512 token chunks (34), bge-small vectors, 384 dims. int8: per-dimension scale; binary: sign bits with Hamming distance; rescore: binary top 10 reordered with float vectors. Timing: plain TypeScript on a laptop, 200 runs.

float32, 1,536 bytes44 of 47
int8, 384 bytes4444 of 47
binary, 48 bytes4437 of 47
binary + rescore top 103744 of 47
Exact search time per queryabout 14 µs

Measured 2026-11-04 with corpus/scripts/measure-parts-4-7.mts. ANN indexes were not measured: with 34 vectors there is nothing for them to speed up.

Try it on a real example

Same 34 chunks, same 47 questions, vectors stored four ways

Store as
1.1,536 bytes per vector · 44 of 47384 numbers, 4 bytes each.✓

The baseline: full precision.

Choosing

Use it when, avoid it when

Use it when
  • Exact search for small corpora and for testing ANN recall
  • int8 or binary plus rescoring when memory is the limit
Avoid it when
  • A dedicated vector database before measuring a need
  • Changing ANN settings without re-running the golden set

What it costs · Exact search: compute grows with corpus size. ANN: extra memory for the graph and tuning time. Compression: almost free, with rescoring.

In production

Start with what you already run: a Postgres extension, a search engine with vector support, or an in-memory array. Move to a dedicated store when measured latency or memory says so. Always test recall of the ANN index against exact search on your golden set after changing index settings: a fast index that misses answers is a quality bug, and nobody gets an error message. Filtering inside an ANN index can also drop recall when the filter is very selective; test that case.

Watch out

Common mistakes

  • ANN is always accurate. It is approximate by design; measure recall against exact search.
  • Compression ruins quality. int8 lost nothing here, and binary with rescoring lost nothing.
  • A vector database is the RAG system. It is one component; chunking and evaluation matter more.

Check yourself

Remember this

Remember these

  • Exact search is fine for thousands to tens of thousands of vectors.
  • ANN indexes like HNSW trade a little recall for speed at large scale.
  • int8 kept every answer here at a quarter of the memory.
  • Binary plus rescoring kept every answer at 1/32 of the memory.

Say it in an interview

  1. 01At small scale I use exact search; it is correct and fast, 16 microseconds over 34 vectors.
  2. 02HNSW trades a little recall for speed; I test it against exact search on the golden set.
  3. 03int8 kept all 44 answers at a quarter of the memory.
  4. 04Binary vectors lost 4 answers, and rescoring the top 10 with floats brought them back.
  5. 05I choose a store for filters, updates and operations, not only for search speed.

Quick quiz

Three questions

Question 1

Q1

Our 34 chunks take about 16 microseconds per exact search. What does that suggest?

Question 2

Q2

Binary vectors (48 bytes) found 37 of 47. What brought it back to 44?

Question 3

Q3 · from RAG-19

What does cosine similarity compare?

Saved on this device. No account needed.