Chapter 5 · concept 1 of 4 · show all
2Tokens, cost and your first RAG
4Chunking
- 4.1Why we chunk, and the main chunking strategies
- 4.2Structure aware chunking and tables
- 4.3Chunk size, overlap and token based sizing
- 4.4Contextual chunking and late chunking
- 4.5Parent child and multi representation indexing
- 4.6Chunk metadata, provenance and the chunking debugging playbook
- Chapter overview
7Finding the right chunks
8When retrieval fails
Embeddings and similarity from first principles
How a model turns text into a list of numbers, why close numbers mean close meaning, and how cosine similarity ranks chunks, measured on our 34 chunks.
A real question
How can a computer tell that “reboot the router” and “restart my modem” mean the same thing?
An embedding model turns each text into a list of numbers. Texts with similar meaning get similar numbers, so search can compare meaning instead of exact words.
The problem
Why this matters
Search only works when the question and the document use the same words.
“How do I restart my modem?” and “Steps to reboot the router” share no words, yet their vectors scored 0.80. An unrelated pair scored 0.39.
Why not the simpler way
Keyword search matches words. People do not always use the words in the document: they say "modem" when the manual says "router", or "money back" when the policy says "reimbursement". Synonym lists help a little and never end. You want search to match on meaning.
The idea
The idea in plain words
An embedding model is a neural network trained so that texts with similar meaning produce similar vectors. A vector is just a list of numbers, here 384 of them. Each chunk becomes one point in a 384-dimensional space, and the question becomes another point. Retrieval finds the chunk points nearest the question point.
A library sorts books by topic, not by title. Books about bicycles sit together even if one says "bike" and another says "cycling".
How it works
Step by step
1Text becomes numbers
The model reads a chunk and outputs 384 numbers, a vector. It does this once per chunk, when you build the index.
2Compare directions
The question is turned into a vector too. Cosine similarity measures how close the two point: 1 is the same direction, 0 is unrelated.
3Sort and keep the top
Score every chunk, sort from high to low, and pass the best few to the model.
In detail
Cosine similarity is the dot product of two vectors divided by their lengths: cos(a, b) = (a · b) / (|a| |b|). It ranges from -1 to 1. Our model normalizes every vector to length 1, so cosine and dot product give the same number and the cheaper dot product is enough.
Asymmetric prefixes. Some models want a hint on the query. bge-small expects "Represent this sentence for searching relevant passages: " in front of questions and nothing in front of passages. Leaving it out silently lowers quality.
Scores are relative. On our corpus even the least related chunk scored 0.395 for the E4012 question. A score means "closer than the others", not "relevant". Thresholds are their own topic (RAG-23).
The code
Try it in Python
from sentence_transformers import SentenceTransformer
import numpy as np
model = SentenceTransformer("BAAI/bge-small-en-v1.5")
PREFIX = "Represent this sentence for searching relevant passages: "
def search(question: str, chunks: list[str], k: int = 3):
docs = model.encode(chunks, normalize_embeddings=True) # once, at index time
q = model.encode(PREFIX + question, normalize_embeddings=True) # per question
scores = docs @ q # dot product = cosine here
best = np.argsort(-scores)[:k]
return [(chunks[i], float(scores[i])) for i in best]
See it
What we measured
Measured · vector search on 47 golden questions
Polder corpus, recursive 512 token chunks (34), Xenova/bge-small-en-v1.5 q8, CLS pooling, normalized, query prefix.
Measured 2026-11-04 with corpus/scripts/measure-parts-4-7.mts. The paraphrase and unrelated pairs are sentences we wrote for the test.
Try it on a real example
“What does error E4012 mean on the FiberBox?” scored against all 34 chunks
The table is found, but second: the introduction talks about error codes in general and scores 0.025 higher. Similar topic is not the same as the answer.
Choosing
Use it when, avoid it when
- Questions phrased differently from the documents
- Many languages or synonyms in the corpus
- Exact codes, IDs and numbers on their own: add keyword search (RAG-27)
- Reading a single score as relevant or not
What it costs · One embedding per chunk at index time, one per question at query time. A small local model runs on a CPU.
In production
Embedding is an index-time cost per chunk and a query-time cost per question. With a small local model, our 34 chunks took about 56 ms each on a laptop CPU. The bigger risk is quiet mismatch: a different model, a missing prefix or a changed normalization makes every score wrong without any error message. Store the model name and settings next to the vectors.
Watch out
Common mistakes
- A high score means relevant. Unrelated chunks scored around 0.4 here, never 0.
- Any two embedding models are interchangeable. Each model has its own space; vectors from two models cannot be compared (RAG-25).
- Cosine and dot product differ. For normalized vectors they are the same number.
Check yourself
Remember this
Remember these
- An embedding is a fixed-length list of numbers for a piece of text: 384 for our model.
- Cosine similarity compares directions; with length-1 vectors it equals the dot product.
- Use the same model for chunks and questions, or the numbers mean nothing.
- On our corpus, the right chunk came first for 30 of 47 questions and in the top 3 for 44.
Say it in an interview
- 01An embedding model maps text to a vector so similar meaning lands close together.
- 02I rank chunks by cosine similarity; with normalized vectors that is a dot product.
- 03The same model and prefix must embed both chunks and queries.
- 04Scores are relative: on my corpus unrelated chunks still scored about 0.4.
- 05With bge-small on 34 chunks, the right chunk was in the top 3 for 44 of 47 questions.
Quick quiz