Chapter 1 · concept 3 of 4 · show all
2Tokens, cost and your first RAG
4Chunking
- 4.1Why we chunk, and the main chunking strategies
- 4.2Structure aware chunking and tables
- 4.3Chunk size, overlap and token based sizing
- 4.4Contextual chunking and late chunking
- 4.5Parent child and multi representation indexing
- 4.6Chunk metadata, provenance and the chunking debugging playbook
- Chapter overview
7Finding the right chunks
8When retrieval fails
RAG vs fine tuning vs long context and prompt caching
Four ways to give a model your knowledge, what each one is good at, and what they cost per question on the same corpus.
A real question
Should I train the model on my documents, paste them all in, or use RAG?
It depends on three numbers: how big your documents are, how often they change, and how many questions you get. Each option wins somewhere.
The problem
Why this matters
To make a model know our documents, we have to train it on them.
Training (fine tuning) shapes style and format. For facts that change, giving the model the right page at question time is faster to update and can cite its source.
Why not the simpler way
The question "should we use RAG?" often gets a habit for an answer. Some teams build a vector database for twenty documents. Others try to fine tune facts into a model and then cannot update them. With context windows of a million tokens, a third group pastes everything into every prompt. Each choice fits some cases and is expensive in others.
The idea
The idea in plain words
There are four ways to give a model knowledge. Fine tuning changes the model's weights with your examples. Long context puts all your documents in the prompt. Prompt caching does the same but keeps the processed prompt on the provider's side, so repeats cost less. RAG puts only the few passages that matter in the prompt.
Compare it to preparing for questions. Fine tuning is studying for months. Long context is bringing the whole library to every meeting. Caching is leaving the library in the meeting room. RAG is bringing the three pages you need.
How it works
Step by step
1Fine tuning: teach a behaviour
You change the model's weights with examples. Good for tone and format; slow to update and cannot cite a source.
2Long context: paste everything
Simple when the documents are small. You pay for every token on every question, and facts in the middle are used less well.
3Cache it, or retrieve a few pages
A prompt cache makes the pasted part cheap while it stays warm. RAG sends only the few chunks that matter.
In detail
Fine tuning is strong for behaviour: tone, output format, a narrow task. For new facts it is weaker. In a peer-reviewed comparison, retrieval beat unsupervised fine tuning for injecting new knowledge (Ovadia et al., EMNLP 2024). It also cannot cite a source, and every update means another training run.
Long context is the simplest when the corpus is small and stable. Cost and latency grow with every token on every call, and facts in the middle of a long prompt are used less well. Caching removes most of the cost of the repeated part, while the cache is warm. Loading a small knowledge base into the cache instead of retrieving has been called cache augmented generation (Chan et al., arXiv 2412.15605, 2024).
RAG keeps the prompt small and the facts updatable: change a document, re-index its chunks, and the next answer uses it. The cost is the retrieval system you have to build and measure.
The code
Try it in Python
def cost_per_question(prompt_tokens, cached_tokens=0, price_in=4.0, price_cached=0.20):
"""USD per question for the input side, prices per million tokens."""
return ((prompt_tokens - cached_tokens) * price_in + cached_tokens * price_cached) / 1e6
corpus, question, rag_prompt = 7592, 52, 1078
print(f"long context {cost_per_question(corpus + question):.4f}")
print(f"RAG {cost_per_question(rag_prompt):.4f}")
print(f"cached corpus {cost_per_question(corpus + question, cached_tokens=corpus):.4f}")
See it
What we measured
Measured · input cost per question, Claude Opus 5.5 list prices
Polder corpus, 7,592 tokens. Prices $4 per million input tokens and $0.20 per million cached input tokens (vendor price list, checked 2026-11-03). Cache writes and output tokens excluded. Tokens counted with o200k, not the priced model's tokenizer, so costs are estimates.
Calculated 2026-11-03 from measured token counts. Caching wins on this tiny corpus only while the cache is warm; a cold call pays the full write. At 10,000 documents of this size (about 3 million tokens) the corpus no longer fits a 1M window and only RAG remains.
Try it on a real example
Input cost per question on our 7,592 token corpus (Claude Opus 5.5 list prices, checked 2026-11-03)
About 7,650 tokens per question at $4 per million
The simplest option, and the most expensive one here.
Choosing
Use it when, avoid it when
- RAG: large or changing corpora, citations, per-user access
- Caching: a small, stable corpus with steady traffic
- Fine tuning: style, format and narrow skills
- Fine tuning to add facts that change
- Long context without caching for high traffic
- A vector database for a corpus that fits in one cached prompt
What it costs · Fine tuning costs training runs per update. Long context costs every token on every call. Caching costs a write per refresh. RAG costs an index and an evaluation set to keep it honest.
In production
Decide with three numbers: corpus size in tokens, how often it changes, and questions per day. A corpus that fits comfortably in the window, changes rarely and gets steady traffic is a good fit for caching. A corpus that is large, changes daily or needs per-user access control points to RAG. Many systems combine them: fine tune for format, retrieve for facts, cache the fixed instruction.
Watch out
Common mistakes
- Long context makes RAG obsolete. It moves the line. Above the window size, or with daily changes and access rules, retrieval is still needed.
- Fine tuning teaches the model your documents. It learns patterns from them; it cannot quote or cite them reliably.
- RAG is always the cheapest. On a tiny corpus with a warm cache it was not.
Check yourself
Remember this
Remember these
- Fine tuning changes behaviour; it is a poor way to add facts that change.
- Long context is simple, but you pay for every token on every call.
- Caching makes repeated context cheap while it stays warm.
- RAG scales to large, changing documents and gives citations.
Say it in an interview
- 01Four options: fine tuning, long context, prompt caching and RAG.
- 02Fine tuning is for behaviour; retrieval is for facts that change and need citations.
- 03On my 7,592 token corpus, a warm cache cost $0.0017 a question against $0.0043 for RAG.
- 04Above the window size, or with daily updates and access rules, RAG is the only option.
- 05I decide with corpus tokens, change rate and traffic, not with a habit.
Quick quiz