Chapter 2 · concept 1 of 2 · show all

Tokens and context windows for RAG

What a token is, why every limit and price in RAG is counted in tokens, and how to budget a prompt so the answer still fits.

A real question

Why is my prompt “too long” when it only has a few paragraphs?

Models do not count words. They count tokens, small pieces of text, and codes and numbers cost more pieces than you expect.

The problem

Why this matters

The belief

One word is one unit, so a 500 word chunk is 500 of whatever the model counts.

What really happens

Models count tokens. Common words are one token; codes split into several. The error code E4012 alone is 3 tokens.

Why not the simpler way

The simple way is to count in words or characters, because that is what you can see. Then a prompt that "looked short" gets cut off, a chunk of "500 words" is too long for the embedding model, and the bill does not match your estimate. Models never see words. Every limit and every price is in tokens, so you have to count the same way.

The idea

The idea in plain words

A token is a small piece of text from the model's fixed vocabulary. Common words are one token. Rare words, codes and numbers split into several. The context window is how many tokens one call can hold, the prompt and the answer together.

Think of it as luggage with a weight limit. You pay by the kilo, not by the number of items, and one heavy item can matter more than three light ones.

How it works

Step by step

1
Text is cut into tokens
A tokenizer splits text into pieces from a fixed vocabulary and turns each piece into a number.
2
One window holds everything
The context window is a token budget for the whole call: instruction, documents, question and the answer.
$Price is per token, on every question. Three chunks cost a fraction of pasting the whole corpus.$
3
Every token costs
Price is per token, on every question. Three chunks cost a fraction of pasting the whole corpus.

In detail

A tokenizer turns text into integer ids. The model reads ids, attends over all of them, and writes new ids one at a time. On our corpus the o200k tokenizer gives 1.39 tokens per word: 7,592 tokens for 5,452 words. An error code like E4012 is 3 tokens; a plan id like PT-MOB-25GB is 6.

Median prompt in the Mini RAG Lab: 1,078 tokens across 47 golden questions (recursive 512 chunks, top 3). The answer reserve is the max_tokens the lab sends.

A RAG prompt has four parts that share one budget: the instruction, the retrieved chunks, the question, and room for the answer. The chunks are almost all of it. That is why chunk size and top k are the two settings that move cost the most.

The code

Try it in Python

import tiktoken
enc = tiktoken.get_encoding("o200k_base")

def tokens(text: str) -> int:
    return len(enc.encode(text))

def pack(chunks: list[str], budget: int) -> list[str]:
    """Add the best ranked chunks until the context budget is full."""
    out, used = [], 0
    for c in chunks:
        n = tokens(c)
        if used + n > budget:
            break
        out.append(c)
        used += n
    return out

print(tokens("E4012"), tokens("firmware"))  # 3 2

See it

What we measured

Measured · input tokens per question

Polder corpus (25 documents, 7,592 tokens) and the 47 golden questions. RAG prompt: instruction, top 3 fused chunks (recursive 512) and the question, counted with o200k.

Whole corpus in every promptbaselineabout 7,650

RAG prompt, medianabout 7,6501,078

RAG prompt, largestabout 7,6501,503

Measured 2026-11-03 with the Mini RAG Lab logic. The corpus fits in one window, so RAG wins here on cost per call (about 7 times less input) and on citations, not on fit. At 10,000 documents of this size the corpus would be about 3 million tokens and would no longer fit a 1M window.

Try it on a real example

See how three real sentences split into tokens

Sentence

11 words → 11 tokens

The · router · rest · arts · itself · when · the · firmware · update · finishes · .

Plain English is close to one token per word. Even here, “restarts” is two pieces.

Choosing

Use it when, avoid it when

Use it when

Every size, limit and budget in RAG: count in tokens with the model's own tokenizer
Cost estimates: input tokens times price, plus the answer reserve

Avoid it when

Characters or words as a chunk size
Filling the window just because it is large: cost and middle-of-context misses grow

What it costs · Input cost scales with chunk size times top k on every call. At Claude Haiku 5.5 list prices ($0.10 per million input tokens, vendor price list checked 2026-11-03), 1,000 questions a day cost about $0.11 of input with RAG and about $0.76 with the whole corpus. Token counts here are o200k; Claude's tokenizer gives different counts, so treat costs as estimates.

In production

Count tokens with the tokenizer of the model you call, not a rule of thumb; model families split text differently. Set a token budget for retrieved context and stop adding chunks when it is full. Keep a fixed answer reserve so a long context never cuts the answer short.

A bigger window is not free accuracy. In peer-reviewed tests, models used facts at the start and end of a long context better than facts in the middle (Liu et al., "Lost in the Middle", TACL 2024). Fewer, better chunks often beat more chunks.

Watch out

Common mistakes

A token is a word. On our corpus it is 1.39 tokens per word, and more for codes and ids.
A 1M window means retrieval is obsolete. It changes when you need retrieval, not whether: cost per call, latency and citations still favour a small prompt.
The window is only for the prompt. The answer comes out of the same budget.

Check yourself

Remember this

Remember these

A token is a piece of text, often part of a word.
On our documents: 1.39 tokens per word on average, more for codes.
The context window holds the prompt and the answer together.
Count tokens with the model's own tokenizer, not a rule of thumb.

Say it in an interview

01Models read tokens, so every size, limit and price in RAG is in tokens.
02On my corpus the tokenizer gives 1.39 tokens per word, and codes split further.
03A prompt is instruction, chunks, question and an answer reserve in one budget.
04My RAG prompt is a median 1,078 tokens against about 7,650 for the whole corpus.
05Long windows do not remove retrieval: cost, latency and lost-in-the-middle remain.

Try it yourself: Token LabPaste text and see how a model splits it into tokens, and why sizes are counted in tokens.

Quick quiz

Three questions

Your corpus is 7,592 tokens. Why use retrieval instead of pasting all of it into every prompt?

Why does the error code E4012 cost 3 tokens when the word firmware costs 2?

What has to fit inside the context window of one call?

Saved on this device. No account needed.