Chapter 12 · concept 1 of 4 · show all
2Tokens, cost and your first RAG
4Chunking
- 4.1Why we chunk, and the main chunking strategies
- 4.2Structure aware chunking and tables
- 4.3Chunk size, overlap and token based sizing
- 4.4Contextual chunking and late chunking
- 4.5Parent child and multi representation indexing
- 4.6Chunk metadata, provenance and the chunking debugging playbook
- Chapter overview
7Finding the right chunks
8When retrieval fails
Scaling, multi tenancy and enterprise architecture
What actually changes from 34 chunks to millions: exact search time grows linearly (30 ms at 100,000 vectors on a laptop), memory grows to gigabytes, and isolation between customers becomes the hard part.
A real question
Our lab has 34 chunks. What changes at a million?
Less than you might think for search, more for operations. Exact search grows linearly and stays fast for a long time; memory, rebuild time and isolation between customers are what force new architecture.
The problem
Why this matters
Exact search stops working once you have real data.
On a laptop, plain dot products over 384-number vectors took 0.5 ms for 1,000 vectors, 3.4 ms for 10,000 and 30 ms for 100,000. Memory is the sooner limit: a million float vectors need about 1.5 GB.
Why not the simpler way
Architecture diagrams for "enterprise RAG" are full of boxes. Many are only needed at a scale most projects never reach, and some needs, like tenant isolation, arrive on day one of a second customer. Knowing the numbers keeps the design proportionate.
The idea
The idea in plain words
Scaling covers corpus size, query volume and update rate. Multi-tenancy means serving several customers or departments from one system without their data mixing. Enterprise architecture adds identity, audit, data residency and operations to the pipeline.
An office building grows floor by floor. The lifts need upgrading eventually, but the locks on each company's door are needed from the first tenant.
How it works
Step by step
- 1
Do the arithmetic
Vectors × dimensions × bytes. 1 million × 384 × 4 bytes is about 1.5 GB, before metadata and the keyword index. Compression cuts it (RAG-22).
- 2
Partition
Shard by size, or namespace per tenant. Search only the partitions a request may see.
- 3
Isolation first
Tenant id on every chunk, enforced in the index query, tested with cross-tenant golden items.
In detail
Corpus size. Exact search to roughly 100,000 vectors per partition on modest hardware; then ANN (HNSW, IVF) and compression (int8, binary with rescoring) (RAG-22).
Query volume. Replicas for search; batching for the reranker and embedder; caching of stable results (RAG-30).
Update rate. Incremental pipelines with queues; per-partition rebuilds (RAG-47).
Tenancy models. Index per tenant (strong isolation, more indexes), namespace per tenant in a shared store, or a shared index with a mandatory tenant filter (cheapest, easiest to get wrong).
Enterprise needs. Single sign-on with roles (RAG-46), audit logs, data residency, key management, and per-tenant cost reporting.
The code
Try it in Python
def plan(n_chunks: int, dims: int = 384, bytes_per: int = 4, replicas: int = 2) -> dict:
gb = n_chunks * dims * bytes_per / 1e9
return {"vectors_gb": round(gb, 2), "with_replicas_gb": round(gb * replicas, 2),
"int8_gb": round(gb / 4, 2), "binary_gb": round(gb / 32, 3),
"exact_ok": n_chunks <= 100_000}
def search(tenant_id: str, query_vec, index):
part = index.partition(tenant_id) # namespace per tenant
return part.search(query_vec, filter={"tenant_id": tenant_id}) # and an explicit filter
print(plan(1_000_000))
See it
What we measured
Measured · exact search time per query and float32 memory
Random unit vectors, 384 dimensions, Float32Array dot products in Node on a laptop CPU, average of 5 to 20 runs. Timing only; quality is not tested with random vectors.
Measured 2026-11-04 with corpus/scripts/measure-part-8.mts. Memory is calculated (vectors × 384 × 4 bytes), without metadata or the keyword index.
Try it on a real example
Exact search time and float32 memory as the index grows (random vectors, laptop CPU)
A small team knowledge base.
Choosing
Use it when, avoid it when
- Exact search until measurement says otherwise
- Tenant id on every chunk, enforced in the query
- Arithmetic on memory and rebuild time before choosing infrastructure
- Distributed vector clusters for thousands of chunks
- A shared index with tenant filtering only in application code
What it costs · Partitioning adds operational work; isolation tests are cheap and essential.
In production
Write cross-tenant tests: the same question asked by two tenants must never cite the other's documents, checked on every build. Track per-tenant index size, query volume and cost. Plan rebuild time per partition; at a million chunks, re-embedding is hours of compute, and that is what makes model migrations expensive.
Watch out
Common mistakes
- You need ANN from the start. Exact search took 30 ms at 100,000 vectors here.
- Multi-tenancy is a scale problem. It is a security problem from the second customer.
- Scaling RAG means scaling the vector database. Rebuilds, the reranker and the model scale too.
Check yourself
Remember this
Remember these
- Exact search grows linearly: 30 ms at 100,000 vectors on a laptop.
- Memory limits arrive first: about 1.5 GB per million float vectors.
- Partition by tenant and enforce the tenant in every query.
- Measure ANN recall against exact search before switching.
Say it in an interview
- 01Exact search grows linearly; on a laptop it took 30 ms over 100,000 vectors.
- 02Memory limits come first: about 1.5 GB per million 384-dimension float vectors.
- 03I partition by tenant and enforce the tenant id in every index query.
- 04Cross-tenant golden items run on every build.
- 05Re-embedding time at scale is what makes model migrations expensive.
Quick quiz