RAG Efficiency

Caching reused work you’d already paid for. RAG — Retrieval-Augmented Generation — is about what you put in the prompt in the first place when the model needs knowledge it doesn’t have. The naive instinct is to stuff the whole document in. RAG’s whole premise is that you can stuff less and get a better answer — and “less” is exactly the throughline lever: fewer input tokens per call, less prefill latency, fewer dollars.

What RAG is, in one breath

question ──embed──▶ search a vector store ──▶ top-k relevant chunks ──┐
                                                                       ├─▶ prompt ─▶ model ─▶ answer
                                              (stuff chunks + question)┘

You chunk your knowledge base, embed each chunk, and at query time retrieve the few chunks most relevant to the question and paste them into the prompt. The model answers from that injected context instead of from its training memory.

Why retrieve at all — three things it beats

RAG isn’t just for “knowledge the model lacks.” It’s a cost strategy versus two alternatives:

• Beats stuffing the whole document. A 50-page manual is ~30,000 tokens. If every question only needs two paragraphs, sending all 30,000 tokens per call is paying for 99% dead weight — on every call.
• Beats brute-force long context. Even with a 1M-token window, filling it triggers the lost-in-the-middle problem and maximal prefill cost. Retrieving the relevant slice is both cheaper and more accurate than dumping everything and hoping attention finds it.
• Beats fine-tuning for fast-changing facts. Update the index, not the weights.

Note

The mental shift: context is a budget, not a backpack. RAG spends that budget on the few tokens that actually answer the question.

The cost levers

top-k: fewer, better chunks

k is how many chunks you retrieve and stuff. It’s the most direct token knob in RAG. Every extra chunk is extra input tokens on every call.

Worked example. Chunks are ~400 tokens. Question + system prompt ~300 tokens. Compare k=3 vs k=20:

top-3:   300 + 3 × 400  = 1,500 input tokens/call
top-20:  300 + 20 × 400 = 8,300 input tokens/call

top-20 costs ~5.5× the input tokens of top-3 — and prefills ~5.5× slower. If top-3 already contains the answer (it usually does, with good retrieval), the extra 17 chunks are pure cost and extra noise the model can get “lost” in. Smaller k is often both cheaper and more accurate. That’s the rare free lunch — take it.

Chunk size

Chunks too large waste tokens (you pull a whole section to answer one sentence) and blur relevance. Chunks too small fragment ideas across many chunks, forcing k up to reassemble them. The sweet spot — often a few hundred tokens with slight overlap — minimizes tokens retrieved per answer. Chunk size and k are coupled: right-sized chunks let you drop k.

Reranking: send less, but better

Embedding search is fast but coarse. A reranker is a second, sharper model that re-scores an initial candidate set and keeps only the genuine best. The pattern:

retrieve top-20 (cheap embedding search)
   └─▶ rerank ──▶ keep top-3 ──▶ stuff only those 3 into the prompt

You search broadly so you don’t miss the answer, then send narrowly so you don’t pay for the chaff. Reranking lets you have a high retrieval recall with a low generation-token bill — you pay a small reranking cost to avoid a large generation cost.

Embedding cost vs generation cost

Embeddings are cheap — a fraction of generation pricing per token, and computed once per chunk at index time, then reused across every query. Generation is the expensive, per-call cost. So spend on retrieval to save on generation: a better embedding pass and a reranker cost little and shrink the expensive thing — the tokens you generate over and the context you prefill.

Cache the embeddings

Embed each chunk once at indexing time and store the vectors. Re-embedding your whole corpus on every query (or every deploy) is a silent, avoidable cost. Only the incoming query needs embedding at runtime.

The retrieval latency budget

RAG adds a step before the model: the vector search (and rerank) runs first. That’s latency the user feels on top of generation. The trade is usually a clear win — retrieval is milliseconds and saves prefill time by shrinking the prompt — but a slow reranker or an over-large k can eat the savings. Budget it: retrieval + rerank should be a small fraction of total response time, and the token reduction should more than pay for the added hop.

The quality–cost trade

Setting	Tokens/call	Quality risk
High k, big chunks	high	noise; lost-in-the-middle
Low k, right-sized chunks	low	may miss the answer if retrieval is weak
Broad retrieve + rerank to low k	low	best of both, at a small rerank cost

The failure mode of cutting k too aggressively is missing the relevant chunk — a wrong answer to save a few tokens. The defense is exactly reranking: retrieve wide for recall, send narrow for cost. Measure answer quality as you tighten k; stop when quality dips, not before.

Under the hood — how retrieval stays in the millisecond budget

The page claims vector search is “milliseconds” even over a large corpus — that’s only true because production retrieval never does a brute-force scan. Comparing the query vector against every chunk is linear in corpus size and would be seconds-to-minutes at millions of chunks. Instead, vector stores build an approximate nearest-neighbor (ANN) index — most commonly an HNSW graph (hierarchical navigable small-world) or an inverted-file (IVF) partition — that turns the search into a short walk through a navigation structure, roughly logarithmic in corpus size instead of linear. The trade is in the name: approximate. You accept a small chance of missing the true nearest neighbor in exchange for a thousands-fold speedup, and you pay for it in memory — the index sits resident in RAM, often larger than the raw vectors themselves. So RAG’s retrieval hop is cheap in latency precisely because it spent memory and a sliver of recall to get there — the same cost-shifting bargain the rest of this page makes with tokens.

The architect’s lens

RAG’s discipline in one line — context is a budget, not a backpack:

• Why does it exist? Because the naive instinct is to stuff the whole document, but if a question needs two paragraphs, sending a 30,000-token manual every call pays for 99% dead weight — RAG retrieves only the few chunks that answer it.
• What problem does it solve? Input-token cost and accuracy together: top-3 vs top-20 at 400-token chunks is ~5.5× fewer input tokens and less noise to get lost in — often cheaper and more accurate at once.
• What are the trade-offs? It adds a retrieval hop (embedding + ANN search + optional rerank) before the model and couples chunk size to k; spend on cheap retrieval to save on expensive generation, but a slow reranker or over-large k can eat the win.
• When should I avoid it? When k is pushed so low that retrieval misses the answer-bearing chunk — the defense is retrieve-wide-then-rerank-narrow, not blind cuts; measure quality as you tighten.
• What breaks if I remove it? You fall back to stuffing everything (max prefill cost + lost-in-the-middle) or to fine-tuning facts in (stale, expensive to update) — both worse for fresh, large, or citable knowledge.

Check your understanding

1. In RAG, why does a smaller top-k often improve both cost and accuracy at once?
2. Compute the input tokens for top-3 vs top-20 with 400-token chunks and a 300-token question+prompt. What’s the ratio?
3. What does a reranker let you do that plain embedding search alone does not?
4. Why is it sound economics to spend more on retrieval (embeddings, reranking) to save on generation?
5. What’s the risk of pushing k too low, and what technique defends against it?

Show answers

1. Fewer chunks means fewer input tokens (lower cost and faster prefill) and less irrelevant text for the model to get distracted by (the lost-in-the-middle effect), so accuracy can rise as cost falls — provided retrieval still surfaces the answer in the top few.
2. top-3 = 300 + 3×400 = 1,500 tokens; top-20 = 300 + 20×400 = 8,300 tokens; ratio ≈ 5.5×.
3. Retrieve a broad candidate set cheaply (high recall) and then re-score it with a sharper model to keep only the genuine best few, so you can send a small, high-quality context instead of a large noisy one.
4. Embeddings are cheap and computed once per chunk and reused; generation is the expensive per-call cost. A small retrieval/rerank spend shrinks the expensive thing (generated and prefilled tokens), so the net cost drops.
5. You may miss the chunk that actually contains the answer, producing a wrong answer to save a few tokens. Reranking — retrieve wide for recall, then rerank down to a small k — defends against it.