Caching: Prefix, Semantic, Response

Prompt economy cut the tokens you send. Caching attacks a different waste: tokens you send again and again. If the prompt economy question is “do I need this token?”, the caching question is “have I already paid for this exact work?” There are three answers, at three layers, each catching a different flavor of repetition — and each is a direct strike at the throughline: less latency, less compute, fewer dollars per call.

Three caches, three kinds of repetition

Request comes in
   │
   ├─ Exact same request as before?      → RESPONSE cache   → return stored answer (no model call)
   ├─ Same *meaning* as a past request?   → SEMANTIC cache   → return stored answer (no model call)
   └─ Same stable *prefix*, new tail?     → PREFIX cache     → model call, but prefill the prefix cheaply

The first two skip the model entirely. The third still calls the model but stops re-billing the part of the prompt that never changes.

1. Prefix / prompt caching

This is the application-layer face of the server-side prefix caching from Part 3. The insight is the same: a transformer’s prefill over a fixed prefix produces the same KV state every time, so that state can be stored and reused. The provider exposes it as a knob.

On Anthropic’s Messages API you mark a stable prefix with cache_control (the prompt-caching breakpoint). The render order is tools → system → messages, so the stable content — a frozen system prompt, your few-shot block, a long reference document — goes first, and the volatile content — the user’s actual question — goes after the breakpoint. Any byte change before the breakpoint invalidates the cache, so a datetime.now() in the system prompt silently kills it.

The economics. A cache read costs roughly 0.1× a normal input token; the first write costs about 1.25× (5-minute TTL). So caching pays off fast:

Prefix = 2,000 tokens, reused over 100 calls.

No cache:   2,000 × 100              = 200,000 input-token-units
Cached:     2,000 × 1.25  (write)
          + 2,000 × 0.1 × 99 (reads) =   2,500 + 19,800 = 22,300 units

That’s a ~9× reduction on the prefix, plus lower latency because cached prefill is near-instant. Break-even is two calls — beyond that it’s pure savings. The catch: cache entries have a TTL (commonly ~5 minutes), so the win is real only when calls reuse the prefix within that window.

What to cache

Cache the big, stable, repeated stuff: system prompts, tool definitions, few-shot blocks, a document everyone asks questions about. Put it before the breakpoint; keep it byte-identical. Put the per-request question after it.

2. Semantic caching

Prefix caching needs an identical prefix. Semantic caching is looser: it returns a stored answer when a new query means the same thing as an old one. “What’s your refund policy?” and “How do I get my money back?” are different bytes but the same question.

How it works: embed the incoming query into a vector, search a vector store of past (query, answer) pairs, and if the nearest neighbor’s cosine similarity exceeds a threshold, return the stored answer — no model call at all.

query ──embed──▶ vector ──nearest-neighbor search──▶ similarity ≥ threshold?
                                                         │
                                          yes ──▶ return cached answer (skip the model)
                                          no  ──▶ call model, store (query, answer)

The savings are total — you skip the entire generation — but so is the risk. Staleness and false hits are the failure modes:

• A threshold too low returns a stored answer for a query that only looks similar — wrong answer, confidently served.
• A cached answer can go stale (the refund policy changed; the cache didn’t).

So semantic caching fits high-repetition, slowly-changing domains (FAQs, support macros) and needs a TTL plus invalidation when the underlying truth changes. Tune the threshold against real traffic: too high and you never hit; too low and you serve garbage.

3. Response / exact caching

The simplest cache: hash the full request (prompt + params) and store the response. Identical request in, stored response out, zero model cost. It’s a plain key-value lookup — fast, cheap, and dumb in the good way.

Exact caching only fires on byte-identical requests, so its hit rate is low for free-form chat but high for deterministic, parameterized calls — a nightly job that summarizes the same documents, an enrichment pipeline that re-processes unchanged rows. When it hits, it’s the cheapest cache of all.

Choosing and invalidating

Cache	Hits when	Skips model?	Main risk
Prefix	stable prefix repeats	no (cheaper prefill)	TTL expiry; silent invalidation by a changed byte
Semantic	meaning repeats	yes	false hits, staleness
Response	exact request repeats	yes	low hit rate; staleness if source changes

Every cache trades freshness for cost, so every cache needs an invalidation story. Prefix caches self-expire on a short TTL. Semantic and response caches return content, so they must be invalidated when the underlying answer changes — by TTL, by versioning the cache key on the data source, or by explicit purge. A stale cache is worse than no cache: it serves a wrong answer at full confidence for free.

Layer them

These aren’t either/or. A production stack runs all three: exact cache out front for free repeats, semantic cache behind it for paraphrases, and prefix caching on the calls that do reach the model. Each layer shaves cost off the calls the layer above let through. See the capstone’s prompt-caching lever for this stacked up end to end.

It actually happened — prompt caching becomes a billing knob (2024)

For years, server-side prefix reuse was an inference-engine optimization invisible to API users — the provider might reuse a KV-cached prefix internally, but you couldn’t ask for it or see it on the bill. In 2024 that flipped: within a few months, several major LLM providers shipped explicit prompt/prefix caching, exposing the server-side prefix cache as a feature you control and pay a discounted rate for. The mechanism was identical to the research idea — store the KV state of a fixed prefix, skip re-prefilling it — but the packaging changed the economics overnight: cached prefix tokens are billed at a large discount, so a once-internal optimization became a line item teams design their prompts around. The lesson is the one this page opens with — the cache only helps if the cached part is genuinely stable and reused inside its TTL, and a single changed byte before the breakpoint quietly puts you back on the full-price path.

The architect’s lens

Three caches, but the same question — have I already paid for this exact work?

• Why does it exist? Because real traffic repeats — the same prefix, the same meaning, or the same exact request — and recomputing it is pure waste; each cache catches a different flavor of that repetition.
• What problem does it solve? Re-billed work: prefix caching drops a 2,000-token reused prefix to ~0.1× per read (a ~9× cut, break-even at two calls), while semantic and response caches skip the model entirely on a meaning- or byte-match.
• What are the trade-offs? Every cache trades freshness for cost — prefix caches self-expire on a ~5-min TTL and die on a single changed byte before the breakpoint; semantic caches risk false hits and staleness; response caches have low hit rates on free-form chat.
• When should I avoid it? Avoid semantic caching for fast-changing truth or high-stakes answers where a confident false hit is worse than a fresh call; avoid relying on prefix caching when calls don’t reuse the prefix inside its TTL.
• What breaks if I remove it? You pay full price to re-prefill stable prefixes and re-generate answers you’ve already produced — and a stale cache is worse than none, serving a wrong answer at full confidence for free.

Check your understanding

1. Which of the three caches still calls the model, and what does it save instead?
2. Why must the stable content go before the cache_control breakpoint and the user’s question after it?
3. A 2,000-token prefix is reused over 100 calls. Roughly how much cheaper is the prefix with prompt caching, given ~1.25× write and ~0.1× read costs?
4. What are the two main failure modes of semantic caching, and what causes each?
5. Why is a stale cache sometimes worse than having no cache at all?

Show answers

1. Prefix/prompt caching still calls the model; it saves the cost and latency of re-prefilling the stable prefix (cache reads cost ~0.1× a normal input token). The semantic and response caches skip the model entirely.
2. Prefix caching is a prefix match — any byte change before the breakpoint invalidates the cache. The stable content must come first so it stays byte-identical and cacheable; the volatile question goes after so it doesn’t poison the cached prefix.
3. About 9×: no cache ≈ 200,000 units (2,000 × 100); cached ≈ 2,500 (one write) + 19,800 (99 reads at 0.1×) ≈ 22,300 units.
4. False hits (threshold too low returns a stored answer for a query that’s only superficially similar) and staleness (the cached answer is correct for an old version of the truth that has since changed).
5. It serves a wrong or outdated answer at full confidence and for free, which can be more damaging than a fresh (if costlier) correct answer — the cost saving masks an incorrect result.