Part 6 · Application-Layer Efficiency

Parts 3 through 5 assumed you own the model: you set the batch size, you size the KV cache, you quantize the weights. Most teams shipping features don’t. They call an API — Anthropic’s Messages API, OpenAI, or a hosted endpoint — and a meter runs on every token in and every token out. You can’t touch the kernels, the scheduler, or the precision. So this part asks a different question: when the model is a black box behind a per-token bill, what does each call cost in latency, context budget, and dollars — and which levers, all of them outside the model, make it cheaper?

The pivot: you pay per token, by the call

When you own the model, cost is amortized hardware. When you call an API, cost is brutally direct and per-request:

cost_per_call ≈ (input_tokens × input_price) + (output_tokens × output_price)

Two facts dominate everything in this part. First, output tokens cost several times more than input tokens — commonly around 5× on the Claude model lineup. Second, every token in your prompt is paid on every single call: a system prompt, your few-shot examples, the retrieved document you stuffed in — all of it re-billed each time, forever. A 2,000-token preamble across a million calls is two billion input tokens you chose to pay for.

“Memory” in this part isn’t HBM — that’s the provider’s problem. Your memory budget is the context window: a finite token allowance per request that you fill, and every token you spend filling it adds prefill latency and dollars (and, as we’ll see, can hurt quality).

The five levers

You can’t change the model. You can change what you send it, how often you send it, which model you send it to, and what shape you demand back.

   your request ─┐
                 ├─ Prompt economy ── send fewer, denser tokens
                 ├─ Caching        ── stop re-paying for repeated prefixes
                 ├─ RAG            ── retrieve a little, not stuff a lot
                 ├─ Routing        ── send easy work to the cheap model
                 └─ Structured out ── force valid output, kill retries
                                          └──▶ provider (black box)

Lever	What it cuts	Paid in
Prompt & context economy	input tokens on every call	a little prompt-engineering effort
Caching	re-billing of stable prefixes	cache-write premium + staleness risk
RAG efficiency	tokens stuffed per query	retrieval infra + latency budget
Model routing	dollars on easy requests	mis-routing risk → eval cost
Structured output	retries + over-long replies	schema design, slight rigidity

Note

These compose, and they multiply. A lean prompt that is also cached, that retrieves the top 3 chunks instead of 20, routed to Haiku when Haiku suffices, returning a constrained JSON object — each lever is a fraction, and fractions multiply into an order of magnitude.

Why this is where the money actually leaks

Owning-the-model efficiency is a one-time engineering investment. Application-layer waste is recurring and compounds with scale. A fat prompt doesn’t cost more to write — it costs more on call one million. The encouraging flip side: these levers need no GPUs, no retraining, and no infra migration. They are mostly prompt edits, a cache header, a retriever, and a routing if-statement. The cheapest efficiency win you will ever ship lives in this part.

Roadmap

1. Prompt & context economy — you pay per input and output token; context isn’t free; trim, compress, cap.
2. Caching — prefix/prompt caching, semantic caching, and exact response caching; when each applies.
3. RAG efficiency — retrieve a little well instead of stuffing a lot; chunk size, top-k, reranking.
4. Model routing & cascades — not every request needs the biggest model; cheap-first, escalate on need.
5. Structured output & constrained decoding — valid JSON the first time means no retries, and retries cost tokens.

The bridge ahead

Part 10 — the capstone — takes one realistic application and stacks these levers one at a time, measuring the blended cost after each. Two of its levers come straight from this part: prompt caching and model routing. Learn the levers here; watch them compound there.

Under the hood — why an output token costs more than an input token

The ~5× output premium isn’t arbitrary pricing; it falls out of how a transformer runs. All of your input tokens are processed in one parallel prefill pass — the GPU does them as a single big batched matrix multiply, which is compute-bound and uses the hardware efficiently, so the per-token cost is low. Output tokens are generated one at a time: every new token needs its own forward pass that re-reads the full weights and the growing KV cache from memory. That decode phase is memory-bound and barely uses the GPU’s compute, so each output token ties up far more hardware per token than an input token does. Providers price what it costs them to serve — and a sequential, memory-bound decode step genuinely costs more than a slice of a parallel prefill. The asymmetry is the same prefill-vs-decode split you met when you owned the model, now showing up on your invoice.

Check your understanding

1. What fundamentally changes about cost when you call a model API instead of owning the model?
2. In the per-call cost formula, which tends to be more expensive — input or output tokens — and roughly by how much on the Claude lineup?
3. Why is a long system prompt a recurring cost rather than a one-time one?
4. In this part, what plays the role that “memory” played when you owned the model?
5. Name the five application-layer levers and give a one-line reason each one saves money.

Show answers

1. Cost becomes direct and per-request — you pay a metered amount for every input and output token on every call — instead of being amortized fixed hardware you control.
2. Output tokens are more expensive, commonly around 5× the input-token price on the Claude models. (Always check current pricing.)
3. Because every token in the prompt is re-sent and re-billed on every call; a fixed preamble multiplied by N calls is a cost that grows with traffic, not a one-time charge.
4. The context window — a finite per-request token budget you fill, where every token spent adds prefill latency and dollars (and can hurt quality).
5. Prompt economy (send fewer/denser tokens), caching (stop re-paying for stable prefixes), RAG (retrieve a little instead of stuffing a lot), routing (send easy work to the cheap model), structured output (force valid output so you don’t pay for retries).