Revision · Application-Layer Efficiency

Part 6 pivoted from owning the model to calling one: a meter runs on every token in and out, and you can’t touch the kernels. Its throughline is that every application-layer lever is a fraction, and fractions multiply into an order-of-magnitude saving — with no GPUs and no retraining.

What this part covered

• You pay per token, by the call — cost is direct and per-request, output tokens run ~5× input, and every prompt token is re-billed on every call forever; your “memory” budget is the context window, not HBM.
• Prompt & context economy — the leanest lever: a fat 2,000-token prompt vs a lean 850-token one is ~55% cheaper at 1M calls/month, and a bigger context window is a ceiling not a target — the “lost in the middle” U-curve means padding can bury the passage that mattered.
• Cap the expensive side — because output carries the ~5× premium, max_tokens and stop sequences are the highest-leverage trims; multi-turn history is re-sent every turn, so summarize or drop it before cost grows quadratically.
• Caching stops re-paying for repetition — three caches for three flavors: prefix (still calls the model but reads a stable prefix at ~0.1×, ~9× cheaper, break-even at two calls), semantic (skips the model on meaning-match, risks false hits/staleness), and response (exact byte-match, cheapest when it hits).
• RAG retrieves a little well — context is a budget, not a backpack: top-3 vs top-20 is ~5.5× fewer input tokens and less noise, so smaller k is often cheaper and more accurate; retrieve wide then rerank narrow, and spend on cheap retrieval to save on expensive generation.
• Model routing & cascades — the biggest single lever on dollars-per-call: model prices span ~an order of magnitude, so routing an 80/15/5 easy/medium/hard split cuts blended cost from 10.0 (all-Opus) to ~1.75 units (~83% cheaper) — guarded by evals and verification against mis-routing.
• Structured output & constrained decoding — masking invalid tokens guarantees valid output in one call, killing the retry tax (a 10% failure rate ≈ 10% extra cost) and forbidding preamble on the expensive output side; if your code parses it, constrain it.
• The levers compose and multiply — a lean prompt that is also cached, RAG-trimmed, routed to Haiku, and returning constrained JSON stacks fractions into a large recurring win.

The takeaway

This is where money actually leaks, because application-layer waste is recurring and compounds with traffic — a fat prompt costs more on call one million, not call one. None of these levers need GPUs, retraining, or an infra migration; they are prompt edits, a cache header, a retriever, and a routing if-statement. The capstone stacks prompt caching and model routing on a real workload to watch them compound.