Revision · Frontier

Part 11 maps the edge of what the field was figuring out around 2024–2025, where the unit of cost stops being a single forward pass. The throughline holds — what does this cost, and how do we make it cheaper? — but the frontier is dominated by cost drivers that multiply: loops, iterations, internal tokens, and billions of vectors, where every small per-unit leak compounds.

What this part covered

• The cost of agentic systems — ReAct-style loops re-send a growing context on every step, so token cost grows roughly quadratically (the “agent tax”); prompt caching across turns, context compaction, cheaper sub-agents, and parallel tool calls claw it back.
• Reasoning models & test-time compute — the o1-era shift to spending extra inference compute on long internal chain-of-thought to buy accuracy, meaning you pay for “thinking tokens” you may never see — worth it only when the task pays, and often better distilled back into a cheap model.
• Multimodal & diffusion efficiency — images become tokens (a vision-encoder cost tied to pixel count), while diffusion’s real cost driver is the denoising step count; latent diffusion denoises in a compressed space and few-step distillation (consistency, turbo) slashes iterations.
• Embeddings & vector-DB efficiency — retrieval is its own cost surface with two centers, embedding inference and search; ANN indexes (HNSW/IVF) give sublinear search, vector quantization (PQ, int8/binary) and Matryoshka truncatable dimensions shrink storage, all governed by the recall-vs-cost knob.
• Small language models — capable 1–8B models from better data and distillation, where “good enough” beats a frontier call on cost, latency, and privacy, and on-device is their natural home.
• The DeepSeek efficiency moment — a dated case study (V3 in Dec 2024, R1 in Jan 2025) in how architectural and training-efficiency choices reset expectations about what a capable model has to cost, and moved the market.
• The 2025 serving frontier — disaggregation going mainstream, cross-user prefix sharing, KV-cache offloading and tiering, and speculative decoding becoming default settings rather than tricks.
• Where AI systems are going — the throughlines that outlast any technique: memory bandwidth is still the wall, sparsity beats density, precision keeps falling, disaggregation spreads everywhere, and test-time compute becomes a new scaling axis — all serving cost per useful token.

The takeaway

The frontier is the same book-long question on new ground: find the resource the new workload abuses, then refuse to pay full price for it. Because these drivers multiply, measuring the whole workload — not the single call — is what separates a real win from a leak that compounds. Read the specifics as a dated snapshot; the durable content is the shape of each cost driver and the lever that bends it. From here, the capstone stops explaining levers and starts pulling them on one real app.