Part 11 · Frontier

Part 9 took you to the senior serving track — disaggregation, long-context tricks, multi-LoRA, beyond-GPU silicon. This part goes one step further out, to the edge of what the field was actively figuring out around 2024–2025. The throughline does not change — what does this cost, in latency, memory, and dollars, and how do we make it cheaper? — but the frontier moves the question onto new ground: agents that loop, models that “think” before answering, images and video, vector databases, and shrinking models down to where they almost disappear.

Read this as a snapshot, not a settled syllabus

Everything in Part 11 is recent and moving fast. Dates are attached on purpose. Treat specific model names, benchmark claims, and “fastest/cheapest” superlatives as true around 2025 and likely to drift. The durable content is the shape of each cost driver and the lever that bends it — those outlive any one release.

What “frontier” means here

Parts 0 through 10 mostly optimized a single forward pass or a single API call: make one request cheaper. The frontier is where the unit of cost stops being one call:

• An agent turns one task into a loop of many calls, each re-sending a growing context — cost multiplies before you notice.
• A reasoning model spends extra inference compute on a long internal chain-of-thought to buy accuracy — you now pay for tokens you may never see.
• A multimodal or diffusion system pays per image-token and per denoising step — the cost driver moved from sequence length to pixel count and iteration count.
• A retrieval system pays to embed, index, and search millions of vectors — a memory-and-recall problem the LLM never sees.
• A small language model asks the opposite question: how little model can the job actually tolerate?

Each is a new place for the meter to run, and each has its own emerging lever for slowing it down.

The shape of the frontier

   cost driver moved here          the lever that bends it
   ─────────────────────────       ───────────────────────────────
   agent loop (many calls)    →     cache turns, compact context, sub-agents
   thinking tokens (accuracy) →     scale test-time compute only when it pays;
                                     distill the reasoning back into a cheap model
   image tokens / denoise steps →   latent space, few-step distillation
   vector search at scale     →     ANN indexes, quantization, truncatable dims
   "too much model"           →     small models, MoE, on-device

The pattern is the same one the whole book has used: find the resource the new workload abuses, then refuse to pay full price for it. Only the workloads are new.

Roadmap

1. The cost of agentic systems — why ReAct-style multi-step tool-use loops multiply token cost and latency: every step re-sends a growing context. The “agent tax,” and how prompt caching across turns, context compaction, cheaper sub-agents, and parallel tool calls claw it back. Builds directly on prompt economy.
2. Reasoning models & test-time compute — the 2024–2025 shift to trading inference compute for accuracy: long chain-of-thought, OpenAI’s o1 (Sept 2024) and the reasoning-model wave, the “thinking tokens” you pay for, and when test-time scaling beats just distilling it back into a cheaper model.
3. Multimodal & diffusion efficiency — how images become tokens and what a vision encoder costs, and why diffusion’s iterative denoising step count is its real cost driver. Latent diffusion and few-step distillation (consistency and “turbo” models).
4. Embeddings & vector-DB efficiency — the retrieval side of RAG as its own cost surface: embedding inference, ANN indexes (HNSW/IVF) for sublinear search, vector quantization (PQ, int8/binary), Matryoshka embeddings for truncatable dimensions, and the recall-vs-cost knob.
5. Small language models — the counter-trend to ever-bigger models: when a small, specialized, or on-device model is the right answer, and “good enough” beats “frontier.”
6. The DeepSeek efficiency moment — a dated case study (late 2024–early 2025) in how architectural and training-efficiency choices reset expectations about what a capable model has to cost.
7. The 2025 serving frontier — where production serving had gotten to by 2025, pulling together the levers from the serving stack and Part 9.
8. Where AI systems are going — a forward-looking synthesis: the bets, the open problems, and which of today’s “frontier” tricks are likely to become tomorrow’s defaults.

The mental model

Earlier parts cut the cost of one call. The frontier is mostly about cost drivers that multiply — loops, iterations, internal tokens, billions of vectors. When something multiplies, even a small per-unit win compounds enormously, and a small per-unit leak compounds just as fast. That is why the frontier rewards measuring the whole workload, not the single call.

The pages that follow each take one frontier workload, find where its meter runs fastest, and name the technique — usually young, often dated to 2024–2025 — that slows it down without quietly degrading the result. Read them as the current front line of the same question the book opened with: what does this cost, and how do we make it cheaper?