Where AI Systems Are Going

The Efficiency Mindset opened the book with a claim: efficiency is not a bag of tricks but a way of seeing, and every technique slots into a map you already hold. This closing page earns that claim by reading the direction off the map. The individual systems will change — new models, new frameworks, new silicon every few months — but a handful of throughlines have held across the entire book and show every sign of continuing. If you remember nothing else, remember these, and remember the question that generated all of them: what does this cost in latency, memory, and dollars, and how do we make it cheaper?

The throughlines

  1. Memory bandwidth is still the wall
  2. Sparsity beats density            (MoE, activate a slice)
  3. Precision keeps falling           (FP16 → FP8 → FP4)
  4. Disaggregation everywhere         (separate the jobs)
  5. Test-time compute is a new axis   (think longer, not just bigger)
  ───────────────────────────────────────────────────────────
  all of it driving ONE number down:  cost per *useful* token

1. Memory bandwidth is still the wall

The single most load-bearing fact in this book is that LLM decode is memory-bound: each token reads the whole working set of weights (and KV) from memory, and per-token latency is roughly bytes_read ÷ bandwidth, not limited by arithmetic. Compute has grown far faster than memory bandwidth for years, and nothing on the horizon reverses that — so the wall isn’t moving. Almost every other trend on this page is a way of reading fewer bytes per useful token. When you meet a new technique, the first question is still: does it reduce the bytes moved per token, and if not, why does it help?

2. Sparsity beats density

Mixture of Experts decoupled the two numbers that used to move together — total parameters (knowledge, paid in memory) and active parameters (compute, paid per token). The DeepSeek moment showed this at frontier scale: ~671B total, ~37B active. The direction is clear — frontier models are increasingly sparse, buying quality with resident memory while keeping per-token compute small. Sparsity isn’t the only form; pruning and attention sparsity push the same way. The bet across the field: grow what sits in memory, shrink what runs per token.

3. Precision keeps falling

Each step down the precision ladder — FP32 → BF16 → FP8 → INT4 — roughly halves bytes per parameter, halving both memory footprint and the memory-bound latency floor, until accuracy gives out. The frontier of this trend as of ~2025:

  FP16/BF16  ── mature, the inference default
  FP8        ── hardware-native since Hopper (H100); now mainstream,
                used even for training (see the DeepSeek moment)
  FP4        ── emerging: NVIDIA's Blackwell generation added
                hardware FP4 support; early, promising, not yet routine

FP4 is the one to watch and to hedge on: hardware support is real and recent, but whether 4-bit becomes a routine serving (let alone training) format depends on accuracy techniques still maturing. The historical pattern from the numeric-precision page is the guide — each rung down the ladder earns dedicated silicon once its accuracy turns out to be tolerable, and what looks aggressive this year becomes the default a couple of years later.

4. Disaggregation everywhere

The Advanced part’s unifying idea — stop forcing one resource to do two incompatible jobs — generalizes far beyond prefill/decode disaggregation. The 2025 serving frontier showed the KV cache itself disaggregating into a cluster-wide, tiered resource. The throughline: components that used to be co-located (the two phases, the KV cache, base vs. adapter, even storage tiers) are being separated and tuned independently, because a separated resource can be right-sized to its own bottleneck instead of compromised for someone else’s.

5. Test-time compute is a new scaling axis

For years “make it better” meant “make it bigger.” Reasoning and test-time compute added a second axis: spend more compute at inference — let the model think longer, sample more, verify itself — to get better answers from the same weights. This reshapes the cost conversation. Quality is no longer a fixed property of a model you paid for once at training; it’s a dial you pay for per request in decode tokens. That makes the agentic and reasoning workloads token-hungry in a new way — and makes every decode-cheapening lever in this book (speculative decoding, KV efficiency, small drafters, routing) matter more, not less, because there are now far more tokens to make cheap.

The one number everything serves

Underneath all five trends is a single quantity the whole book has been circling: cost per useful token.

  cost per useful token  =   $ per token        ← precision, sparsity, hardware
                           ×  tokens per answer  ← reasoning, prompts, routing
                           ÷  usefulness          ← quality, eval, did it help?

Every lever in the book pushes on one of these factors. Quantization, MoE, and better silicon cut dollars per token. Caching, routing, prompt economy, and small models cut tokens spent per answer. Evals and quality work raise usefulness so the tokens you do spend aren’t wasted. The frontier is not “bigger models” or “more GPUs” in the abstract — it is the relentless, compounding push to drive cost per useful token down, on every axis at once. That number is where the book’s recurring question finally points.

How to read the next breakthrough

When the next headline technique lands, don’t ask “is it new?” — ask which factor of cost per useful token it moves, what it spends to move it, and at what exchange rate. That’s the efficiency mindset from page one, and it will outlast every specific system named in this book. The techniques are perishable; the lens is not.

It actually happened — FP4 reaches silicon (2024)

The precision ladder kept descending on schedule. At GTC in March 2024, NVIDIA announced the Blackwell architecture with a second-generation Transformer Engine adding hardware support for 4-bit floating point (FP4) — the next rung below the FP8 that Hopper introduced in 2022. The pattern is the one the numeric-precision page predicted: a format that looked too aggressive becomes hardware-native once the accuracy turns out tolerable for enough workloads, doubling math throughput and halving bytes again. Treat FP4 as emerging, not settled — routine 4-bit serving and training still depend on quantization and numerics work that was maturing as of ~2025 — but its arrival in silicon is the clearest sign that “read fewer bytes per token” remains the field’s central, durable move.

Where this leaves you

You now hold the map the first page promised: a hardware floor set by memory bandwidth, model-level levers (precision, sparsity, distillation, small models), system-level levers (batching, KV cache, disaggregation, speculative decoding), and app-level levers (prompts, caching, routing, RAG) — all of them, in the end, instruments for driving cost per useful token down. The specific tools will keep changing. The discipline won’t: measure the dominant cost, pick the altitude that attacks it, trade deliberately, get to the frontier, and ship. That loop is what makes an AI systems engineer, and it’s the same loop whether you’re tuning a phone-sized model or a trillion-parameter MoE. The question never changes — what does this cost, and how do we make it cheaper? — and now you know how to answer it.

Check your understanding

1. Why is “memory bandwidth is still the wall” the throughline that most of the others are downstream of?
2. What two numbers did Mixture of Experts decouple, and which direction is the field pushing each?
3. Where on the precision ladder does FP4 sit as of ~2025, and why should you describe it as “emerging” rather than standard?
4. Test-time compute added a “new scaling axis.” How does it change where quality gets paid for, and why does that make decode-cheapening levers matter more?
5. Write out the three factors of “cost per useful token” and name one lever from the book that moves each.

Show answers

1. Because LLM decode is memory-bound: per-token latency is roughly bytes_read ÷ bandwidth, and bandwidth has grown far slower than compute, so it stays the binding constraint. Most other trends (lower precision, sparsity, KV efficiency, small models) are fundamentally ways to read fewer bytes per useful token — they exist because of the wall.
2. Total parameters (knowledge, paid for in resident memory) and active parameters per token (compute, paid per token). The field is pushing total up (more knowledge in memory) while keeping active small (cheap per-token compute) — grow what sits in memory, shrink what runs per token.
3. FP4 is the rung below FP8, with hardware support added in NVIDIA’s Blackwell generation (announced 2024). It’s “emerging” because routine 4-bit serving and especially training still depend on quantization/numerics techniques that were maturing as of ~2025 — the silicon exists, but 4-bit is not yet a settled default the way FP8/BF16 are.
4. It moves quality from a fixed property bought once at training time to a per-request dial: spend more decode tokens (think longer, sample, self-verify) for better answers from the same weights. That multiplies the number of tokens per answer, so every lever that makes decode tokens cheaper (speculative decoding, KV efficiency, routing, small drafters) applies to far more tokens and matters more.
5. Dollars per token (moved by quantization / MoE / better silicon), tokens spent per answer (moved by caching / routing / prompt economy / small models), and usefulness per token (raised by evals and quality work so spent tokens aren’t wasted).