Revision · Part 0 · Foundations — Why AI Efficiency Matters

Part 0 sets the mental model the whole book runs on: an AI system’s bill is not a mystery but a computable quantity, and every technique ahead is a deliberate trade against the same three currencies. The throughline never changes — what does this cost in latency, memory, and dollars, and how do we make it cheaper?

What this part covered

• The one recurring question — every page returns to “what does this cost in latency, memory, and dollars, and how do we make it cheaper?”, treating “it works” as the start of the problem, not the end.
• A token is a forward pass — each generated token runs the whole model once, touching every parameter, so a 500-token answer is 500 sequential passes and cost scales with model size.
• FLOPs per token ≈ 2 × params — the multiply-accumulate rule makes model cost concrete: a 70B model burns ~140 GFLOPs per token, and $/1M-tokens is the lingua franca that follows from it.
• A model is numbers in matrices — parameters are just weights being multiplied, and params × bytes = memory is the master lever behind quantization (FP16 ≈ 2 bytes/param → 7B ≈ 14 GB).
• Weights are not the whole bill — activations and the KV cache also consume VRAM at runtime, and at long context or high batch the KV cache can rival or exceed the weights.
• Training vs inference are two cost regimes — training is forward+backward+optimizer (~6× params FLOPs, ~16–20 bytes/param, throughput-bound, paid once); inference is forward-only (~2× params, ~2 bytes/param, latency-sensitive, paid on every request forever).
• Decode is memory-bandwidth-bound — generating one token at a time reads all the weights but does little math per byte, so the GPU waits on memory, not compute — the asymmetry that drives the whole serving stack.
• Anatomy of a request — queue, prefill, decode, network; prefill is compute-bound and cheap per token, decode dominates long outputs (~96% of latency in the worked example), so the dominant cost is request-shaped.
• Measure before you optimize — Amdahl’s Law means a 10× speedup on 4% of the bill is at best ~3.6%; profile to find the dominant stage before choosing a lever.
• The levers map and Pareto thinking — app-level (spend fewer/cheaper tokens), system-level (more tokens per GPU-second), model-level (cheaper per token), standing on hardware; you can’t maximize latency, throughput, cost, and quality at once, so aim for the frontier and ship.

The takeaway

Efficiency is a way of seeing, not a bag of tricks: name what a technique costs, name what it buys, and check the exchange rate before pulling the lever. With the three currencies, the FLOPs-and-bytes arithmetic, and the app/system/model levers map in hand, you have the scaffold every later part hangs on. Next, Part 2 opens the transformer box to locate exactly where those FLOPs and bytes go.