The Efficiency Mindset

Where the Cost Goes gave us the discipline — find the dominant cost and aim there. This page turns that into a mindset you carry through every remaining chapter. Efficiency is not a bag of tricks; it is a way of seeing. Once you internalize the trade-off space and the levers that move it, each new technique slots into a map you already hold.

You cannot maximize everything

Four things are in constant tension, and you cannot win all four at once:

Want	Pulls against
Low latency	Throughput (big batches add wait) and cost (more/faster GPUs)
High throughput	Latency (batching makes individuals wait longer)
Low cost	Quality (smaller/quantized models) and latency
High quality	Cost and latency (bigger models, more tokens, more passes)

Every real decision is a point in this 4D space. Continuous batching buys throughput by spending a little latency. INT4 quantization buys cost and speed by spending a little quality. There is no setting that is best on all axes — there is only the setting that’s best for your constraints. The job is to choose the trade deliberately instead of accidentally.

The throughline as a lens

This is why the book’s recurring question — what does this cost in latency, memory, and dollars, and how do we make it cheaper? — is a lens, not a slogan. Point it at any technique and it forces the honest accounting:

• What does it cost? Name the axis it spends (latency? quality? memory?).
• What does it buy? Name the axis it improves.
• At what exchange rate? A 4× memory cut for a 1% quality drop is a steal; a 2× memory cut for a 30% quality drop is a trap.

Run every optimization through that lens and you’ll never again adopt one because it’s fashionable. You adopt it because the exchange rate is good for your workload.

Measure before you optimize

The fastest way to waste a week is to optimize the wrong stage. Recall the worked examples: for long outputs, decode is ~96% of latency; for long prompts with short answers, prefill dominates. You cannot guess which world you’re in — you must measure. Profile a real request, attribute the latency and the GPU-seconds to queue/prefill/decode/network, and only then choose a lever. A 10× speedup on 4% of the bill is at best a 3.6% win; a 2× speedup on 90% of the bill is a 45% win. Amdahl’s Law is the whole game.

The cardinal sin

Optimizing by vibes — reaching for the technique you read about yesterday instead of the one your profile demands. Measure first. Every chapter ahead assumes you’ll point its tools at a measured dominant cost, not a guessed one.

The levers map

Every technique in this book lives at one of three altitudes. This is the scaffold the whole book hangs on — keep it in your head:

 ┌──────────────────────────────────────────────────────────────┐
 │ APP-LEVEL   prompts · caching · RAG · model routing · trim    │  Part 6, 10
 │   "spend fewer tokens / call a cheaper model"                 │
 ├──────────────────────────────────────────────────────────────┤
 │ SYSTEM-LEVEL  batching · KV cache · scheduling · parallelism  │  Parts 3, 5
 │   "serve more tokens per GPU-second"                          │
 ├──────────────────────────────────────────────────────────────┤
 │ MODEL-LEVEL  quantization · distillation · pruning · MoE      │  Parts 2, 4
 │   "make each token cheaper to compute"                        │
 └──────────────────────────────────────────────────────────────┘
      all standing on  ──►  HARDWARE: FLOPs, bandwidth, $/hour     Part 1

Altitude	Core move	You change	Example chapters
Model-level	Make each token cheaper	The model itself	Quantization, distillation, MoE
System-level	More tokens per GPU-second	How you run it	Batching, KV cache, parallelism
App-level	Spend fewer/cheaper tokens	What you ask for	Prompt economy, caching, routing

The biggest wins often come from the highest altitude — the cheapest token is the one you never generate (a cache hit, a trimmed context, a smaller model routed to an easy query). But each layer multiplies the others, which is why we cover all three.

Pareto thinking, not perfection

You will never find the optimal configuration, because the axes trade against each other. What you can find is the Pareto frontier — the set of configurations where you can’t improve one axis without sacrificing another. Anything off the frontier is strictly wasteful: there’s a setting that beats it on some axis for free. The efficiency mindset, distilled:

1. Measure to find the dominant cost.
2. Pick the altitude (app → system → model) where a lever attacks that cost.
3. Trade deliberately — know the exchange rate before you pull the lever.
4. Get to the frontier, then stop chasing perfection and ship.

Carry that loop into every chapter. The techniques will change; the loop never does. The most efficient engineers aren’t the ones who know the most tricks — they’re the ones who always know which trick the numbers are asking for. Now let’s go meet the hardware those numbers come from.

It actually happened — efficiency as a strategic weapon (early 2025)

For years the assumption held that frontier-grade quality required frontier-grade budgets. Then, in late 2024 and early 2025, the DeepSeek team released open models that reached competitive quality while reporting a training and serving cost a small fraction of what comparable labs were assumed to spend — the product of aggressive efficiency work (mixture-of-experts routing, low-precision training, careful systems engineering) rather than simply more GPUs. The market reaction was violent: in a single day in late January 2025, the news helped wipe a record amount of value off chipmakers, on the theory that if intelligence could be served this cheaply the world might need less compute than it had been buying. Whatever the long-run truth, the episode made this page’s thesis concrete — efficiency is not a cost-saving footnote; it is product strategy. The team that reaches the same quality at a fraction of the cost doesn’t just save money; it changes the game.

Check your understanding

1. Name the four competing objectives and give one example of two that directly trade against each other.
2. The book’s recurring question is called a “lens.” What three things does it force you to name about any optimization?
3. Why is “measure before you optimize” not optional — what does Amdahl’s Law imply for a 10× speedup on 4% of the bill?
4. List the three altitudes of the levers map and the core move at each. Which often yields the biggest win, and why?
5. What is the Pareto frontier, and why is any configuration off it strictly wasteful?

Show answers

1. Latency, throughput, cost, and quality. Example: low latency trades against throughput (big batches improve throughput but make individuals wait), or low cost trades against quality (smaller/quantized models are cheaper but less capable).
2. What it costs (which axis it spends), what it buys (which axis it improves), and the exchange rate between them (e.g. how much quality you give up per unit of memory or cost saved).
3. Because the size of a speedup is capped by the fraction of the bill it touches. A 10× speedup on 4% of the cost yields at most ~3.6% overall (the 4% shrinks to 0.4%), while a 2× speedup on 90% yields ~45% — so you must measure to know which stage dominates before choosing a lever.
4. App-level (spend fewer/cheaper tokens), system-level (serve more tokens per GPU-second), and model-level (make each token cheaper to compute). The app level often wins biggest because the cheapest token is one you never generate — a cache hit, trimmed context, or a query routed to a smaller model.
5. The Pareto frontier is the set of configurations where you can’t improve one axis without worsening another. A configuration off the frontier is wasteful because some other setting beats it on at least one axis “for free,” with no sacrifice elsewhere.