Part 0 · Foundations — Why AI Efficiency Matters

You can already call a model. You send a prompt, you get tokens back, a meter ticks somewhere, and a bill arrives at the end of the month. This book is about the gap between using a model and understanding it: do you know why a single token costs what it costs, and which levers actually make it cheaper? Most engineers can wire up an API in an afternoon and yet cannot explain why a 2,000- token answer costs ~10× a 200-token one, or why a model that “fits in 16 GB” still falls over at batch size 8. That gap is what we close.

The one question

Every page in this book returns to a single question, and you should too:

What does this cost — in latency, memory, and dollars — and how do we make it cheaper?

Those are the three currencies of AI systems. They trade against each other constantly: you can buy latency with money, buy memory with quality, buy throughput with latency. There is no free lunch, only informed trades. We treat “it works” as the start of the problem, not the end. A demo that costs $40 per thousand requests and answers in 9 seconds works — and is also unshippable. Efficiency is what turns a working prototype into a product.

Principles before tools

Tools change every quarter; the physics does not. vLLM, TensorRT-LLM, and whatever ships next are all just clever ways to dodge the same bottlenecks: finite FLOPs, finite memory bandwidth, finite dollars per GPU-hour. So we build the mental model first — FLOPs per token, bytes per parameter, bandwidth per second — and only then name the tools that exploit it. Learn the principle and every new framework becomes “oh, that’s just batching the memory-bound step.” Learn only the tool and you are stranded the day it is deprecated.

The shape of the book

The spine is inference and serving — how a token is actually produced and what it costs to serve at scale — because that is where the money goes for almost everyone running models in production.

Parts	Theme	What you’ll be able to reason about
1–5	Inference & serving (the spine)	Hardware limits, the transformer mechanically, KV caches, quantization, and the serving stacks that tie them together
6	Application layer	Cutting cost from outside the model: prompts, caching, RAG, routing
7	Training & fine-tuning	The other cost regime — when to fine-tune vs prompt, and how to do it cheaply
8	Operating (LLMOps)	Measuring what matters: TTFT, TPOT, goodput, cost models
9	Advanced & rare	Frontier techniques: disaggregation, long-context, multi-LoRA, custom silicon
10	Capstone (Claude API)	Build a real app, measure its bill, and cut it with levers you control without owning a GPU

We read in dependency order: each page uses only ideas already covered. You do not need a GPU — every concept is made concrete with worked numbers you can check on paper.

Roadmap: the rest of Part 0

This part lays the conceptual foundation the other 70-odd pages stand on.

Page	What it gives you
The Cost Problem	Why a token is billions of FLOPs, and the $/1M-tokens framing
What a Model Actually Is	Parameters as numbers in matrices; params × bytes = memory
Training vs Inference	Two cost regimes, and why this book centers inference
Where the Cost Goes	Anatomy of a request: queue, prefill, decode, network
The Efficiency Mindset	The trade-off space and the levers map for the whole book

How to read this book

After each page, answer the Check your understanding questions in your own words before revealing the answers. If you can teach the cost story back, you have it. Depth over speed.

By the end you will look at any model API and instinctively decompose its bill into latency, memory, and dollars — and know which lever to pull first. Let’s start with why a token costs money at all.

It actually happened — inference cost goes board-level (2022)

When ChatGPT launched in late 2022 and crossed a hundred million users within about two months, the bill that mattered was not training — it was inference. OpenAI’s leadership publicly described the compute cost of running it as “eye-watering,” with each conversation costing on the order of cents that, multiplied across an enormous daily user base, became one of the company’s largest line items. Nothing about the model had changed from the research demo; what changed was the serving scale. That is the moment the questions in this book — what does a token cost, and how do we make it cheaper? — stopped being academic and became a survival concern for everyone shipping AI.

Check your understanding

1. What is the single recurring question this book asks of every technique?
2. Name the three “currencies” of AI systems and give one example of trading one for another.
3. Why does this book emphasize principles over specific tools like vLLM?
4. Which parts form the “spine,” and why is the spine inference and serving rather than training?
5. What does the Part 10 capstone optimize, and what constraint makes it different from Parts 1–5?

Show answers

1. “What does this cost — in latency, memory, and dollars — and how do we make it cheaper?”
2. Latency, memory, and dollars. Examples: you can buy lower latency with more money (more/bigger GPUs), or cut memory by accepting lower quality (e.g. quantization). The point is there’s no free lunch, only informed trades.
3. Because tools change every quarter but the underlying physics (finite FLOPs, memory bandwidth, and dollars per GPU-hour) does not; understanding the principle lets you instantly decode any new tool.
4. Parts 1–5. The spine is inference and serving because that is where the cost actually accrues for most people running models in production — a model is trained once but served billions of times.
5. It optimizes a real app built on the Claude API — cutting its cost and latency using levers you control from outside the model (caching, routing, context trimming, streaming), since you don’t own the GPU.