Part 1 · The Hardware Reality

Every clever idea in machine learning eventually has to run on a physical machine: transistors that flip, wires that carry electrons, and memory chips that hold numbers. If you want to reason about whether a system is fast, cheap, or even possible, you have to start there. This part builds the mental model from the silicon up.

The throughline for the entire book is a single question: what does this cost — in latency, memory, and dollars — and how do we make it cheaper? Almost every optimization you will ever read about (batching, quantization, KV caching, speculative decoding, parallelism) is a trick to move fewer bytes or do fewer FLOPs. To recognize a good trick, you first need to know which resource is actually scarce.

Why hardware first

It is tempting to treat the GPU as a magic box that “does the math fast” and move on to architectures and prompts. That works right up until you ask a real engineering question:

• Why does my 70B model need two GPUs but my 7B model fits on one?
• Why does generating 500 tokens take 10x longer than the prompt that fed it?
• Why does INT8 quantization make inference faster, not just smaller?
• Why does adding more users barely change my per-token latency?

None of these have answers at the level of “the model.” They have answers at the level of how many bytes move across which wire, and how fast. Efficiency in AI is, at bottom, the physics of moving bytes and doing floating-point operations. Get that physics right and the rest of the book becomes a series of consequences.

Note

You do not need a GPU to follow this part. Every claim is backed by a worked numeric example: we write down the formula, plug in real (approximate) hardware numbers, and state the result. By the end you will be able to estimate latency and memory on the back of a napkin.

The two numbers that govern everything

A modern accelerator has two headline capabilities, and the tension between them explains most of what follows:

   COMPUTE                       MEMORY BANDWIDTH
   how many FLOPs/sec            how many bytes/sec you can
   the chip can do               read from its own memory (HBM)

   measured in TFLOP/s           measured in TB/s
   (often ~hundreds to           (often several TB/s on a
    ~1000+ for low precision)     datacenter GPU)

Compute has grown far faster than memory bandwidth over the last decade. That imbalance is the central villain of this book: it means that for many real workloads — LLM inference above all — the chip spends most of its time waiting for numbers to arrive, not computing on them. The expensive silicon sits idle. Understanding when you are bandwidth-bound versus compute-bound is the difference between an optimization that helps and one that does nothing.

Roadmap of this part

1. GPUs From First Principles — why a GPU is thousands of small cores instead of a few big ones, and the memory hierarchy (registers → SRAM → L2 → HBM → host RAM) that feeds them.
2. Memory Bandwidth vs Compute: The Roofline — arithmetic intensity (FLOPs per byte) and the roofline model that tells you, for any kernel, whether compute or bandwidth is the wall you hit.
3. Why LLM Inference Is Memory-Bound — the key page. Generating one token at a time turns matmuls into matrix-vector products, so the bottleneck is reading weights from memory. This is why time-per-token has a hard floor.
4. Numeric Precision: FP32 to INT4 — how shrinking each number from 4 bytes to 1 (or half a byte) halves the memory and the bandwidth pressure, and what accuracy it costs.
5. Interconnect & Why Big Models Need Many GPUs — when a model is too big for one chip’s memory, you split it; now the wires between GPUs become the bottleneck.

By the end of this part, “make it cheaper” will stop being a slogan and become a calculation.

By the numbers — compute outran bandwidth, generation by generation

The claim that “compute has grown faster than memory bandwidth” isn’t hand-waving — it is visible across three datacenter-GPU generations (approximate peak figures):

GPU (year)     FP16 tensor compute    HBM bandwidth
V100 (2017)    ~125 TFLOP/s           ~0.9  TB/s
A100 (2020)    ~312 TFLOP/s           ~2.0  TB/s
H100 (2022)    ~990 TFLOP/s           ~3.35 TB/s

compute   V100→H100:  ~125 → ~990    ≈ 8×
bandwidth V100→H100:  ~0.9 → ~3.35   ≈ 3.7×

Across those two generations peak compute grew roughly twice as fast as memory bandwidth. So with each generation the ridge point — the FLOPs-per-byte you must hit to keep the cores busy — climbs, and more workloads fall on the memory-bound side of the roofline. That is precisely why the rest of this part obsesses over bytes moved, not FLOPs done.

Check your understanding

1. What are the two headline capabilities of a modern accelerator, and what are their units?
2. Over the last decade, which has grown faster: compute or memory bandwidth — and why does that matter?
3. Why does this book insist on starting with hardware rather than model architectures?
4. Give one engineering question whose real answer lives at the hardware level, not the model level.
5. What single cost question is the throughline of the whole book?

Show answers

1. Compute (FLOPs per second, measured in TFLOP/s) and memory bandwidth (bytes per second read from HBM, measured in TB/s).
2. Compute has grown much faster than memory bandwidth. It matters because many workloads now spend most of their time waiting on memory while the expensive compute sits idle — they are bandwidth-bound.
3. Because efficiency is ultimately the physics of moving bytes and doing FLOPs; questions about speed, memory, and cost only have answers at the level of which bytes move across which wire and how fast.
4. Any of: why a 70B model needs two GPUs, why generation is far slower than prompt processing, why INT8 speeds up inference, or why adding users barely changes per-token latency.
5. “What does this cost — in latency, memory, and dollars — and how do we make it cheaper?”