Revision · The Hardware Reality

Part 1 built the cost model from the silicon up: every “make it cheaper” question reduces to how many bytes move across which wire, how fast, and how many FLOPs you actually get from each byte. The throughline — compute has outgrown memory bandwidth — is the villain that the rest of the book keeps fighting.

What this part covered

• Compute vs memory bandwidth — a GPU has two headline numbers (TFLOP/s and TB/s), and across V100→A100→H100 compute grew ~8× while bandwidth grew only ~3.7×, so more workloads land on the memory-bound side.
• GPU shape: throughput over latency — thousands of simple cores plus matmul-only tensor cores hide memory stalls with many in-flight threads, which is why warp divergence and branchy code waste the chip.
• The memory hierarchy — registers → SRAM → L2 → HBM → host RAM, each step ~10× slower; weights live in the ~80 GB of HBM, the slowest thing the cores touch every step, so its bandwidth caps token speed.
• Arithmetic intensity and the roofline — FLOPs-per-byte decides your wall; below the ~300 FLOPs/byte ridge point (H100) you are memory-bound and faster compute buys nothing.
• Why inference is memory-bound — decode generates one token at a time, turning matmuls into matrix-vector products (intensity ~1), so time/token ≈ bytes_read / bandwidth and a 70B FP16 model floors at ~42 ms/token on an H100.
• Batching climbs the roofline — reusing each weight read across many requests amortizes the fixed weight-read, slashing dollar-per-token until the critical batch size flips decode compute-bound.
• Numeric precision — halving bytes-per-param (FP32→BF16→INT8→INT4) halves both memory footprint and the latency floor in one move; accuracy is negligible down to 8 bits and becomes engineering below it.
• Interconnect and multi-GPU — when weights exceed one card’s HBM you shard across GPUs, and the wires (NVLink ~hundreds of GB/s, PCIe ~32 GB/s, inter-node network) become the new bottleneck, so N GPUs never buy N× speed.

The takeaway

Get the physics right — bytes moved, FLOPs done, and which wire they cross — and the rest of the book reads as consequences. Inference is memory-bound, so the highest-leverage levers all move fewer weight-bytes per token: shrink the numbers (precision) or reuse them more (batching), and stay on the fewest, fastest-connected GPUs you can. Part 4 turns the precision knob into a full compression toolkit; Part 5 wires these facts into an always-on server.