Part 4 · Model Compression

Part 3 made the system cheaper: batching, KV caching, paged attention, speculative decoding. Those levers leave the model’s weights untouched and squeeze more throughput out of the same parameters. Part 4 turns the other knob — we shrink the model itself. Fewer bits per weight, fewer weights, a smaller student, or a sparsely-activated giant. The recurring question for this whole part: what does this model cost — in latency, memory, and dollars — and how do we make it cheaper by changing the weights, not just the schedule?

Why the model is the cost

A modern LLM is, to first approximation, a big pile of numbers you must move from memory to the compute units on every token. Inference is usually memory-bound: the GPU finishes the math long before the bytes arrive. So the dominant costs trace directly back to the size and shape of the weight matrices:

• Memory — a 70B-parameter model in FP16 is ~140 GB. That doesn’t fit on a single 48 GB or 80 GB GPU, so you shard across cards (more dollars) or you can’t serve it at all.
• Latency — each token reads (most of) those bytes. Halve the bytes and you roughly halve the read time on a memory-bound workload.
• Dollars — fewer/smaller GPUs, lower power, higher batch density. Every byte you don’t store or move is money you don’t spend.

Compression attacks all three at the source.

The four levers

  Dense model in FP16
        │
        ├── Quantization ── same weights, fewer bits  (FP16 → INT8/INT4)
        ├── Distillation ── fewer, smarter weights    (train a small student)
        ├── Pruning      ── remove weights that barely matter
        └── MoE          ── many weights, few active per token

Lever	What changes	Wins on	Pays in
Quantization	bits per weight	memory + latency	a little accuracy
Distillation	model size	inference forever	one upfront training run
Pruning/sparsity	number of weights	memory (compute, if structured)	fine-tuning + modest accuracy
MoE	active fraction	compute/FLOPs	memory (all experts resident)

Note

These levers compose. A production stack might distill a smaller model, then quantize it to INT4, then serve it with the Part 3 system tricks. Compression and system efficiency are multiplicative, not either/or.

Roadmap

1. Quantization fundamentals — represent weights and activations in fewer bits; the scale/zero-point mapping; PTQ vs QAT; the outlier problem.
2. The quantization zoo — GPTQ, AWQ, GGUF/llama.cpp, bitsandbytes, SmoothQuant: which format for which job.
3. Distillation — train a small student to mimic a big teacher; trade one training run for cheaper inference forever.
4. Pruning & sparsity — drop weights that contribute little; why structured sparsity is faster but unstructured is easier.
5. Mixture of Experts — big-model quality at small-model FLOPs, at the price of big-model memory.

The mental model

Quantization and pruning shrink an existing model. Distillation trains a new, smaller one. MoE reshapes the model so most of it sits idle per token. Same goal — fewer dollars per token — reached from four different directions.

By the end of this part you’ll be able to look at a deployment and reason about which lever actually moves your bottleneck. If you’re memory-bound and out of VRAM, quantization is the first reach. If inference volume is enormous and forever, distillation amortizes. If you need frontier quality on a FLOPs budget, MoE — provided you can afford the memory to hold every expert.

Check your understanding

1. Why is LLM inference usually described as “memory-bound,” and why does that make compression effective?
2. Roughly how much memory does a 70B-parameter model need in FP16, and why is that a problem for a single GPU?
3. Which lever trains a brand-new model rather than shrinking an existing one?
4. MoE reduces FLOPs per token but is described as “hungry on memory.” Why?
5. Why are compression and the Part 3 system levers described as multiplicative rather than competing?

Show answers

1. The GPU finishes the arithmetic faster than it can read the weight bytes from memory, so token latency is dominated by bytes moved. Compression reduces bytes, so it directly reduces the binding cost.
2. ~140 GB (70 × 10⁹ params × 2 bytes). That exceeds a single 48 GB or 80 GB GPU, forcing sharding across multiple cards or making it un-servable on one card.
3. Distillation — it trains a small student model from scratch (guided by a teacher) rather than editing existing weights.
4. Only a fraction of experts run per token (low FLOPs), but all experts must be resident in memory to be routable, so total memory stays large.
5. They attack different costs — compression shrinks the weights, system levers schedule the work better — so their savings stack instead of overlapping.