Revision · Model Compression

Part 4 turned the other knob: instead of scheduling the work better, it shrinks the model itself — fewer bits per weight, fewer weights, a smaller student, or a sparsely-activated giant. Because inference is memory-bound, every byte you don’t store or move is money you don’t spend, and these levers attack memory, latency, and dollars at the source.

What this part covered

• Quantization fundamentals — map continuous weights onto a small integer grid via scale and zero-point; finer granularity (per-channel, per-group) localizes damage, and INT4 turns a 140 GB two-GPU model into a ~35 GB one-GPU model for a single-digit accuracy dip.
• The outlier problem — activations grow a few channels 10–100× larger that stretch the range and crush ordinary values, which is why activations are harder to quantize than weights and why it emerges only past ~6–7B params.
• Weight-only vs weight+activation — weight-only (INT4 weights, FP16 activations) buys memory and latency; quantizing activations too (INT8) additionally unlocks faster integer matmuls, but only after you tame outliers.
• The quantization zoo — GPTQ (second-order rounding), AWQ (scale up salient weights), GGUF/llama.cpp k-quants (laptop/CPU), bitsandbytes/NF4 (the QLoRA engine), SmoothQuant (migrate outliers for INT8 compute); pick by where you run and what your bottleneck is.
• Distillation — train a small student on the teacher’s soft-label distribution (“dark knowledge”), trading one upfront training run for a permanently cheaper model — a distilled 7B keeping ~90% of a 70B’s quality serves each token ~8× cheaper.
• Pruning and sparsity — delete near-zero weights, but unstructured zeros don’t speed a dense matmul; only structured pruning or 2:4 semi-structured sparsity (~2×) converts to wall-clock speed, and either way you pay a fine-tuning recovery run.
• Why sparsity is harder to cash in — a dropped bit is always fewer bytes moved, but a zeroed weight only helps if both storage and kernel skip it, which is why quantization is the reliable first reach and pruning the second.
• Mixture of Experts — decouple parameter count from per-token compute: N experts hold the knowledge, top-k run per token, so Mixtral 8×7B answers like ~47B at ~13B FLOPs — cheap on compute but hungry on memory, since all experts must stay resident.

The takeaway

These four levers reach the same goal — fewer dollars per token — from different directions, and they compose: distill a smaller student, quantize it to INT4, then serve it with the system tricks. The right lever depends on your bottleneck — quantize first when you’re out of VRAM, distill when inference volume is enormous and forever, reach for MoE when you can afford the memory to buy frontier quality on a FLOPs budget. Part 5 wires these compressed models into a production serving system where latency, memory, and dollars are all measured at once.