Quantization Fundamentals

The overview framed inference as memory-bound: the bytes arrive slower than the math completes. Quantization is the most direct attack on that bottleneck — store each number in fewer bits, move fewer bytes, fit a bigger model. This page is the why and the mechanics; the next page is the zoo of concrete formats.

The idea

A weight is just a real number. FP16 spends 16 bits to store it; INT8 spends 8; INT4 spends 4. If we can represent the weight well enough in 4 bits, we cut memory and bytes-moved to a quarter — and on a memory-bound workload, throughput roughly scales with the inverse of bytes moved.

The cost is precision. The trick is mapping a continuous range of real values onto a small grid of integers with minimal damage.

The mapping: scale and zero-point

Pick the range of values to represent, [min, max]. Map it to the integer range (e.g. [-128, 127] for signed INT8) with an affine transform:

quant   q = round(x / scale) + zero_point
dequant x ≈ scale * (q - zero_point)

scale      = (max - min) / (qmax - qmin)
zero_point = the integer that maps to real 0

Worked mapping. Suppose a weight tensor ranges over [-0.6, 0.8] and we target signed INT8 (qmin = -128, qmax = 127):

scale      = (0.8 - (-0.6)) / (127 - (-128)) = 1.4 / 255 ≈ 0.00549
zero_point = -128 - round(-0.6 / 0.00549) = -128 + 109 = -19
a weight 0.30 → round(0.30 / 0.00549) + (-19) = 55 - 19 = 36
dequant      → 0.00549 * (36 - (-19)) = 0.00549 * 55 ≈ 0.302   (error ≈ 0.002)

Each weight now costs 1 byte instead of 2, and reconstructs to within one half-step of scale.

Symmetric vs affine

If the range is symmetric around zero you can drop the zero-point (set it to 0) — symmetric quantization, slightly cheaper math. Asymmetric/affine ranges keep the zero-point so that real 0 maps exactly, which matters for ReLU-style activations that pile up at zero.

Granularity: per-tensor vs per-channel/group

One scale for an entire weight matrix is cheap but crude — a single outlier column forces a huge scale, crushing precision for everyone. Finer granularity helps:

Granularity	One scale per…	Cost	Accuracy
Per-tensor	whole matrix	cheapest	worst
Per-channel	each output row/column	small overhead	better
Per-group	each block of e.g. 64/128 weights	a bit more metadata	best for INT4

Group-wise (e.g. 128-weight groups, each with its own scale) is the standard for 4-bit weights — it localizes the damage of any single large weight.

The outlier problem

Activations in transformers develop a few outlier features — channels with values 10–100× larger than the rest. Quantize naively and these outliers stretch the range so far that ordinary values lose almost all precision. This is the central difficulty, and it’s why activations are harder to quantize than weights. Mitigations (next page): keep outlier channels in higher precision, or mathematically migrate the outlier scale from activations into weights.

Weight-only vs weight+activation

• Weight-only (e.g. INT4 weights, FP16 activations): you save memory and bytes moved — the big win for a memory-bound, single-stream LLM. The matmul still runs in FP16 after de-quantizing weights on the fly. This is the common choice for local/single-user inference.
• Weight + activation (e.g. INT8 both): you also get faster integer matmuls on hardware with INT8 tensor cores — valuable for high-throughput, compute-leaning serving. But you must tame activation outliers.

Which saves what

Weight-only quantization buys you memory and latency. Activation quantization additionally buys compute throughput — at the cost of solving the outlier problem.

How you get there: PTQ vs QAT

• Post-Training Quantization (PTQ) — take a trained FP16 model and quantize it directly. To set good ranges you run a small calibration set (a few hundred sample sequences) and observe activation statistics. Cheap, fast, no labels, no gradient steps. Default for LLMs.
• Quantization-Aware Training (QAT) — simulate quantization during training/fine-tuning so the model learns weights robust to the rounding. Higher accuracy at very low bit-widths, but needs a training run. Used when PTQ accuracy at INT4/INT2 isn’t good enough.

The payoff: a 70B model on one GPU

70B params × 2 bytes (FP16) = 140 GB  → needs 2× 80 GB GPUs
70B params × 0.5 byte (INT4) ≈ 35 GB  → fits one 48 GB GPU

That’s the throughline in one line: INT4 turns a two-GPU model into a one-GPU model. Fewer cards, less power, and because you now move ~¼ the weight bytes per token, lower latency too. The dollars fall on every axis at once — for a typically small single-digit-percent dip in accuracy.

It actually happened — outliers that only appear at scale (LLM.int8(), 2022)

The outlier problem isn’t a constant nuisance — it emerges. Dettmers and colleagues (2022), building LLM.int8(), found that below a few billion parameters naive INT8 is basically fine, but past a threshold around ~6–7B parameters a small set of feature dimensions abruptly develop systematic, huge-magnitude outliers that wreck naive quantization across the whole model. Their fix is the blunt-but-effective one this page hints at: decompose the matmul, keep that tiny fraction of outlier dimensions in 16-bit, and run everything else in INT8. It is the discovery that promoted “activation outliers” from a footnote to the central design constraint of every serious method that followed — AWQ and SmoothQuant on the next page are both answers to it.

The architect’s lens

Before the format zoo, settle why quantization is the first lever you reach for:

• Why does it exist? Inference is memory-bound — bytes arrive slower than the math completes — so storing each weight in fewer bits (FP16→INT4) moves ~¼ the bytes, and on a memory-bound workload throughput roughly scales with the inverse of bytes moved.
• What problem does it solve? Memory and bandwidth: INT4 turns a 140 GB FP16 70B model into ~35 GB, collapsing a two-GPU model onto one card and lowering per-token latency, for a typically single-digit-percent accuracy dip.
• What are the trade-offs? Precision for size — you map a continuous range onto a small integer grid via scale and zero-point, and the central difficulty is activation outliers (channels 10–100× larger) that stretch the range and crush ordinary values.
• When should I avoid it? You rarely turn weight quantization off, but don’t quantize activations (weight+activation INT8) unless you’ve tamed outliers and actually need the integer-matmul compute throughput — weight-only INT4 already buys the memory/latency win.
• What breaks if I remove it? The model reverts to its full FP16 footprint: more GPUs, more power, ~4× the weight bytes moved per token, and slower decode on the memory-bound path.

Check your understanding

1. In the affine mapping, what do scale and zero_point each control?
2. A tensor ranges [-0.6, 0.8] quantized to INT8 has scale ≈ 0.0055. What’s the worst-case reconstruction error for any weight, roughly?
3. Why is per-group (e.g. 128-weight) quantization preferred over per-tensor for INT4?
4. Why are activations harder to quantize than weights?
5. Weight-only INT4 leaves activations in FP16. What does it save, and what does it not speed up compared to INT8 weight+activation?

Show answers

1. scale sets the spacing between adjacent integer levels (the step size); zero_point is the integer that maps to real value 0, shifting the grid so asymmetric ranges (and exact zero) are represented.
2. About half a step: scale/2 ≈ 0.0027. Rounding to the nearest level can’t err by more than one half-step.
3. A single large weight forces a big scale for its group only, not the whole matrix, so the precision damage is localized and the other groups keep fine resolution.
4. Activations contain outlier features (channels 10–100× larger), which stretch the quantization range and crush precision for ordinary values; weights are comparatively well-behaved.
5. It saves memory and bytes-moved (so latency on a memory-bound workload), but the matmul still runs in FP16, so it doesn’t gain the integer-matmul compute throughput that INT8 tensor cores provide.