Pruning & Sparsity

Distillation trains a smaller model; quantization shrinks each weight. Pruning takes a third route: keep the architecture, but delete the weights that contribute little. It sounds like the easiest win of all — yet sparsity is famously hard to cash in. This page is mostly about why.

The idea: most weights barely matter

A trained network is over-parameterized. Plenty of weights are tiny; zeroing them changes the output almost not at all. Magnitude pruning is the simplest rule: rank weights by absolute value, set the smallest fraction to zero. Prune 50% of the smallest weights and, after a little fine-tuning, accuracy often barely moves.

So memory should halve and compute should halve — right? Not quite. The catch is which weights you remove and whether the hardware can skip them.

Unstructured vs structured

UNSTRUCTURED                       STRUCTURED
zero scattered individual weights  drop whole rows/cols/heads/neurons

[ 0  3  0  7 ]                     remove entire columns →
[ 5  0  0  0 ]                     [ 3  7 ]
[ 0  0  9  0 ]                     [ 0  0 ]   (smaller, dense matrix)
[ 2  0  0  4 ]                     [ 0  0 ]
hard for hardware to exploit       a genuinely smaller dense matmul

• Unstructured pruning zeros individual weights anywhere. You can reach high sparsity (80–90%) with little accuracy loss — but the matrix is now a dense grid full of zeros. A GPU doing a dense matmul still multiplies by those zeros. You save memory only if you store it in a sparse format, and sparse kernels rarely beat dense ones until sparsity is extreme. So the FLOPs you “saved” are largely not realized as speed.
• Structured pruning removes whole units — an entire attention head, a neuron (FFN channel), a layer. The remaining weights form a smaller dense matrix, which runs faster on ordinary hardware with no special kernels. The win is real and immediate. The cost: it’s coarser, so the same accuracy budget buys you less sparsity.

	Unstructured	Structured
Granularity	individual weights	heads / neurons / layers
Achievable sparsity	high (80–90%)	lower (often 20–50%)
Hardware speedup	little (needs sparse kernels)	yes (smaller dense matmul)
Memory saved	only with sparse storage	yes, directly
Accuracy at equal sparsity	better	worse

The core tension

Unstructured pruning preserves accuracy but the hardware can’t easily exploit it. Structured pruning is hardware-friendly but hurts accuracy faster. You usually can’t have both — which is exactly why sparsity is harder to monetize than quantization.

2:4 semi-structured sparsity — the middle path

NVIDIA GPUs from Ampere (2020) onward support a 2:4 pattern: in every contiguous block of 4 weights, exactly 2 are zero. It’s regular enough that the sparse tensor cores can skip the zeros and deliver up to ~2× matmul throughput, yet fine-grained enough to keep accuracy closer to unstructured. You typically prune to the 2:4 mask, then fine-tune to recover.

every group of 4 → keep 2, zero 2
[ a  0  b  0 | 0  c  0  d ]   ← hardware-skippable, ~2× math

This is the one sparsity scheme that reliably turns into wall-clock speed on commodity accelerators — and even then only ~2×, versus quantization’s ~4× memory cut for free.

You almost always fine-tune afterward

Pruning damages the network; fine-tuning lets the survivors compensate. The standard loop:

train → prune (mask smallest) → fine-tune to recover → (repeat / increase sparsity)

Iterative pruning (prune a little, recover, repeat) reaches higher sparsity than one aggressive cut. This fine-tuning step is an extra cost distillation and PTQ-style quantization can avoid — another reason quantization is the more common first reach.

Why sparsity is harder to cash in than quantization

The honest comparison, since this part is about turning model changes into dollars:

• Quantization: every bit dropped is guaranteed fewer bytes moved and stored — the saving is automatic and dense-kernel-friendly. INT4 is ~4× smaller, full stop.
• Pruning: a zeroed weight only helps if your storage format and your matmul kernel both skip it. On dense hardware, scattered zeros buy nothing. Only structured or 2:4 sparsity converts to speed, and at a steeper accuracy price.

Practical takeaway

Reach for quantization first — it’s the reliable, large, low-effort win. Use structured pruning or 2:4 when you specifically need to shrink the architecture or hit a latency target on supporting hardware, and budget for a fine-tuning run. Unstructured pruning is great for research and model analysis, less so for production speed.

The dollars view

Suppose 2:4 sparsity gives ~1.5× real speedup on a supported GPU after fine-tuning, costing one recovery training run. Versus INT4 quantization: ~4× memory reduction and proportional latency on a memory-bound workload, with only a short calibration pass. For most teams the quantization ROI dominates — which is why pruning, despite its elegant premise, is the second lever you reach for, not the first.

It actually happened — the Lottery Ticket Hypothesis (2019)

Frankle & Carbin (MIT, 2019) noticed something unsettling: inside a trained dense network hides a small subnetwork — a “winning ticket” — that, if you rewind its surviving weights to their original initialization and train only those, matches the full network’s accuracy. They found these tickets by iterative magnitude pruning: train, prune the smallest weights, rewind, repeat. The lesson cuts both ways. It says the capacity to be 80–90% sparse was latent from the start — but it needed the right initialization, discovered only via the full dense training run, so it is not a cheap recipe for training small from scratch. And a winning ticket is still unstructured: fast in theory, slow on a commodity GPU — the very gap between “fewer weights” and “fewer dollars” this page keeps circling.

The architect’s lens

Pruning’s elegant premise meets the hardware that has to cash it in:

• Why does it exist? Trained networks are over-parameterized — many weights are near-zero and zeroing them barely moves the output — so in principle you can delete them almost for free.
• What problem does it solve? Model size and, only sometimes, latency: structured pruning (whole heads/neurons/layers) leaves a genuinely smaller dense matmul, and 2:4 semi-structured sparsity buys up to ~2× on NVIDIA sparse tensor cores.
• What are the trade-offs? Unstructured pruning preserves accuracy but a dense matmul still multiplies by the scattered zeros, so the saved FLOPs aren’t realized as speed; structured pruning is hardware-friendly but hurts accuracy faster — and either way you usually pay a fine-tuning recovery run.
• When should I avoid it? As your first lever — quantization gives a reliable ~4× memory cut for free, so reach for pruning only when you must shrink the architecture or hit a latency target on supporting hardware.
• What breaks if I remove it? Little for most teams — you keep the full dense model and lean on quantization, whose ROI dominates; you forgo only the extra structured/2:4 speedup.

Check your understanding

1. Why doesn’t unstructured pruning to 80% sparsity automatically make a model 5× faster on a normal GPU?
2. What’s the accuracy-vs-hardware trade-off between unstructured and structured pruning?
3. What does the “2:4” pattern mean, and why can hardware exploit it when arbitrary sparsity can’t?
4. Why does pruning typically require a fine-tuning step that PTQ quantization does not?
5. In one sentence, why is sparsity harder to cash in than quantization?

Show answers

1. The zeros are scattered through a dense matrix; a standard dense matmul still multiplies by them. Without a sparse storage format and sparse kernels (which rarely beat dense until sparsity is extreme), the saved FLOPs aren’t realized as speed.
2. Unstructured keeps accuracy better at a given sparsity but the hardware can’t exploit it; structured removes whole units so the remaining dense matmul is genuinely faster, but it hurts accuracy faster at the same sparsity.
3. In every block of 4 weights exactly 2 are zero; the regular, predictable pattern lets NVIDIA sparse tensor cores skip the zeros for up to ~2× matmul throughput, which is impossible with arbitrary unstructured zeros.
4. Pruning deletes weights and damages the network’s function, so fine-tuning lets the surviving weights compensate; PTQ only rounds weights and can recover acceptable accuracy from a small calibration pass without gradient updates.
5. A dropped bit always means fewer bytes moved, but a zeroed weight only helps if both the storage format and the matmul kernel skip it — which only structured/2:4 patterns reliably allow, and at a higher accuracy cost.