Distillation

Quantization shrinks the bits of an existing model. Distillation does something different: it trains a brand-new, smaller model to imitate a big one. You pay an upfront training run once, and then serve a model that is cheaper on every token for the rest of its life. That trade — one training bill for a permanently lower inference bill — is the whole point.

The idea

Take a large, accurate teacher model and a small student model. Train the student not just on the right answers, but on the teacher’s full output distribution. The student learns to produce the same outputs as the teacher, at a fraction of the size and cost.

   ┌─────────────┐   soft labels / logits   ┌──────────────┐
   │  TEACHER    │ ───────────────────────▶ │   STUDENT    │
   │  (large)    │                          │   (small)    │
   └─────────────┘   "imitate me"           └──────────────┘
   expensive, once                          cheap, forever

Why soft labels carry more signal

A hard label says “this token is cat, everything else is wrong.” The teacher’s soft labels — its probabilities over all tokens — say much more: “85% cat, 10% kitten, 4% feline, …”. That distribution encodes the teacher’s learned similarity structure (a cat is closer to a kitten than to a truck). The student learns this richer signal in one example instead of needing thousands to infer it.

To expose this structure you soften the teacher’s distribution with a temperature T before computing the loss:

softmax_T(z)_i = exp(z_i / T) / Σ_j exp(z_j / T)

Higher T flattens the distribution, amplifying the small probabilities (the “dark knowledge”) that carry the relational information. A typical loss blends two terms:

L = α · KL(student_soft ‖ teacher_soft)   ← match the teacher
  + (1-α) · CE(student, hard_label)        ← still get the truth right

Why it works

The teacher has already done the expensive job of carving a good function out of a huge parameter space. The student doesn’t have to rediscover that function from raw data — it just has to fit the teacher’s outputs, a far easier target than the original task. A small model trained from scratch on hard labels often underfits; the same small model trained to mimic a strong teacher reaches noticeably higher quality, because the teacher hands it a smoother, more informative target.

Three flavors

Type	Student matches…	Cost / detail
Response (logit)	teacher’s output probabilities	simplest, most common; only needs teacher outputs
Feature	teacher’s intermediate hidden states	richer signal, needs layer alignment
Relation	relationships between examples/layers	matches structure, not just values

Response-based distillation is the default — you can even do it through an API by collecting the teacher’s outputs on a corpus of prompts.

Examples

• DistilBERT-style — a classic: a student with ~40% fewer parameters than BERT retains roughly ~95–97% of its quality while running materially faster. The template for “encoder distillation.”
• Smaller chat models — many compact instruct/chat models are trained on outputs generated by a larger, stronger model (sequence-level distillation: the teacher’s generated text becomes the student’s training data). The small model inherits much of the big model’s behavior at a fraction of serving cost.

Distillation vs quantization

They’re complementary. Quantization keeps the same model but shrinks each weight; distillation produces a genuinely smaller architecture (fewer layers/width). You can distill first, then quantize the student — the savings stack.

The cost, stated plainly

Distillation is not free magic:

• Upfront training run. You need the teacher (often the expensive part — generating soft labels or synthetic data over a big corpus) plus the student’s training compute. This is real money, paid once.
• Some quality loss. The student is smaller; it will trail the teacher, especially on the long tail of hard cases. You’re trading a few accuracy points for a large, permanent cost reduction.

When distillation pays

The economics favor distillation when inference volume is large and sustained. If you serve billions of tokens, halving the per-token cost dwarfs a one-time training bill. For a low-traffic internal tool, the training run may never amortize — just quantize an off-the-shelf model instead.

The dollars view

Concretely: suppose a 70B teacher costs $X per million tokens to serve and a distilled 7B student that keeps ~90% of the quality costs roughly $X/8. The distillation run might cost the equivalent of a few million tokens of teacher inference. If you serve hundreds of millions of tokens a month, that one-time cost is recovered in days, and from then on every token is ~8× cheaper — in latency, memory, and dollars — for the life of the deployment.

Under the hood — why the loss carries a T² factor

Softening with temperature T doesn’t only flatten the teacher’s distribution — it also shrinks the gradient the student receives from the soft term, by roughly 1/T². The original distillation work (Hinton, Vinyals & Dean, 2015) multiplies the soft-label loss by T² to cancel that, so the soft and hard terms keep comparable pull on the weights no matter how high you crank T. Forget the factor and a high temperature quietly starves the very signal you raised T to expose.

It actually happened — DistilBERT (2019)

When HuggingFace distilled BERT-base they didn’t train on logits alone — they used a triple loss: the soft-label distillation term, the usual masked-language-modelling term, and a cosine-embedding term that aligns the student’s hidden states with the teacher’s (a feature-distillation touch on top of response distillation). The student kept 6 of BERT-base’s 12 layers, initialized from every other teacher layer, landing at ~66M parameters versus ~110M — about 40% smaller, ~60% faster at inference, while retaining roughly 97% of BERT-base’s score on the GLUE benchmark. It became the canonical proof that a small encoder can be halved almost for free, and it is still a sensible default when one must run cheaply.

The architect’s lens

Step back from soft labels and temperature and weigh distillation as a deployment decision:

• Why does it exist? Because a small model trained from scratch on hard labels underfits, while the same model trained to mimic a strong teacher’s full output distribution reaches noticeably higher quality — you buy a permanently cheaper model with one training run.
• What problem does it solve? A high, recurring per-token inference bill: a distilled 7B student keeping ~90% of a 70B teacher’s quality serves each token at roughly 1/8th the cost — in latency, memory, and dollars — forever.
• What are the trade-offs? It moves cost from inference (recurring) to training (one-time, often dominated by generating the teacher’s soft labels) and accepts some quality loss, especially on the long tail of hard cases.
• When should I avoid it? When inference volume is low — the training run may never amortize, so just quantize an off-the-shelf model instead.
• What breaks if I remove it? Nothing structurally — you either keep serving the big teacher (higher quality, far higher cost) or ship a from-scratch small model that underfits the same data.

Check your understanding

1. Why do a teacher’s soft labels carry more learning signal than the hard ground-truth label?
2. What does the temperature T do in the distillation softmax, and why is a higher T useful?
3. Distinguish response, feature, and relation distillation in one line each.
4. Why can a small model distilled from a strong teacher beat the same small model trained from scratch on hard labels?
5. Under what traffic conditions does distillation’s economics actually pay off, and when does it not?

Show answers

1. They give a full probability distribution over all classes, encoding the teacher’s learned similarity structure (which wrong answers are close), so the student learns relational “dark knowledge” the single hard label can’t convey.
2. T softens (flattens) the distribution by dividing logits before softmax; higher T amplifies the small probabilities that carry the relational information, exposing the teacher’s structure to the student.
3. Response = match the teacher’s output probabilities; Feature = match its intermediate hidden states; Relation = match relationships between examples/layers rather than raw values.
4. The teacher provides a smoother, more informative target than raw hard labels; fitting that target is an easier optimization problem than rediscovering the function from scratch, so the student underfits less.
5. It pays when inference volume is large and sustained, so the permanently lower per-token cost outweighs the one-time training bill; for low-traffic workloads the training run may never amortize, and quantizing an existing model is better.