Mixture of Experts

Every lever so far made the model smaller. MoE does the opposite: it makes the model far bigger in total but arranges for only a slice to run on any given token. Recall from the FFN page that the feed-forward block holds most of a transformer’s parameters and does most of its per-token FLOPs. MoE rebuilds exactly that block — and that’s where its whole economics live.

The idea: many experts, a router, top-k

Replace the single dense FFN in a layer with N expert FFNs plus a small router (gating network). For each token, the router scores the experts and sends the token to only the top-k of them (often k=2). The other experts sit idle for that token.

        token
          │
      ┌───────┐
      │ router│  scores all N experts, picks top-k
      └───────┘
       ╱  │  ╲
   [E1] [E2] [E3] … [E8]     ← N experts (all in memory)
    ▲    ▲                    ← only k=2 actually run
    └────┴── combine (weighted) → output

Total capacity scales with N (lots of parameters → lots of knowledge), but compute per token scales with k (only a couple of experts run). That decoupling is the magic: you get a model with the parameter count of a giant, while each token pays the FLOPs of something much smaller.

Activated vs total parameters — the number that matters

This is the key MoE distinction. Total parameters = everything in the checkpoint. Active (activated) parameters = what actually participates in one token’s forward pass.

Worked example — Mixtral 8×7B. Eight experts per MoE layer, router picks top-2:

Total parameters     ≈ 47B   (all 8 experts × all layers + shared attn)
Active per token     ≈ 13B   (only 2 of 8 experts run per layer)

Note it’s not 8×7B = 56B — attention and embedding layers are shared, not replicated, so the total is ~47B. And active is not 2×7B either, for the same reason: ~13B once you count the shared parts that every token uses plus the 2 selected experts. The headline: ~47B of knowledge, ~13B of compute per token. You pay roughly 13B-model FLOPs and latency, but answer with closer to 47B-model quality.

Why this is compute-efficient

On a memory-bound, low-batch inference path, latency tracks the FLOPs and the active bytes touched per token. MoE keeps both near the 13B level, so per-token latency and per-token compute-dollars look like a small model. For the same quality, a dense model would need to activate all ~47B every token — roughly 3.5× the per-token compute.

The catch: hungry on memory

Here’s the throughline’s twist. Only k experts run, but the router could pick any of the N on the next token — so all N experts must be resident in memory at all times.

Memory footprint  → driven by TOTAL params (~47B → ~94 GB in FP16)
Compute per token → driven by ACTIVE params (~13B FLOPs)

So MoE inverts the usual deal: cheap on compute, expensive on memory. Mixtral 8×7B needs the VRAM of a ~47B model even though it computes like a 13B one. You buy quality with memory you must keep powered and resident, whether or not each expert gets used.

MoE is not free efficiency

“Sparsely activated” sounds like pure savings, but the savings are in FLOPs, not memory. If your bottleneck is VRAM (the common LLM constraint), MoE can make things worse, not better. It’s the right tool when you have memory to spare and want frontier quality on a compute/latency budget.

The other costs: routing and batching

• Load balancing. If the router favors a few experts, the rest are dead weight and the popular ones become bottlenecks. Training adds an auxiliary load-balancing loss to spread tokens evenly. It’s a real, ongoing complexity.
• Messier batching. In a batch, different tokens route to different experts, so the neat dense matmul fragments into many small per-expert matmuls of uneven size. Efficient MoE serving needs expert-grouping / capacity-factor tricks and is harder to keep GPU-saturated than a plain dense model.
• Distributed serving. At scale, experts are sharded across GPUs (expert parallelism), so routing means cross-device token shuffles — extra network traffic the dense model never pays.

Where MoE fits among the levers

Lever	Total params	Active per token	Memory	Compute/token
Dense	N	N	N	N
Quantized dense	N (fewer bits)	N	↓↓	↓
Distilled	small	small	↓↓	↓↓
MoE	large	small	↑ (high)	↓ (low)

The one-line summary

MoE trades memory for compute: lots of parameters resident in VRAM, but only a fraction switched on per token — big-model quality at small-model FLOPs, if you can afford to hold the whole thing.

The dollars view

A dense ~47B model and Mixtral 8×7B occupy similar VRAM, so the capital cost (GPUs to hold the weights) is comparable. But the MoE serves each token at ~13B compute — so for a compute/latency-bound workload it’s far cheaper per token and faster, at equal quality. The verdict: MoE wins when you can pay the memory entry fee and your recurring cost is dominated by compute, not by fitting the weights at all.

Under the hood — the router is tiny, and experts aren’t “topics”

The router is almost anticlimactically small: a single linear projection from the hidden state to N logits, a softmax, then the top-k pick — a rounding error next to the experts it steers. And the intuitive story that each expert becomes a domain specialist (“this one handles French, that one handles code”) mostly does not hold. Analysis of Mixtral’s routing found little semantic or topical specialization; which expert fires correlates more with surface/syntactic features and token position than with subject matter, and the choice is remade at every layer for every token. The experts are interchangeable capacity that the router learned to keep balanced, not a panel of subject-matter specialists.

It actually happened — the 1.6-trillion-parameter Switch Transformer (2021)

Google’s Switch Transformer (Fedus, Zoph & Shazeer, 2021) pushed MoE to its logical extreme: route each token to just one expert (top-1 — the “switch,” the cheapest possible routing) and scale total parameters to ~1.6 trillion, far beyond any dense model of its day, while per-token FLOPs stayed roughly constant. It also popularized the auxiliary load-balancing loss this page describes, because top-1 routing collapses onto a few favourite experts even more eagerly than top-2. The result proved the core MoE bargain at scale — parameter count and compute-per-token are separable knobs — and its memory bill scaled with the full 1.6T, exactly the trade the rest of this page is about.

The architect’s lens

The five questions reframe MoE as a memory-for-compute bargain:

• Why does it exist? To decouple a model’s parameter count from its per-token compute — total capacity scales with N experts (knowledge) while compute scales with k (only the top-2 run), so you get ~47B-quality answers at ~13B FLOPs.
• What problem does it solve? The cost of frontier quality on a compute/latency budget: Mixtral 8×7B answers like a ~47B dense model while each token pays only ~13B-model latency and compute-dollars.
• What are the trade-offs? It inverts the usual deal — cheap on compute, expensive on memory, because the router could pick any expert next token, so all ~47B (≈94 GB in FP16) must stay resident. It also adds a load-balancing loss, fragmented batching, and cross-GPU expert shuffles.
• When should I avoid it? When your bottleneck is VRAM — the common LLM constraint — where MoE can make things worse, not better.
• What breaks if I remove it? You fall back to a dense model of equal quality: similar memory, but you must activate all ~47B parameters every token — roughly 3.5× the per-token compute.

Check your understanding

1. What does an MoE layer replace, and what two components does it introduce?
2. For Mixtral 8×7B, why is “total” ~47B rather than 56B, and why is “active” ~13B rather than 14B?
3. MoE reduces FLOPs per token but is called “hungry on memory.” Explain the apparent contradiction.
4. Why does MoE need a load-balancing loss during training?
5. Under what bottleneck does MoE pay off, and under what bottleneck might it actually hurt?

Show answers

1. It replaces the dense feed-forward (FFN) block with N expert FFNs plus a router/gating network that scores experts and selects the top-k to run per token.
2. Attention and embedding layers are shared across experts rather than replicated, so total is ~47B not 8×7B=56B; active is ~13B (not 2×7B) because each token uses those shared layers plus only the 2 selected experts.
3. Compute per token tracks the active params (~13B), but the router may pick any expert next token, so all experts (total ~47B) must stay resident in memory — low FLOPs, high memory.
4. Without it the router tends to over-use a few experts, leaving others as dead weight and overloading the popular ones; the auxiliary loss spreads tokens evenly so capacity is actually used.
5. It pays when the bottleneck is compute/latency and you have spare VRAM; it can hurt when the bottleneck is memory/VRAM, since it needs the footprint of its large total parameter count.