The FFN: Where the Parameters Live

Attention mixes information across tokens, and we saw its cost scales with sequence length. But attention is only half of a transformer layer. The other half — the feed-forward network — is where the parameters actually live, and parameters are what you pay to store in memory and load on every decode step.

A transformer block = attention + FFN

Each layer does two things in sequence, each wrapped with a residual connection and normalization:

  x ──►[ LayerNorm ]──►[ Attention ]──►(+ x)──►[ LayerNorm ]──►[ FFN ]──►(+)──► out
                         mix tokens                              per-token

Attention lets tokens talk to each other. The FFN (feed-forward network, also called the MLP) then processes each token’s vector independently — same weights applied at every position. It’s where the model does most of its per-token “thinking,” and where most of its weights sit.

What the FFN does: expand, activate, project

The FFN is two linear layers with a non-linearity between them. The trick is that it expands the hidden dimension before projecting back:

   d_model            d_ff = 4 × d_model        d_model
  ┌────────┐  W1     ┌──────────────────┐  W2  ┌────────┐
  │ 4096   │ ──────► │      16384        │ ───► │ 4096   │
  └────────┘ expand  └────────┬─────────┘ proj └────────┘
                       activation (GELU/SwiGLU)

• W1 projects up from d_model to d_ff, conventionally ~4× larger (4,096 → 16,384).
• A non-linearity (GELU, or SwiGLU in modern models) is applied.
• W2 projects back down to d_model.

That 4× expansion is the point: it’s where the bulk of the weights concentrate.

Why most parameters live here

Count the matrices in one layer, with d = d_model and d_ff = 4d:

Matrix	Shape	Params
Attention Q, K, V, O (4 matrices)	d × d each	4d²
FFN W1 (up)	d × 4d	4d²
FFN W2 (down)	4d × d	4d²

The FFN’s two matrices alone are 8d², versus 4d² for all of attention’s projections. So roughly two-thirds of every layer’s parameters are in the FFN (and essentially all of the rest are the attention projection matrices — not the attention computation itself, which has no weights beyond Q/K/V/O).

Note

Note what is not a parameter: the n×n attention scores from the last page. Those are transient activations, recomputed per request. The FFN and projection weights are fixed — loaded into GPU memory once and reused for every token of every request. That’s why they dominate the memory footprint and the decode-time bandwidth bill.

Why this dominates inference cost

Two consequences, both throughline:

• Weight memory: the FFN + projections are the model’s size on disk and in VRAM. When we say “an 8B model needs ~16 GB in FP16,” most of those bytes are FFN weights.
• Matmul FLOPs and bandwidth: during decode you generate one token at a time, so the matmuls are tiny — but you still must read every weight from memory to do them. Since the FFN holds most weights, it dominates the memory traffic that makes decode memory-bound. Shrinking these weights (quantization, MoE) is the highest-leverage cost cut there is.

Worked example: estimate parameters from n_layers and d_model

A clean rule of thumb: a transformer’s parameter count is approximately

$$N \approx n_{\text{layers}} \times 12 \times d_{\text{model}}^2$$

The 12 comes from per-layer: 4d² (attention Q/K/V/O) + 8d² (FFN up+down) = 12d².

Plug in Llama-3-8B: n_layers = 32, d_model = 4,096.

• d² = 4,096² = 16,777,216
• per layer: 12 × 16,777,216 ≈ 201M params
• × 32 layers ≈ 6.4 billion

Add the ~1 GB embedding table and the final output projection and you land near the advertised 8B. The estimate is close because the formula captures where the weight actually is: 8 of every 12 d² — two-thirds — sits in the FFN.

In FP16, 8B params × 2 bytes = 16 GB just to hold the model. That number is the floor under every latency and dollar figure for this model: you cannot decode a single token without those 16 GB resident and readable.

Sets up Mixture of Experts

If two-thirds of the weights are in the FFN, the obvious question is: do we need all of them for every token? Mixture of Experts (Part 4) replaces the single big FFN with many smaller expert FFNs and activates only a couple per token — keeping the model’s knowledge capacity while loading a fraction of the FFN weights per step. It’s a direct attack on exactly the cost this page just located.

Under the hood — modern FFNs have three matrices, not two

The “expand with W1, project with W2” picture is the classic GELU FFN. Most current models (Llama, Mistral, PaLM) instead use a gated variant — SwiGLU — which splits the up-projection into two parallel matrices, a gate and an up, combines them element-wise after one passes through a SiLU/Swish activation, then a third down matrix projects back.

  classic GELU FFN:   x → [W_up] → GELU → [W_down]                  (2 matrices)
  SwiGLU FFN:         x → ( SiLU([W_gate]) ⊙ [W_up] ) → [W_down]    (3 matrices)

Three matrices instead of two would add ~50% more FFN parameters at the same width, so models that adopt SwiGLU shrink the hidden width to compensate — d_ff drops from 4× d_model to about 8/3 ≈ 2.67× — keeping the parameter count (and therefore the weight-memory and decode bandwidth this page is all about) roughly constant. You spend a little extra compute for measurably better quality-per-parameter, which on a memory-bound workload is close to free. The two-thirds-in-the-FFN accounting still holds; only the internal recipe changed.

The architect’s lens

The FFN is two-thirds of the model by weight — so it sets most of the bill. The five questions:

• Why does it exist? Attention mixes information across tokens but carries few weights; the FFN transforms each token’s vector independently — expand d_model to ~4× wider, apply a nonlinearity, project back — which is where the model does most of its per-token “thinking.”
• What problem does it solve? Per-token capacity: its two matrices are 8d² of the 12d² in each layer, so roughly two-thirds of the parameters — the model’s stored knowledge — live here.
• What are the trade-offs? Those weights are the model’s size (most of an 8B model’s 16 GB) and, because decode must read every weight from HBM per token, the FFN dominates the memory-bound bandwidth bill — you cannot shrink it without touching quality.
• When should I avoid it? You never remove it, but when FFN weight-loading is the cost, the lever is Mixture of Experts (activate only a couple of expert FFNs per token) or quantization — load a fraction of those 8d² weights per step instead of all of them.
• What breaks if I remove it? The model loses most of its parameters and its only per-token nonlinear transform; you are left with token-mixing that has nowhere to store what it learned.

Check your understanding

1. What are the two sub-components of a transformer block, and which one applies the same weights independently at each token position?
2. Describe the FFN’s expand-activate-project shape and the conventional expansion factor.
3. Counting d² terms, why do roughly two-thirds of a layer’s parameters live in the FFN rather than in attention?
4. The attention scores are large but are not parameters. What’s the difference between a parameter and an activation, and why does it matter for the memory bill?
5. Using N ≈ n_layers × 12 × d_model², estimate the parameters for a model with 32 layers and d_model = 4,096, and state its FP16 memory footprint.

Show answers

1. Attention (mixes information across tokens) and the FFN/MLP (applies the same weights to each token independently).
2. W1 projects d_model up to d_ff (~4× larger), a non-linearity like GELU/SwiGLU is applied, then W2 projects back down to d_model. The conventional expansion factor is 4×.
3. Attention’s Q/K/V/O projections total 4d²; the FFN’s up and down matrices total 8d². So FFN is 8d² of the 12d² per layer — two-thirds.
4. A parameter is a fixed learned weight loaded once and reused for every token; an activation (like the attention scores) is transient, recomputed per request. Parameters dominate persistent VRAM and the per-token memory traffic that makes decode memory-bound.
5. 12 × 4,096² × 32 ≈ 6.4B params (≈8B with embeddings and output head); at 2 bytes/param in FP16, ≈16 GB.