What a Model Actually Is

The Cost Problem showed that a token costs ~2 × params FLOPs. To understand the memory side of the bill — and every compression trick later in the book — you need to know what a “parameter” physically is. The answer is refreshingly boring: a model is a large pile of numbers arranged into matrices. Demystifying this now makes quantization, KV caches, and the rest feel like simple consequences rather than magic.

A model is numbers in matrices

When people say a model “has 7 billion parameters,” they mean it stores 7 billion individual numbers — the weights — learned during training. These numbers are organized into matrices (2D grids of numbers) and grouped into layers. A 70B model isn’t 70B of mystery; it’s 70B floats sitting in GPU memory, waiting to be multiplied.

A layer's core operation:

   input vector            weight matrix           output vector
   [ x1 x2 x3 ... ]   ×   [ w11 w12 ...  ]   =   [ y1 y2 ... ]
                          [ w21 w22 ...  ]
                          [  ...         ]

   Each output = sum of (input × weight). That's it.

Layers and the forward pass

A transformer stacks dozens of identical layers (e.g. ~80 for a 70B model). Your input — a sequence of token vectors — flows through layer 1, whose output feeds layer 2, and so on to the top, which emits a score for every word in the vocabulary. That top-to-bottom flow is the forward pass from the last page.

The crucial point for cost: the forward pass is overwhelmingly matrix multiplications. Attention and the feed-forward blocks are all matmuls between your activations and those stored weight matrices. A handful of other operations (normalization, the softmax, activation functions) exist, but they are cheap. So “running a model” is, to first order, “multiply a lot of matrices using the stored weights” — which is exactly why we counted FLOPs as 2 × params.

Where the parameters actually live

In most LLMs the majority of parameters sit in the feed-forward (FFN) blocks, not attention. We quantify this in The FFN: Where the Parameters Live. For now, just hold that params = weights = the numbers being multiplied.

Params × bytes = memory

Here is the formula that governs the entire memory story:

Memory for weights ≈ (number of parameters) × (bytes per parameter)

Each number needs storage, and how many bytes depends on its numeric precision — the format used to store a float:

Precision	Bytes/param	Note
FP32 (full)	4	Training default; rarely needed for inference
FP16 / BF16 (half)	2	The standard inference baseline
INT8	1	8-bit integer quantization
INT4	0.5	4-bit; aggressive, very memory-cheap

Now apply it to a 7B model and watch the memory bill move:

FP32:  7e9 × 4   = 28 GB
FP16:  7e9 × 2   = 14 GB   ← the usual baseline
INT8:  7e9 × 1   =  7 GB
INT4:  7e9 × 0.5 = 3.5 GB

So “a 7B model in FP16 is ~14 GB” is not a fact to memorize — it’s just 7 billion × 2 bytes. The same math scales: a 70B model in FP16 is ~140 GB, which already exceeds a single 80 GB GPU and forces you to split it across several (a problem we tackle in Part 5).

Why this is the master lever

Halving bytes-per-param roughly halves the weight memory — and, because decode is memory-bound, often speeds it up too. That single observation is the seed of all of Part 4 (Model Compression). When you later read “we quantized to INT4,” translate it instantly to “weight memory dropped ~4× versus FP16.” The full precision spectrum gets its own page at Numeric Precision.

Weights are not the whole bill

The stored weights are the fixed cost — present the moment you load the model. Two more memory consumers appear at runtime: activations (intermediate vectors as data flows through layers) and the KV cache (saved attention state that grows with context length and batch size). For short prompts the weights dominate; for long contexts and big batches the KV cache can rival or exceed them. We size that carefully in KV Cache Math. For now, the foundation is set:

A model is numbers in matrices. Count them and you get FLOPs (×2). Multiply them by bytes and you get memory. Everything else is detail.

Under the hood — not every parameter costs the same to run

The “params × bytes = memory” rule is exact for storage, but the FLOPs side has a wrinkle. A sizable slab of an LLM’s parameters lives in two vocabulary-sized matrices: the token embedding (look up a row per input token) and the output projection that scores every word in the vocabulary. The embedding is a gather — you read only the rows for the tokens you actually have, not a full matmul — so those parameters cost memory to store but almost no FLOPs to run. The weight matrices inside the layers are different: every one is multiplied on every token, which is why the FLOPs ≈ 2 × params rule holds for the layer weights even though a slice of the total parameter count barely participates in the arithmetic. For a small model with a large vocabulary, that embedding table can be a surprising fraction of the parameters — pure memory cost with little compute attached.

Check your understanding

1. In plain terms, what is a parameter, and how are parameters organized inside a model?
2. Why can we say the forward pass is “mostly matrix multiplications”?
3. Write the formula for weight memory and use it to explain why a 7B FP16 model needs ~14 GB.
4. Fill in the memory for a 7B model at FP32, INT8, and INT4. How does INT4 compare to FP16?
5. Besides the stored weights, name two other things that consume GPU memory at inference time.

Show answers

1. A parameter is a single learned number (a weight). Parameters are arranged into matrices, which are grouped into layers that stack to form the model.
2. Attention and the feed-forward blocks are implemented as matmuls between the input activations and the stored weight matrices; the non-matmul operations (normalization, softmax, activations) are comparatively cheap.
3. Memory ≈ params × bytes/param. For 7B in FP16: 7 × 10⁹ × 2 bytes = 14 × 10⁹ bytes ≈ 14 GB.
4. FP32: 28 GB; INT8: 7 GB; INT4: 3.5 GB. INT4 (0.5 bytes/param) uses about one-quarter the memory of FP16 (2 bytes/param).
5. Activations (intermediate vectors flowing through the layers) and the KV cache (saved attention keys/values that grow with context length and batch size).