Memory Bandwidth vs Compute: The Roofline

In GPUs From First Principles we met two competing numbers: how many FLOPs the chip can do per second, and how many bytes per second it can read from HBM. This page gives you the single tool that decides which one limits any given kernel — the roofline model — and the one quantity it hinges on: arithmetic intensity.

Two clocks running at once

Every kernel does two things: it moves bytes (reads inputs from memory, writes outputs back) and it does FLOPs (floating-point multiply-adds). These happen on two separate “budgets”:

• The compute budget: peak FLOP/s. Say ~1000 TFLOP/s for FP16 on an H100.
• The bandwidth budget: peak bytes/s from HBM. Say ~3.35 TB/s.

A kernel takes as long as the slower of the two. If it needs more time to fetch its bytes than to crunch its FLOPs, the cores finish early and wait — you are memory-bound. If the FLOPs take longer, the memory system idles — you are compute-bound.

Arithmetic intensity: FLOPs per byte

The deciding ratio is arithmetic intensity (AI):

arithmetic intensity = total FLOPs done / total bytes moved
                       (units: FLOPs per byte)

It answers: for every byte I drag out of memory, how much math do I get out of it? High intensity means you reuse each byte many times — the chip can stay busy. Low intensity means you touch each byte once and move on — memory can’t keep up and the cores starve.

The roofline

Plot achievable performance (FLOP/s) against arithmetic intensity (FLOPs/byte). You get a roof with two pieces:

 performance
 (FLOP/s)
   ^
   │                      ________________  ← compute roof (peak FLOP/s)
   │                     /
   │                    /   COMPUTE-BOUND
   │                   /    (flat: limited by FLOP/s)
   │   MEMORY-     /·/
   │   BOUND      / ·
   │   (sloped:  /  · ridge point
   │   limited  /   ·   AI* = peak FLOP/s ÷ bandwidth
   │   by BW)  /    ·
   │         /     ·
   └────────────────────────────────────────►  arithmetic intensity
            low                      high       (FLOPs/byte)

• On the left, the slanted line is the memory bandwidth: performance = AI × bandwidth. Double the intensity, double the achievable FLOP/s — because the wall is how fast bytes arrive.
• On the right, the line goes flat at peak FLOP/s. More intensity buys nothing; you are already saturating the compute units.
• The corner where they meet is the ridge point. Its intensity is:

AI* (ridge) = peak FLOP/s / peak bandwidth
            = 1000 × 10^12 FLOP/s  /  3.35 × 10^12 byte/s
            ≈ 300 FLOPs per byte

The rule is brutal and simple: if your kernel’s arithmetic intensity is below the ridge point, you are memory-bound, full stop. No amount of faster compute helps; you need to move fewer bytes or reuse them more. For an H100, you need to do ~300 FLOPs for every byte you read just to break even with the memory system. That is a high bar — and most of inference falls far short of it.

Worked example: matmul vs matrix-vector

Let’s compute arithmetic intensity for two operations using FP16 (2 bytes/element).

Case A — a big square matmul. Multiply two N×N matrices, N = 4096.

FLOPs  = 2 · N^3            = 2 · 4096^3   ≈ 1.37 × 10^11 FLOPs
bytes  = 3 · N^2 · 2        = 3 · 4096^2 · 2 ≈ 1.0 × 10^8 bytes
         (read A, read B, write C; 2 bytes each)

arithmetic intensity = 1.37e11 / 1.0e8 ≈ 1365 FLOPs/byte

1365 is far above the ~300 ridge point → compute-bound. Each element of A and B is reused N times across the output, so the bytes pay for themselves many times over. This is the GPU’s happy place.

Case B — a matrix-vector product. Multiply the same N×N matrix by a single vector (N = 4096).

FLOPs  = 2 · N^2           = 2 · 4096^2     ≈ 3.4 × 10^7 FLOPs
bytes  ≈ N^2 · 2           = 4096^2 · 2     ≈ 3.4 × 10^7 bytes
         (the matrix dominates; the vector is negligible)

arithmetic intensity ≈ 3.4e7 / 3.4e7 ≈ 1 FLOP/byte

About 1 FLOP/byte — hundreds of times below the ridge point → hopelessly memory-bound. Every weight in the matrix is read once and multiplied just once; there is no reuse to amortize the read. The cores spend ~99.7% of their time waiting for HBM.

Operation	Arithmetic intensity	vs ridge (~300)	Regime
4096³ matmul	~1365 FLOPs/byte	above	compute-bound
4096 matrix·vec	~1 FLOP/byte	far below	memory-bound

Note

The two cases use the same weights. The only difference is whether you push a whole batch of vectors (a matrix) through them, reusing each weight many times, or a single vector, reusing each weight once. That difference — batch size — is the lever that moves you along the roofline. Hold that thought.

The cost lens

The roofline tells you which optimization will help before you write it. If you are memory-bound (left of the ridge), buying a chip with more FLOP/s is wasted money; the fix is fewer/cheaper bytes — quantization, fusion, batching. If you are compute-bound, the opposite. Knowing your side of the ridge is the cheapest way to avoid optimizing the wrong thing.

That matrix-vector product is not a toy. It is exactly what LLM token generation looks like — which is why the next page argues that inference is fundamentally memory-bound.

It actually happened — the roofline gets its name (2009)

The diagram on this page isn’t folklore; it has a citation. In April 2009, Communications of the ACM published “Roofline: An Insightful Visual Performance Model for Multicore Architectures” by Samuel Williams, Andrew Waterman, and David Patterson at Berkeley. They were not thinking about transformers — GPUs were barely used for general compute and the deep-learning boom was years away — yet the model they drew to reason about multicore CPUs turned out to be the right lens for GPU kernels too. The insight that made it stick: a kernel’s fate is decided by one number, arithmetic intensity, and you can read off whether compute or bandwidth is your wall by where it lands under the roof. Fifteen-plus years later it is the first thing an inference engineer sketches when asked “why is decode slow?”

Check your understanding

1. Define arithmetic intensity and give its units.
2. What does the slanted part of the roofline physically represent, and what does the flat part represent?
3. How is the ridge-point intensity computed, and roughly what is it for an H100?
4. Why is a large N×N·N×N matmul compute-bound while an N×N·vector product is memory-bound, even with identical weights?
5. If a kernel sits left of the ridge, why won’t a GPU with higher peak FLOP/s make it faster?

Show answers

1. Arithmetic intensity = total FLOPs done ÷ total bytes moved, in FLOPs per byte. It measures how much math you extract per byte read from memory.
2. The slanted part is the memory bandwidth limit (performance = intensity × bandwidth); the flat part is the compute limit (peak FLOP/s), where more intensity buys nothing.
3. Ridge intensity = peak FLOP/s ÷ peak bandwidth ≈ 1000 TFLOP/s ÷ 3.35 TB/s ≈ 300 FLOPs/byte for an H100.
4. The matmul reuses each input element N times across the output (high reuse → high intensity), while the matrix-vector product reads each weight once and multiplies it once (no reuse → ~1 FLOP/byte).
5. Because it is bottlenecked on how fast bytes arrive from HBM, not on FLOP/s; faster compute units just idle longer waiting for memory. You must move fewer bytes or reuse them more.