GPUs From First Principles

The overview promised that efficiency is the physics of moving bytes and doing FLOPs. To make that concrete we need to know what the machine actually looks like inside. This page builds the GPU from first principles: why it is shaped the way it is, and how data flows through it.

Two philosophies: latency vs throughput

A CPU and a GPU are both made of transistors, but they spend them on opposite goals.

A CPU optimizes for latency: finish one task as fast as possible. So it spends its transistor budget on a handful of very sophisticated cores — big caches, branch predictors, out-of-order execution, deep speculation — to make a single thread of instructions fly. A modern CPU might have 8–64 such cores.

A GPU optimizes for throughput: do an enormous amount of work per second, even if any one item is slow. So it spends its transistors on thousands of small, simple cores. It has almost no fancy per-core machinery; instead it hides latency by having so many threads in flight that whenever one stalls waiting for memory, another is ready to run.

   CPU                          GPU
   ┌──────┐ ┌──────┐            ┌─┬─┬─┬─┬─┬─┬─┬─┬─┬─┐
   │ FAT  │ │ FAT  │            ├─┼─┼─┼─┼─┼─┼─┼─┼─┼─┤  thousands of
   │ core │ │ core │            ├─┼─┼─┼─┼─┼─┼─┼─┼─┼─┤  tiny cores
   └──────┘ └──────┘            ├─┼─┼─┼─┼─┼─┼─┼─┼─┼─┤
   ┌──────┐ ┌──────┐            ├─┼─┼─┼─┼─┼─┼─┼─┼─┼─┤
   │ FAT  │ │ FAT  │            └─┴─┴─┴─┴─┴─┴─┴─┴─┴─┘
   └──────┘ └──────┘            built to keep all of them busy
   few cores, deep tricks

This is why GPUs are perfect for neural networks: a matrix multiply is the same simple operation repeated across millions of independent elements. That is throughput work, not latency work.

SMs, threads, and warps

The cores are not a flat pool. They are grouped into Streaming Multiprocessors (SMs) — an H100 has ~130+ of them. Each SM is a small factory with its own scheduler, registers, and a slab of fast on-chip memory.

When you launch work, you launch thousands of threads. The hardware groups threads into warps of 32. The crucial rule: all 32 threads in a warp execute the same instruction at the same time, each on its own data. This is “SIMT” (single instruction, multiple threads).

Why warps matter for cost

Because a warp runs in lockstep, an if/else that sends some threads one way and some the other forces the warp to run both branches, masking off the inactive threads. This “warp divergence” wastes compute. It is one reason ML kernels prefer dense, branch-free arithmetic — predictable work keeps every lane busy.

The memory hierarchy: the part that actually governs speed

Compute is cheap and plentiful. The hard part is feeding those cores. Memory on a GPU is a steep pyramid: tiny and blindingly fast at the top, huge and comparatively slow at the bottom.

        ┌───────────────────────────┐
        │ REGISTERS                  │  ~KB/thread   fastest, per-thread
        ├───────────────────────────┤
        │ SHARED MEMORY / L1 (SRAM)  │  ~100s KB/SM  on-chip, per-SM
        ├───────────────────────────┤
        │ L2 CACHE                   │  ~tens of MB  shared by all SMs
        ├───────────────────────────┤
        │ HBM  (global / "VRAM")     │  ~80 GB       on-package, ~TB/s
        ├───────────────────────────┤
        │ HOST RAM  (over PCIe)      │  ~100s GB     off-chip, ~tens GB/s
        └───────────────────────────┘
              slower and bigger as you go down

Two facts about this pyramid drive everything in later pages:

• Each step down is roughly an order of magnitude slower. A register access is essentially free; an HBM access costs hundreds of cycles; a trip to host RAM over PCIe is glacial by comparison.
• HBM is where your model weights live. “HBM” (High Bandwidth Memory) is the 80 GB you see advertised. It is fast for off-chip memory — but it is still the slowest thing the cores touch on every step, which is exactly why it becomes the bottleneck.

Approximate HBM specs for two common datacenter GPUs:

GPU	HBM capacity	HBM bandwidth
A100	80 GB	~2.0 TB/s
H100	80 GB	~3.35 TB/s

Bandwidth is the number to tattoo on your arm: it caps how many bytes per second you can pull from where your weights live.

Worked example: how many FP16 params fit in 80 GB?

A parameter stored in FP16 (half precision) takes 2 bytes. So:

params that fit = capacity / bytes_per_param
               = 80 GB / 2 bytes
               = 80 × 10^9 bytes / 2 bytes
               = 40 × 10^9 params
               = 40 billion parameters

So a single 80 GB GPU holds roughly a 40B-parameter model in FP16 — and that is before leaving room for activations, the KV cache, and CUDA overhead, which in practice might cost you 10–20 GB. That is the first hard wall in the book: a 70B model in FP16 needs ~140 GB and simply does not fit on one 80 GB card. We will spend a whole page on what to do about it, and another on shrinking the bytes-per-param.

The cost lens

Notice we computed a memory limit using only capacity and bytes-per-param — no model architecture required. Most “will it fit / how fast” questions reduce to arithmetic this simple once you know the hardware numbers. That is the whole point of starting here.

Under the hood — tensor cores, the matmul-only engines

The “thousands of small cores” above are the general-purpose CUDA cores that handle ordinary arithmetic. But almost all of an LLM’s work is one operation — matrix multiply — and since Volta (the V100, 2017) NVIDIA GPUs carry a second, specialized unit built for exactly that: the tensor core. Where a CUDA core multiplies two numbers, a tensor core performs a whole small matrix multiply-accumulate (on the order of a 4×4 by 4×4 tile) in a single operation, and there are many per SM.

That is where the headline “TFLOP/s” numbers come from. The bulk of a modern GPU’s advertised throughput is tensor-core throughput, available only to matmul-shaped work — which is why ML kernels bend over backwards to express everything as dense matrix multiplies. Each generation also widened the formats the tensor cores accept: Volta did FP16, Turing (2018) added INT8, Ampere (A100) added TF32 and BF16, and Hopper (H100) added FP8. Lower-precision formats run through the tensor cores at higher rates, which is the hardware reason the numeric-precision knob buys speed and not just memory.

The catch returns us to the roofline. All that tensor-core compute only helps if you can feed it: in memory-bound decode the tensor cores sit mostly idle, waiting on HBM — the expensive matmul engines starving for exactly the reason the next pages explore.

The architect’s lens

Step back from the transistors and run the five questions on the choice of GPUs as your inference substrate:

• Why does it exist? A CPU spends its transistor budget on a few latency-optimized cores; a GPU spends the same budget on thousands of simple cores plus matmul-only tensor cores, because neural-network work is one simple operation across millions of independent elements — throughput work, not latency work.
• What problem does it solve? Feeding enough parallel arithmetic to make a billion-parameter forward pass tractable, and hiding memory stalls behind thousands of in-flight threads so the cores stay busy.
• What are the trade-offs? You give up single-thread speed and branch flexibility for raw throughput — an if/else inside a warp runs both sides (warp divergence) — and the steep memory pyramid (registers → SRAM → L2 → HBM) means the cores are only as fast as you can feed them from the ~80 GB at ~2–3.35 TB/s.
• When should I avoid it? Branchy, serial, or tiny workloads with little arithmetic per byte — there a CPU’s fat cores win and the tensor cores sit idle; a small model that isn’t latency-critical may not justify hardware costing 10–50× a normal server.
• What breaks if I remove it? Inference falls back to a handful of CPU cores: the dense matmuls (~2 × params FLOPs/token) lose their massively parallel engine and per-token latency rises by orders of magnitude — the memory-bound decode loop loses both the bandwidth and the parallelism it leans on.

Check your understanding

1. In one sentence each, contrast what a CPU and a GPU optimize for.
2. What is a warp, and why does an if/else inside one waste compute?
3. List the memory hierarchy from fastest to slowest and note roughly how speed changes per step.
4. Why is HBM bandwidth “the number to tattoo on your arm”?
5. Recompute: how many parameters fit in an 80 GB GPU if each param is stored in 1 byte (INT8) instead of 2?
6. What is a tensor core, and why does most of a GPU’s advertised TFLOP/s apply only to matrix-multiply work?

Show answers

1. A CPU optimizes for latency (finish one task as fast as possible with a few sophisticated cores); a GPU optimizes for throughput (do massive total work using thousands of simple cores and hide stalls with many threads).
2. A warp is a group of 32 threads that execute the same instruction in lockstep, each on its own data. An if/else makes the warp run both branches with inactive lanes masked off, wasting the masked compute (warp divergence).
3. Registers → shared memory/L1 (SRAM) → L2 cache → HBM (global) → host RAM over PCIe; each step down is roughly an order of magnitude slower (and larger).
4. Because it caps how many bytes per second you can read from where the weights live, and reading weights is the bottleneck for memory-bound workloads like inference.
5. 80 GB / 1 byte = 80 × 10^9 = 80 billion parameters (before overhead).
6. A tensor core is a specialized unit (introduced with Volta/V100) that performs a small matrix multiply-accumulate per operation, rather than a single scalar multiply like a CUDA core. The bulk of a GPU’s headline throughput comes from these tensor cores, which accelerate only matmul-shaped work — so kernels express computation as dense matrix multiplies to reach it.