Numeric Precision: FP32 to INT4

Why LLM Inference Is Memory-Bound gave us a lever with a name: per-token latency is roughly bytes_read / bandwidth, so the way to go faster on a memory-bound workload is to read fewer bytes. The most direct way to do that is to make each number smaller. This page is about how few bits a weight can use before the model gets dumb — and why halving the bits roughly halves both your memory footprint and your latency.

Anatomy of a float

A floating-point number is scientific notation in binary. Three fields:

  ┌─────┬──────────────┬───────────────────────────┐
  │sign │   exponent   │          mantissa          │
  └─────┴──────────────┴───────────────────────────┘
   ±      how BIG          the significant digits
          (dynamic range)  (precision)
   value ≈ ±1.mantissa × 2^(exponent)

Two knobs trade off against each other given a fixed bit budget:

• Exponent bits → dynamic range. How big and how tiny a number you can represent before it overflows to infinity or underflows to zero.
• Mantissa bits → precision. How finely you can distinguish nearby values.

The whole story of ML number formats is how to split a shrinking bit budget between range and precision.

The formats that matter

Format	Bits	Bytes/param	Exponent	Mantissa	Headline property
FP32	32	4	8	23	The reference. Full range and precision.
TF32	19*	4 (storage)	8	10	FP32 range, fewer mantissa bits; faster matmuls on tensor cores.
FP16	16	2	5	10	Half the bytes, but small exponent → can overflow/underflow in training.
BF16	16	2	8	7	Keeps FP32’s range, sacrifices precision. Won for ML.
FP8	8	1	4 or 5	3 or 2	Two variants (E4M3, E5M2) trading range vs precision.
INT8	8	1	—	—	Integers + a scale factor; 256 levels. Workhorse for inference.
INT4	4	0.5	—	—	16 levels per group; aggressive weight-only quantization.

*TF32 uses 19 meaningful bits internally but is still stored/handled in a 4-byte slot.

Why BF16 beat FP16 for ML

FP16 and BF16 are both 16 bits, but they split those bits differently. FP16 keeps 10 mantissa bits and only 5 exponent bits — precise, but its range is so narrow that gradients during training routinely overflow to infinity or vanish to zero, forcing fiddly “loss scaling.” BF16 throws away mantissa bits (only 7) to keep all 8 of FP32’s exponent bits, so it has the same dynamic range as FP32. Models tolerate low precision far better than they tolerate values silently becoming inf/0, so BF16 made mixed-precision training “just work.” Range beat precision.

Why fewer bytes is faster, not just smaller

This is the payoff and the throughline. On a memory-bound workload, the cost is bytes moved. Precision sets bytes-per-param directly. So:

time/token ≈ weight_bytes / bandwidth
           = (params × bytes_per_param) / bandwidth

Halve bytes_per_param and you halve weight_bytes, which halves the latency floor and halves the HBM footprint at the same time. One change, two wins. This is why quantization is the first optimization any inference team reaches for: going FP16 → INT8 doesn’t just let a bigger model fit, it makes the same model emit tokens roughly twice as fast, because decode was waiting on memory the whole time.

Compute-bound workloads benefit too — for a different reason

On the compute-bound side (prefill, training), lower precision also helps, but via FLOP/s: tensor cores run 8-bit math at far higher throughput than 32-bit. So precision is a lever on both roofline axes — fewer bytes when memory-bound, more FLOP/s when compute-bound.

The accuracy cost

You don’t get this for free. Fewer bits means coarser values, and at some point the model’s quality degrades:

• FP16/BF16: essentially lossless for inference; the modern default.
• FP8 / INT8: typically a very small quality hit (often <1% on benchmarks) with good calibration. Widely deployed.
• INT4: noticeable but often acceptable degradation, if you quantize cleverly — per-group scales, keeping sensitive layers (or outlier channels) in higher precision. This is where quantization stops being a free lunch and becomes engineering.

The trend line is clear: each halving of precision roughly halves memory and bandwidth cost, with an accuracy penalty that starts negligible and grows as you push past 8 bits. The art is finding the lowest precision your task tolerates.

Worked example: a 70B model at each precision

Memory for weights = params × bytes_per_param. For 70 × 10⁹ params:

FP32 (4 bytes):   70e9 × 4   = 280 GB
BF16 (2 bytes):   70e9 × 2   = 140 GB
FP8  (1 byte):    70e9 × 1   =  70 GB
INT8 (1 byte):    70e9 × 1   =  70 GB
INT4 (0.5 byte):  70e9 × 0.5 =  35 GB

Precision	Bytes/param	70B weights	Fits on one 80 GB GPU?
FP32	4	280 GB	No (needs ~4 GPUs)
BF16/FP16	2	140 GB	No (needs 2)
FP8/INT8	1	70 GB	Yes (just barely)
INT4	0.5	35 GB	Yes, with room

Read that table as a cost story. In FP32 the model needs four GPUs and four GPUs’ worth of bandwidth bills; in INT8 it drops onto a single 80 GB card, eliminating the cross-GPU interconnect entirely and roughly halving time-per-token versus BF16. Precision is the cheapest, highest-leverage knob you have.

It actually happened — FP8 lands in silicon (2022)

For years 16 bits (FP16, then BF16) was the practical floor for the math units themselves. That changed with NVIDIA’s Hopper architecture: the H100, unveiled in 2022, shipped a “Transformer Engine” with native FP8 support in its tensor cores, in the two variants this page describes — E4M3 (more precision) and E5M2 (more range). That September, NVIDIA, Arm, and Intel jointly published an FP8 format specification, a rare cross-vendor agreement meant to keep 8-bit floating point portable across hardware. The significance: FP8 wasn’t a software emulation trick but a hardware format, so a chip could now both store weights in one byte and multiply them at roughly double the FP16 rate. It is the same pattern as BF16, which began as Google Brain’s “brain float” for its TPUs — each step down the precision ladder eventually earns its own dedicated silicon once the accuracy turns out to be tolerable.

The architect’s lens

Precision is the highest-leverage knob in the book — run it through the five questions:

• Why does it exist? Because decode is memory-bound and time/token ≈ weight_bytes / bandwidth, and precision sets bytes_per_param directly — so FP16 → INT8 → INT4 is the most direct way to read fewer bytes per token.
• What problem does it solve? It cuts memory footprint and latency in one move: halving bytes-per-param halves weight memory and the latency floor together, and can pull a model onto fewer (or one) GPU — a 70B drops from 140 GB to 35 GB at INT4.
• What are the trade-offs? Accuracy, falling as you descend: FP16/BF16 is effectively lossless, FP8/INT8 costs <1% with good calibration, but INT4 brings noticeable degradation unless you quantize cleverly — per-group scales and keeping sensitive/outlier channels higher-precision.
• When should I avoid it? When the task can’t absorb the quality hit — past 8 bits it stops being a free lunch, so don’t blanket-quantize sensitive layers to INT4 on accuracy-critical work without mixed-precision care.
• What breaks if I remove it? Stuck at FP16/FP32, a 70B model needs 140–280 GB (2–4 GPUs), every decode step streams 2–4× the bytes — so 2–4× higher per-token latency — and you pay the cross-GPU interconnect tax you could have quantized away.

Check your understanding

1. What do the exponent and mantissa fields each control in a float?
2. FP16 and BF16 are both 16 bits — why did BF16 win for ML?
3. On a memory-bound workload, why does halving the precision roughly halve the per-token latency, not just the memory footprint?
4. Lower precision helps compute-bound workloads too, but through a different mechanism — what is it?
5. Compute the weight memory for a 70B model in INT4, and say whether it fits on one 80 GB GPU.

Show answers

1. The exponent controls dynamic range (how big or small a number can be); the mantissa controls precision (how finely nearby values are distinguished).
2. They split their 16 bits differently: FP16 keeps more mantissa but only 5 exponent bits, giving a narrow range that overflows/underflows during training. BF16 keeps all 8 of FP32’s exponent bits (same range as FP32), trading away mantissa precision — and models tolerate low precision better than values becoming inf/0.
3. Because per-token latency ≈ weight_bytes / bandwidth, and weight_bytes = params × bytes_per_param. Halving bytes_per_param halves weight_bytes and therefore the latency floor.
4. Through FLOP/s: tensor cores execute lower-precision math at much higher throughput, so prefill/training (compute-bound) run faster.
5. 70 × 10⁹ × 0.5 byte = 35 GB, which fits comfortably on one 80 GB GPU with room for activations and KV cache.