The Quantization Zoo: GPTQ, AWQ, GGUF

The fundamentals page covered the mapping and the outlier problem. This page is the practical sequel: the named methods and file formats you’ll actually meet. A note up front — these almost all apply to open-weight models you run yourself (Llama, Mistral, Qwen, etc.). With a closed API model you don’t touch the weights, so quantization is the provider’s problem, not yours.

The contenders

GPTQ — layer-wise, second-order

GPTQ quantizes one layer at a time and uses second-order information (an approximation of the Hessian) to decide how to round each weight so the layer’s output error stays small. Intuitively: it doesn’t just round each weight to the nearest grid point; it nudges the still-unquantized weights to compensate for the error already introduced. GPU-oriented, excellent at INT4, a one-time offline pass. The reference format for 4-bit GPU serving.

AWQ — activation-aware, protect the salient weights

AWQ’s insight: not all weights matter equally — the ones multiplied by large activation channels are salient. Instead of keeping them in higher precision (awkward for hardware), AWQ scales those weight channels up before quantizing (and scales the matching activations down to compensate), so the important weights land on a finer part of the grid. Activation-aware, very strong at INT4, fast kernels. Often ties or beats GPTQ on accuracy with simpler calibration.

GGUF / llama.cpp — the local-inference format

GGUF is the file format of the llama.cpp ecosystem — the de-facto way to run models on a laptop, CPU, or Apple Silicon, with optional GPU offload. Its k-quants (Q4_K_M, Q5_K_M, Q6_K, …) are mixed-precision group schemes that spend more bits on the layers that need them. CPU+GPU flexible, single self-contained file, huge community. If you’re running a model on your own machine, this is usually how.

Q4_K_M  ≈ 4.x bits/weight, mixed   → best size/quality for laptops
Q5_K_M  ≈ 5.x bits/weight          → noticeably safer accuracy
Q6_K    ≈ 6.x bits/weight          → near-FP16 quality, larger file
Q8_0    ≈ 8 bits/weight            → almost lossless, biggest

bitsandbytes — on-the-fly, the QLoRA engine

bitsandbytes does quantization at load time with no calibration pass — load FP16 checkpoint, get INT8 (LLM.int8()) or NF4 (4-bit NormalFloat) weights in memory immediately. NF4’s grid is shaped to the bell-curve distribution of neural weights rather than uniform steps. It’s the backbone of QLoRA: freeze the model in NF4, train small LoRA adapters on top — fine-tune a big model on one consumer GPU. Convenient and training-friendly; its inference kernels are typically slower than a dedicated GPTQ/AWQ build.

SmoothQuant — migrate the outliers

SmoothQuant targets the weight+activation INT8 case. It mathematically migrates the activation outlier magnitude into the weights (divide activations by a per-channel factor, multiply weights by it — the product is unchanged), making activations smooth enough to quantize to INT8. This is what unlocks the faster INT8 integer matmuls for high-throughput serving.

Which one when

Method	Bits	Where it runs	Needs calibration?	Best for
GPTQ	INT4/3	GPU	yes (small set)	4-bit GPU serving
AWQ	INT4/3	GPU	yes (small set)	4-bit GPU, top accuracy
GGUF (k-quants)	~2–8	CPU + GPU	no (offline convert)	local / laptop / Mac
bitsandbytes	INT8 / NF4	GPU	no (on-the-fly)	QLoRA fine-tuning, quick loads
SmoothQuant	INT8 W+A	GPU	yes	high-throughput INT8 compute

Decision shortcut

Running on a laptop or CPU → GGUF. Serving INT4 on a GPU → GPTQ or AWQ. Fine-tuning a big model on one GPU → bitsandbytes/QLoRA. Chasing INT8 compute throughput at scale → SmoothQuant.

Weight-only vs weight+activation, again

GPTQ, AWQ, GGUF, and bitsandbytes-NF4 are predominantly weight-only — they save memory and bytes-moved (latency on a memory-bound workload), but the matmul runs in FP16. SmoothQuant (and INT8 paths) quantize activations too, unlocking integer-matmul compute. Match the method to whether your bottleneck is memory or compute.

The dollars view

Concretely, on an open 13B model: FP16 is ~26 GB and needs a 40 GB-class GPU; the same model as Q4_K_M GGUF is ~7–8 GB and runs on a laptop with 16 GB of RAM, or an 8 GB consumer GPU. The quantization step is a one-time, minutes-to-an-hour offline job; the savings — a cheaper GPU (or none), less power, less data moved per token — recur on every single request thereafter. That’s the recurring trade of this whole part: a small upfront cost, paid back forever.

Under the hood — reading a name like Q4_K_M

The cryptic GGUF labels decode cleanly. The number (Q4, Q5, Q6) is the base bits-per-weight. The K marks a k-quant: weights are grouped into small blocks that share a scale, and those block-scales are themselves grouped into super-blocks — a two-level scheme that localizes rounding damage cheaply. The trailing letter is the size/quality mix: _S, _M, _L (small/medium/large) spend progressively more bits on the tensors a model is most sensitive to (often parts of attention and the FFN down-projection), leaving the rest at the base width. So Q4_K_M reads as “4-bit k-quant, medium mix” — the usual sweet spot — while Q4_K_S trades a sliver of quality for a smaller file. The headline ≈4.x bits-per-weight is therefore an average, not a uniform width.

It actually happened — QLoRA fine-tunes a 65B model on one GPU (2023)

QLoRA (Dettmers et al., 2023) stacked three tricks to fine-tune a 65B model on a single 48 GB GPU — work that previously needed a multi-GPU node. First, NF4: a 4-bit data type whose levels are placed to match the bell-curve distribution of neural weights, so no codes are wasted on values that never occur. Second, double quantization: even the per-block scales are quantized, shaving a little more off each weight. Third, paged optimizers that spill optimizer state to CPU RAM on memory spikes. The frozen base sits in NF4 while small LoRA adapters train on top, and the resulting Guanaco models landed within striking distance of far pricier fine-tunes. It is the clearest demonstration that quantization isn’t only a serving trick — it reshapes what hardware you need to train on.

The architect’s lens

The zoo isn’t five rivals — it’s one decision keyed to where you run and what’s the bottleneck:

• Why does it exist? Because naive round-to-nearest INT4 loses too much accuracy, so each method (GPTQ’s second-order rounding, AWQ’s salient-weight scaling, GGUF’s mixed k-quants, NF4, SmoothQuant’s outlier migration) is a different answer to “quantize hard without wrecking quality.”
• What problem does it solve? Matching a recipe to a deployment: GGUF for laptop/CPU, GPTQ or AWQ for INT4 GPU serving, bitsandbytes/NF4 for QLoRA fine-tuning, SmoothQuant for INT8 compute at scale.
• What are the trade-offs? Most (GPTQ, AWQ, GGUF, NF4) are weight-only — they win memory and bytes-moved but the matmul still runs FP16; only SmoothQuant/INT8 paths quantize activations to unlock integer matmuls, and bitsandbytes’ convenience costs inference-kernel speed.
• When should I avoid it? When you use a closed API model — you don’t touch the weights, so quantization is the provider’s problem, not a knob you turn.
• What breaks if I remove it? You ship the FP16 open-weight model: a 13B jumps from ~7–8 GB (Q4_K_M) back to ~26 GB, needing a 40 GB-class GPU instead of a laptop.

Check your understanding

1. What information does GPTQ use that plain round-to-nearest doesn’t, and what is it trying to keep small?
2. AWQ identifies “salient” weights. What makes a weight salient, and what does AWQ do instead of keeping it in higher precision?
3. You want to run a 13B model on a MacBook with no discrete GPU. Which format, and why?
4. Why is bitsandbytes the natural fit for QLoRA fine-tuning?
5. SmoothQuant doesn’t reduce model size much versus other INT8 methods — so what problem is it actually solving?

Show answers

1. Second-order (approximate Hessian) information about how each weight affects the layer output; it minimizes the layer’s output error, compensating by adjusting the still-unquantized weights as it goes.
2. A weight is salient when it’s multiplied by a large-magnitude activation channel. AWQ scales those weight channels up (and the activations down) before quantizing so the important weights land on a finer region of the grid, avoiding mixed-precision hardware awkwardness.
3. GGUF with k-quants via llama.cpp — it’s built for CPU/Apple-Silicon execution, ships as one self-contained file, and Q4_K_M fits a 13B model in ~7–8 GB.
4. It quantizes on the fly to NF4 with no calibration pass, so you can freeze a big model in 4-bit memory and train small LoRA adapters on top on a single consumer GPU.
5. Activation outliers that otherwise block INT8 activation quantization; by migrating outlier magnitude into the weights it makes activations smooth enough to quantize, unlocking fast INT8 integer matmuls for compute-bound serving.