LoRA, QLoRA & PEFT

The memory math showed full fine-tuning carries the whole 16-bytes-per-parameter burden — gradients and optimizer state for every weight. For a 7B model that is 112 GB of state to adapt a model to your support tickets, and you must store a full new copy of the weights per task. That is absurdly expensive for what is usually a small behavioral change. Parameter-Efficient Fine-Tuning (PEFT) asks the obvious question: can we freeze almost everything and train a sliver? The throughline answer is emphatic — yes, and it cuts the bill by orders of magnitude.

The idea: freeze the base, train an add-on

Full fine-tuning updates all parameters. PEFT freezes the pretrained weights — no gradients, no optimizer state for them — and trains a small set of new parameters bolted on. The base never moves, so the expensive 98 GB of gradient-and-optimizer bookkeeping shrinks to cover only the tiny trainable set.

LoRA (Low-Rank Adaptation) is the dominant flavor. For a weight matrix W of shape d × k, instead of learning a full update ΔW (also d × k), LoRA learns it as a product of two skinny matrices:

ΔW = B · A        A is r × k   (down-projection)
                  B is d × r   (up-projection)
                  r ≪ min(d, k)   ← the "rank"

  W_effective = W (frozen) + B·A (trained)

You train only A and B. The rank r — typically 8, 16, or 64 — controls how many parameters that is.

Worked example: trainable params with rank r

Take one 4096 × 4096 attention projection:

Full update ΔW:   4096 × 4096            = 16,777,216 params
LoRA r = 8:       (8 × 4096) + (4096 × 8) = 65,536 params
ratio:            65,536 / 16,777,216    ≈ 0.39 %

A LoRA adapter on this matrix trains 0.39% of the weights it influences. Apply LoRA across a 7B model’s attention projections at r=8 and you train on the order of ~4 million parameters — under 0.1% of 7B. Now run the optimizer math from two pages ago against that number:

Adam state for 4M trainable params:  4e6 × 16 bytes ≈ 64 MB

64 MB of optimizer state instead of 84 GB. The frozen base still has to be resident (14 GB of FP16 weights), but it needs no gradients and no optimizer moments. The whole job collapses onto a single GPU.

Where the savings actually land

LoRA barely changes the forward compute or the resident base weights. What it slashes is the training-only state — gradients, optimizer moments, and the per-task checkpoint you save (megabytes, not gigabytes). That is the expensive part of training, and LoRA makes it nearly free.

Why it works: the update is low-rank

LoRA is not a hack that happens to fit; it rests on an empirical finding. The pretrained weights are high-rank and information-dense, but the update you apply when adapting to a narrow task has low intrinsic rank — the change lives in a small subspace. You are not re-learning language, just steering an already-capable model. So a rank-8 or rank-16 B·A captures most of what a full ΔW would, at a fraction of a percent of the parameters.

QLoRA: 4-bit base + LoRA on top

The base still costs HBM just to sit there — 14 GB for 7B, and a painful ~130 GB for a 65B model in FP16. QLoRA removes that wall by quantizing the frozen base to 4-bit (NF4) and training the LoRA adapter in BF16 on top. The frozen weights are read in 4-bit; only the tiny adapter is full precision.

65B base in FP16:   65e9 × 2   ≈ 130 GB   → multiple GPUs
65B base in 4-bit:  65e9 × 0.5 ≈ 32.5 GB  → fits one 48 GB GPU
  + LoRA adapter optimizer state ≈ tens of MB
  + activations (with checkpointing) ≈ a few GB

That is the famous QLoRA result: fine-tune a 65B model on a single 48 GB GPU, where full fine-tuning would have demanded a multi-node cluster. The base is read-only and quantized; the trained part is microscopic.

Adapters are swappable — merge or keep separate

Because a LoRA adapter is a separate few-megabyte file, one frozen base can serve many tasks by swapping adapters. Two ways to deploy:

• Merge — fold the adapter into the base (W' = W + B·A) once. Zero inference overhead, indistinguishable from a fully fine-tuned model — but you lose swappability and are back to one full model per task.
• Keep separate — load the base once and hot-swap adapters per request. Slightly more inference work, but one resident base can host dozens of fine-tunes. This is exactly what makes multi-LoRA serving possible: pay for one model in HBM, serve many specializations.

Cost, summarized

LoRA turns a per-task cost of “a full model’s worth of training state and a full model’s worth of storage” into “a few MB of adapter and tens of MB of optimizer state.” QLoRA then removes the last large fixed cost — the resident base — by quantizing it. Together they are why a single engineer with one rented GPU can fine-tune models that used to need a cluster.

It actually happened — LoRA (2021) and QLoRA (2023)

Both techniques on this page are specific, dated landmarks. LoRA came from Edward Hu and colleagues at Microsoft in 2021; adapting GPT-3-scale models, they reported training on the order of 10,000× fewer parameters than full fine-tuning while matching its quality — the result that turned “fine-tune the whole thing” from default into luxury. QLoRA followed in 2023 from Tim Dettmers and colleagues, contributing the 4-bit NF4 format and demonstrating the headline result this page describes: fine-tuning a 65B model on a single 48 GB GPU, which produced the open Guanaco models at a tiny fraction of the usual hardware. The arc is clean: LoRA shrank the trainable state to almost nothing, and QLoRA then shrank the last big fixed cost — the resident base — so a single rented GPU could do what a cluster used to.

The architect’s lens

Weigh PEFT as the difference between renting one GPU and renting a cluster:

• Why does it exist? Full fine-tuning carries ~16 bytes/param of gradient-and-optimizer state — 112 GB for a 7B model — to make what is usually a small behavioral change; LoRA freezes the base and trains a low-rank B·A add-on instead.
• What problem does it solve? The training-only memory and per-task storage cost: a rank-8 adapter trains ~0.39% of a matrix (~4M params for a 7B model), so Adam state drops from 84 GB to ~64 MB and each task is a few-MB file, not a full checkpoint.
• What are the trade-offs? It barely touches forward compute or the resident base, so the base still costs HBM just to sit there — which is exactly what QLoRA removes by quantizing the frozen base to 4-bit (NF4), fitting a 65B fine-tune on a single 48 GB GPU.
• When should I avoid it? When a single dedicated variant gets heavy, steady traffic — then merge the adapter into the base for zero inference overhead; keep adapters separate only to hot-swap many tasks (multi-LoRA serving).
• What breaks if I remove it? You’re back to full fine-tuning: a full model’s worth of training state and a full checkpoint per task — which is why this used to demand a multi-node cluster.

Check your understanding

1. What does PEFT freeze, and which expensive part of training does that eliminate?
2. For a 4096 × 4096 matrix, how many parameters does a LoRA adapter at r = 8 train, and what fraction is that?
3. Why does low-rank adaptation work — what is special about the update versus the original weights?
4. What does QLoRA change relative to plain LoRA, and what does it make possible for a 65B model?
5. When would you merge a LoRA adapter into the base, and when would you keep it separate?

Show answers

1. It freezes the pretrained base weights (no gradients, no optimizer state for them), eliminating the large gradient-and-optimizer-state memory that full fine-tuning needs for every parameter.
2. (8 × 4096) + (4096 × 8) = 65,536 params, about 0.39% of the matrix’s 16.78M parameters.
3. The pretrained weights are high-rank and dense, but the adaptation update has low intrinsic rank — the change lives in a small subspace — so a skinny B·A product captures most of it.
4. QLoRA quantizes the frozen base to 4-bit (NF4) while training the LoRA adapter in higher precision, cutting the resident-base memory roughly 4×; it lets you fine-tune a 65B model on a single ~48 GB GPU instead of a multi-node cluster.
5. Merge when you want zero inference overhead for a single dedicated model; keep separate when you want one resident base to host many swappable task-specific adapters (multi-LoRA serving).