Multi-LoRA Serving

LoRA and PEFT showed that fine-tuning need not touch the base weights: you train small low-rank adapter matrices and leave the giant backbone frozen. That page was about training cheaply. This page is about the serving payoff that property unlocks — hosting a hundred “different models” for nearly the cost of one.

The problem: N models, N copies

Suppose you offer per-customer fine-tunes: a legal variant, a medical variant, a support-bot variant — 200 of them. The naive deployment is 200 full models. For a 13B model in FP16 that is:

200 × 13B × 2 bytes ≈ 200 × 26 GB ≈ 5.2 TB of VRAM

That is dozens of GPUs sitting mostly idle, because each variant gets a trickle of its own traffic. You are paying for 200 backbones that are byte-for-byte almost identical — LoRA changed less than 1% of the parameters.

The insight: one base, many adapters

A LoRA adapter is tiny. For rank r over a weight of shape d×d, it adds two matrices A (d×r) and B (r×d):

adapter params per layer ≈ 2 × d × r

Worked example. A 13B model, applying rank-16 LoRA to attention projections, lands around 10–50 MB per adapter versus 26 GB for the base. So:

Deployment	Memory
200 full models	~5,200 GB
1 base + 200 adapters (~30 MB each)	~26 GB + ~6 GB ≈ 32 GB

Same 200 variants, ~160× less memory — they fit on one GPU. The base is loaded once; adapters are swapped in per request. Systems like S-LoRA and Punica productionize exactly this.

        ┌──────────── shared base model (26 GB) ────────────┐
        │   W (frozen)                                       │
        └───────────────────────────────────────────────────┘
                 ▲          ▲          ▲          ▲
   request A ── +A_legal   +A_medical  +A_support  +A_cust42
   request B ──────────────┘   (each adapter ≈ 30 MB, pooled in VRAM)

Serving it: batch across adapters

The hard part is the batch. Continuous batching packs many requests into one weight-load — but here each request wants a different adapter. Run them one adapter at a time and you have destroyed batching: back to memory-bound, low-utilization decode.

The trick is to compute the shared base matmul once for the whole batch, then apply each request’s own small LoRA correction. The base path x·W is identical for everyone; only the x·A·B correction differs per request, and that correction is cheap (rank r is small).

  batched output = x · W      (one big shared matmul, all requests)
                 + x · Aᵢ · Bᵢ (per-request, batched grouped-GEMM)

S-LoRA and Punica supply custom CUDA kernels (a “grouped” or “segmented” GEMM) that apply many different adapters within a single batched call. That kernel is the whole ballgame: it preserves the batching win of the shared base while still giving each request its personalized weights.

Why this works at all

The base matmul is the expensive, memory-bound part — and it’s shared, so one weight-load serves the whole mixed batch. The per-adapter correction is low-rank and tiny. You get N models’ behavior at ~1 model’s bandwidth cost.

The costs

• Adapter swap / paging. Hundreds of adapters may not all fit in VRAM; hot ones live on the GPU, cold ones page in from host memory on demand. A request for a cold adapter eats a transfer (tens of MB over PCIe — sub-millisecond to a few ms) before it can run. S-LoRA manages this with a unified paged memory pool, the same philosophy as paged attention.
• Kernel support. Plain frameworks merge LoRA into the base weights for a single adapter and serve it like a normal model — fine for one, useless for hundreds. Multi-LoRA requires the specialized batched kernels above; without them you cannot batch across adapters.
• Slight per-token overhead. The extra x·A·B term adds a little compute and a little latency versus a single merged model. In exchange you collapse 200 backbones into one.

Merge vs serve-separately

If a variant gets heavy, steady traffic, merge its LoRA into a dedicated base copy and serve it normally — no per-token adapter overhead. Multi-LoRA shines for the long tail: many variants, each with light, bursty traffic, that could never justify a whole GPU alone.

It actually happened — S-LoRA served thousands of adapters on one GPU (2023)

The “many adapters, one base” idea was demonstrated at a startling scale in S-LoRA: Serving Thousands of Concurrent LoRA Adapters (Sheng et al., 2023). The system held thousands of adapters in a unified paged memory pool — cold ones in host RAM, hot ones paged into VRAM, the same trick paged attention plays with the KV cache — and applied them with a custom batched kernel so a single mixed batch could carry hundreds of different adapters at once. Punica’s companion contribution was the SGMV kernel (Segmented Gather Matrix-Vector multiply), which does that grouped application in one GPU launch instead of looping per adapter. The capability has since landed in mainstream stacks — vLLM serves multiple LoRA adapters over a shared base out of the box — which is why “a fine-tuned model per customer” went from a research demo to a config flag.

The cost lens

Multi-LoRA is a pure memory-and-dollars play. You are exploiting the fact that fine-tuning changed almost nothing: instead of paying for N redundant backbones, you pay for one backbone plus N rounding-error adapters, and a clever kernel keeps batching alive across them. That is how a startup serves a “custom model per customer” without a custom GPU per customer.

The architect’s lens

The serving payoff of LoRA’s “the base barely moved” property:

• Why does it exist? Because per-customer fine-tunes are byte-for-byte almost identical (LoRA changed <1% of params), so 200 full 13B models (~5.2 TB VRAM, mostly idle) is paying 200× for one backbone.
• What problem does it solve? Hosting the long tail of variants for ~one model’s memory: one shared base + 200 ~30 MB adapters fits in ~32 GB (≈160× less) on a single GPU.
• What are the trade-offs? It requires specialized batched kernels (S-LoRA, Punica’s SGMV) to compute the shared x·W once and apply each request’s low-rank x·A·B correction in one launch — plus adapter paging for cold adapters and slight per-token overhead.
• When should I avoid it? When a variant gets heavy, steady traffic — then merge its LoRA into a dedicated base copy and serve it normally, with no per-token adapter overhead.
• What breaks if I remove it? Each request’s different adapter forces one-adapter-at-a-time execution, destroying the shared weight-load and reverting to low-utilization, memory-bound decode — or you pay for N redundant full backbones.

Check your understanding

1. Why is hosting 200 fully fine-tuned 13B models so wasteful in memory?
2. Roughly how much smaller is a rank-16 LoRA adapter than a 13B base, and what does that imply for how many variants fit on one GPU?
3. What would naively break batching when each request uses a different adapter, and how do S-LoRA/Punica preserve it?
4. What is “adapter paging” and what does it cost on a cold-adapter request?
5. When should you merge a LoRA into a dedicated base copy instead of serving it multi-LoRA style?

Show answers

1. The 200 backbones are nearly identical (LoRA changed <1% of params) yet each full copy costs ~26 GB, so you pay ~5.2 TB of VRAM for 200 mostly-redundant models, most of them idle.
2. An adapter is ~10–50 MB vs ~26 GB for the base — on the order of 1000× smaller — so one base plus hundreds of adapters fits in ~32 GB, i.e. on a single GPU.
3. Each request wanting a different adapter would force one-adapter-at-a-time execution, destroying the shared weight-load. The systems compute the shared base matmul once for the batch and apply each request’s low-rank correction via a grouped/segmented GEMM kernel, keeping the batch intact.
4. Hot adapters stay in VRAM, cold ones page in from host memory on demand; a cold request pays a transfer (tens of MB over PCIe, ~ms) before it can run.
5. When that variant gets heavy, steady traffic — merging removes the per-token adapter overhead. Multi-LoRA is for the long tail of many lightly-used variants.