Autoscaling & Cold Starts

Multi-GPU & multi-node serving sized a single deployment. This last page asks how many of those deployments — replicas — to run as traffic rises and falls. A web server scales out in milliseconds because a stateless process boots instantly. A GPU replica cannot, because before it can serve a token it must drag tens or hundreds of gigabytes of weights into HBM. That asymmetry — bursty demand against slow-to-start, costly supply — is the whole problem, and it is squarely a cost-versus-latency problem.

Why you autoscale at all

A single GPU replica costs dollars per hour whether or not it serves traffic. Real traffic is bursty: quiet at 4am, a spike when your biggest customer’s app goes viral, a wave each business morning. Provision for the peak and you pay for idle GPUs most of the day; provision for the average and you drop requests at the peak. Autoscaling matches replica count to load — scale out under pressure, scale in when it passes — so you pay for roughly the capacity you use.

requests/sec  ▁▁▂▅█▇▅▂▁     replicas should track the load…
replicas      1 1 2 3 4 3 2 1 …  …but each new replica is NOT instant

The cold-start problem

That “scale out” arrow hides the catch. Spinning up a replica is not launching a process — it is loading the model into HBM, and HBM fills at a finite bandwidth from wherever the weights live (local SSD, network storage, object store).

cold start ≈  schedule GPU  +  pull weights to host  +  load weights → HBM  +  warm kernels

The load step dominates, and it is just bytes over a pipe:

load time ≈ model bytes ÷ load bandwidth

70B FP16 = 140 GB:
  from local NVMe ~5 GB/s :  140 / 5  = 28 s
  from network storage ~1.5 GB/s :  140 / 1.5 ≈ 93 s
plus container start, CUDA init, and kernel warmup → often 1-3 minutes total

Tens of seconds to minutes. A web autoscaler reacts in the time a cold GPU replica spends still loading weights. By the time your new replica is ready, the burst it was meant to absorb may be over — and the users who hit the queue during those 90 seconds already timed out.

Mitigations

You cannot make 140 GB cross a pipe instantly, so every mitigation either hides the latency or shrinks the bytes:

• Min-replicas (a warm floor): never scale to zero on a latency-critical service; keep N replicas always loaded so the common case never pays a cold start. Costs idle GPU-hours, buys instant response.
• Warm pools / pre-provisioned spares: keep a few replicas loaded-but-idle ahead of predicted demand, so “scale out” is just routing traffic, not loading weights.
• Model caching: keep weights on fast local NVMe (or in host RAM) so a restart reads at 5 GB/s, not 1.5 GB/s over the network — the 28 s path, not the 93 s one.
• Smaller / quantized models (Part 4): INT8 halves the bytes, so the cold start halves too. A 70 GB model loads in ~14 s where 140 GB took ~28 s — quantization buys faster scaling, not just cheaper steady state.
• Faster loaders: streaming/parallel weight loaders and memory-mapped formats overlap pull and load to approach raw bandwidth.

Scale-to-zero: the trade

The tempting extreme is scale-to-zero: when a model gets no traffic, kill every replica and pay nothing. For a rarely used internal tool or a long-tail model, that is the right call — the occasional user waits a 90 s cold start, but you pay $0 the other 23 hours. For a user-facing product with a TTFT SLO, scale-to-zero is a non-starter: you cannot make a customer wait 90 seconds for the first token. The decision is purely idle cost vs first-request latency, and it differs per workload.

Policy	Idle cost	First-request latency	Fits
Scale-to-zero	$0	full cold start (tens of s–min)	rare/internal/long-tail
Min-replicas ≥ 1	always pay floor	instant (warm)	latency-critical product
Warm pool	pay for spares	instant on scale-out	predictable bursty traffic

Autoscaling signals

Scaling on raw request count is too crude. Better signals reflect actual pressure:

• Queue depth — the load signal from the scheduling page. A growing queue means service is falling behind; scale out before latency breaches.
• GPU utilization — sustained high util means little headroom; but careful, a memory-bound decode workload can be “busy” at modest util, so pair it with the next signal.
• TTFT / TPOT — scale on the user-visible latency itself. If p95 TTFT is creeping toward the SLO, add capacity regardless of what utilization says.

Autoscale on the symptom, not the cause

Queue depth and tail latency are what users feel; GPU utilization is an internal proxy that can mislead on memory-bound decode. Trigger scale-out on the symptom (rising queue, rising p95 TTFT) and you defend the SLO directly — and because each new replica takes ~30–90 s to warm, scale out early, on the leading edge of a burst, not after the queue has already blown the SLO.

Worked example: when is scale-to-zero worth it?

A 70B INT8 replica (70 GB) costs $2/hour and cold-starts in ~30 s. It serves an internal tool used 20 times a day, a few seconds each.

Always-on:        24 h × $2 = $48/day, ~0 s wait
Scale-to-zero:    ~0 active GPU-hours → ≈ $0/day,
                  but each of 20 users waits ~30 s for cold start

Trade $48/day for 20 users occasionally waiting 30 seconds. For an internal tool, obvious win — scale to zero. Re-price it as a paying product where 30 s of dead air loses the customer, and the same $48/day is trivially worth paying. Identical infrastructure, opposite decision — set entirely by how much that first-token latency is worth to this workload.

Under the hood — the seconds after the weights land

“Warm kernels” in the cold-start formula hides real work that happens after the bytes reach HBM. The first CUDA call on a fresh process builds a CUDA context and loads driver modules; the first matmul of each shape makes cuBLAS/cuDNN select (and sometimes autotune) a kernel; compiled engines like TensorRT-LLM may deserialize or JIT their fused kernels; and engines that use CUDA graphs must capture the decode step once before they can replay it cheaply. None of this touches the network, so a faster pipe can’t hide it — which is why a replica that has already served one request is meaningfully faster on its second, and why a good warm pool keeps replicas that have already run a token, not merely ones that finished loading weights.

The architect’s lens

Autoscaling is the replica-count lever, and the cold start is the catch that makes it unlike web scaling. The five questions:

• Why does it exist? Because a GPU replica costs dollars per hour whether or not it serves traffic, and real traffic is bursty — autoscaling tracks replica count to load so you pay for roughly the capacity you use, not the peak.
• What problem does it solve? The idle-cost-vs-dropped-requests dilemma of static provisioning — but it must work around the cold start: loading a 140 GB model into HBM takes 28 s (local NVMe) to 93 s (network), plus CUDA init and kernel warmup, often 1–3 minutes total.
• What are the trade-offs? Idle cost vs first-request latency — min-replicas and warm pools burn idle GPU-hours to keep responses instant, while every mitigation either hides the latency (warm pools, NVMe caching) or shrinks the bytes (INT8 halves a 140 GB load to ~14 s).
• When should I avoid it? Scale-to-zero is a non-starter for a latency-SLO product (90 s of dead air loses the customer) though ideal for a rare internal tool; and scale on the symptom — queue depth, p95 TTFT — not raw GPU utilization, which misreads memory-bound decode as “busy.”
• What breaks if I remove it? You’re back to static capacity: provision for the peak and pay for idle GPUs most of the day, or provision for the average and drop requests every time a burst arrives.

Check your understanding

1. Why can’t a GPU replica scale out as fast as a stateless web server?
2. Write the cold-start load-time formula and estimate it for a 140 GB model loaded from network storage at ~1.5 GB/s.
3. Name three cold-start mitigations and say whether each hides the latency or shrinks the bytes.
4. What exactly is the trade-off in choosing scale-to-zero, and what kind of workload favors it?
5. Why is queue depth or p95 TTFT often a better autoscaling signal than GPU utilization?

Show answers

1. Because before serving a token a replica must load tens to hundreds of GB of weights into HBM at finite bandwidth — tens of seconds to minutes — whereas a stateless web process boots in milliseconds.
2. load time ≈ model bytes ÷ load bandwidth = 140 GB ÷ 1.5 GB/s ≈ 93 s (plus container start, CUDA init, and kernel warmup, often pushing total to 1–3 minutes).
3. Examples: warm pools/min-replicas and faster loaders hide the latency (keep replicas ready or overlap loading); model caching on local NVMe hides it by using a faster pipe; smaller/quantized models shrink the bytes (INT8 halves load time). Any three with correct classification.
4. Idle cost vs first-request latency: scaling to zero pays $0 when idle but makes the next user wait a full cold start; it favors rare, internal, or long-tail workloads with no tight TTFT SLO.
5. Queue depth and p95 TTFT are the user-visible symptoms of falling behind, while GPU utilization is an internal proxy that can read “busy” even with headroom on memory-bound decode; scaling on the symptom defends the SLO directly.