Load Testing & Benchmarking

The metrics page gave you the numbers — TTFT, TPOT, throughput, goodput. This page is about generating them without fooling yourself. A benchmark is an experiment, and like any experiment it is trivially easy to design one that produces a flattering, useless result. The goal here is the opposite: a load test whose numbers you would stake a capacity plan on.

Why honest benchmarking is hard

LLM serving is a closed-loop, stateful system. Throughput depends on batch composition, which depends on prompt and output lengths, which depend on the traffic you feed it. Change the input distribution and every metric moves. So the cost question — how many dollars-per-token does this config really deliver? — has no answer until you pin down what traffic you measured under. Most published numbers omit exactly this, which is why they rarely reproduce.

Input and output shapes are the experiment

The single biggest lever on your results is the distribution of input and output lengths, because it decides how much work is prefill versus decode.

• A benchmark with 32-token prompts and 8-token answers is almost pure prefill — it measures a system that does not exist in production.
• A RAG workload with 4 000-token prompts and 300-token answers is prefill-heavy on input but decode-heavy on output, and it stresses KV memory hard.
• A chat workload with short prompts and 500-token answers is decode-dominated.

Three workloads, same model, wildly different numbers:

short→short   [in 32 | out 8 ]   tiny KV, prefill-bound, flatters tokens/s
long →short   [in 4000| out 50]   huge prefill spike, big TTFT, big KV
short→long    [in 64 | out 800]   decode-bound, batch size is everything

Use realistic shapes. Sample input/output lengths from your actual logs (or a public trace like ShareGPT) rather than fixing them. Fixed-length prompts are the most common way to publish a number that collapses in production.

Concurrency sweeps: find the knee

A single concurrency level tells you almost nothing. The informative experiment is a sweep: hold the workload fixed and raise the number of concurrent clients — 1, 2, 4, 8, 16, 32, 64 … — recording throughput and p99 latency at each step.

            throughput (tok/s)            p99 TTFT (ms)
 conc   ┌───────────────────────┐   ┌───────────────────────┐
   1    │█                      │   │█                       │
   4    │████                   │   │█                       │
  16    │██████████             │   │██                      │
  32    │███████████████        │   │████                    │  ← the knee
  64    │████████████████       │   │██████████████          │  saturated:
 128    │████████████████       │   │████████████████████████│  latency blows up,
        └───────────────────────┘   └───────────────────────┘  throughput flat

Below the knee, adding load buys you throughput almost for free — latency barely moves. Past the knee, the GPU is saturated (KV memory full, queue backing up): throughput flatlines while latency explodes. The knee is your operating point — the maximum concurrency that still respects the SLO, i.e. the load that maximizes goodput. Running past it raises raw throughput on paper while serving everyone too slowly to count.

Steady state and warmup

The first requests after startup are unrepresentative: weights stream into HBM, CUDA graphs and kernels compile/capture, caches are cold, the batch is half-empty. Including them poisons the average.

• Warmup: fire a burst of throwaway requests and discard their metrics before recording.
• Measure at steady state: the batch is full, the queue has reached its equilibrium depth, and throughput has plateaued. Run long enough that a few slow requests do not dominate — a few thousand requests, or several minutes, not ten calls.
• Ramp, then hold: ramp concurrency up to the target, then start the measurement window.

Two-phase warmup

Prefill and decode warm up differently. The very first long prompt may trigger a one-time kernel/graph compilation that adds hundreds of ms — a real cold-start cost worth measuring separately, but not something to smear into your steady-state TTFT.

Tools

You rarely need to write the harness yourself.

• vLLM’s vllm bench serve (and vllm bench throughput) — the CLI that replaced the older benchmark_serving.py script — drives an OpenAI-compatible endpoint, samples real prompt distributions (e.g. ShareGPT), and reports TTFT, TPOT/ITL, and throughput at a chosen request rate. The de-facto standard for serving benchmarks.
• General load generators — locust, k6, or Apache Bench style tools — good for HTTP-level concurrency sweeps, but you must add token-level timing (TTFT/TPOT) yourself, since they only see request-level latency.
• GenAI-Perf / inference-benchmarker style tools — purpose-built for token-streaming metrics and percentiles.

Whatever you use, report the recipe: model, quantization, hardware, input/output distribution, concurrency, and the SLO — otherwise the number is unreproducible.

What to report

A credible result is a small table, not a single hero number:

Concurrency	tok/s (output)	p50 TTFT	p99 TTFT	p50 TPOT	p99 TPOT	Goodput @ SLO
32 (the knee)	4 200	180 ms	420 ms	22 ms	31 ms	31 RPS

From that one row you can compute the cost throughline directly: if this runs on an H100 you rent at ~$2/hour, then 4 200 tok/s × 3 600 s = ~15.1M output tokens/hour, so ~$0.13 per 1M output tokens at the SLO — a number you can take straight to the cost model. A benchmark that cannot produce that line did not measure anything useful.

Pitfall checklist

The four ways benchmarks lie: (1) fixed-length prompts instead of a realistic distribution; (2) reporting throughput while ignoring TTFT; (3) no warmup, so cold-start poisons the numbers; (4) one concurrency level instead of a sweep, so you never find the knee. Avoid all four and your numbers will reproduce in production.

Under the hood — open-loop vs closed-loop, and coordinated omission

How you generate load quietly changes the answer. A closed-loop generator holds a fixed number of clients, each firing its next request only after the previous response returns. When the server slows, the load generator politely slows with it — so the offered rate drops exactly when you most need to see the damage, and the worst latencies never get sampled. Gil Tene named this failure coordinated omission: the test omits the very requests that would have exposed the tail, then reports a flattering p99. An open-loop generator instead injects requests at a target arrival rate (often a Poisson process) regardless of how the server is coping, so a struggling server visibly builds a queue and the tail blows up where it should. This is exactly why MLPerf’s Server scenario uses Poisson arrivals rather than a fixed client pool — a closed loop would report a number that production’s real, uncoordinated traffic never honors.

With honest steady-state numbers in hand, the next problem is watching them on live, unpredictable traffic — observability.

Check your understanding

1. Why does the input/output length distribution change every other metric, and what should you sample it from?
2. What is the “knee” in a concurrency sweep, and why is it the operating point you want?
3. Why must you discard warmup requests, and what one-time cost is worth measuring separately?
4. A vendor reports “8 000 tok/s” with no other detail. Name three things you would demand before trusting it.
5. Given 4 200 output tok/s on a $2/hour GPU, estimate the cost per 1M output tokens (show the arithmetic).

Show answers

1. Length distribution decides the prefill-vs-decode mix, which drives KV memory use, batch composition, TTFT, and throughput — so changing it moves everything. Sample it from your own production logs (or a representative public trace like ShareGPT), not a fixed length.
2. The knee is the concurrency where throughput stops rising and p99 latency starts exploding — the GPU is saturated. It is the maximum load that still meets the SLO, i.e. the point of maximum goodput; running past it inflates raw throughput while breaching latency.
3. Early requests run with cold caches, half-empty batches, and uncompiled kernels, so they are unrepresentative and poison averages. The first long prompt’s one-time kernel/graph compilation is a real cold-start cost worth measuring on its own rather than smearing into steady-state TTFT.
4. Any three of: model + quantization, hardware, input/output length distribution, concurrency level, the SLO, whether warmup was excluded, and which percentiles. Without the recipe the number is unreproducible.
5. 4 200 tok/s × 3 600 s/hour = 15.12M tokens/hour. $2 ÷ 15.12M tokens = ~$0.132 per 1M output tokens.