The Metrics That Matter: TTFT, TPOT, Goodput

The Overview promised to make “make it faster” precise. That starts with naming the numbers. An LLM request is not one latency — it is two phases with opposite bottlenecks, and each phase produces its own metric. Get the names right and the whole operating picture snaps into focus; blur them and you will optimize the wrong half of the system.

The two latencies, from the two phases

A streaming request feels like two events to the user: the wait for the cursor to start moving, then the speed at which words keep arriving. Those map exactly onto prefill and decode.

• TTFT — Time To First Token. From request arrival to the first token streamed back. It is dominated by prefill: the model must process the entire prompt before it can emit anything. TTFT is the UX-critical number for streaming interfaces — it is how long the screen sits blank. Long prompts and queue waiting both inflate it.
• TPOT — Time Per Output Token (also called ITL, inter-token latency). The average gap between successive streamed tokens once generation is underway. It comes from decode, where each step reads the weights from HBM to make one token. TPOT sets the reading speed the user perceives.

request ──► [ QUEUE ]──► [ PREFILL ]──► tok1 ─ tok2 ─ tok3 ─ ... ─ tokN ─► done
            └─────────── TTFT ──────────┘   └TPOT┘ └TPOT┘

end-to-end = TTFT + TPOT × (output_tokens − 1)

Worked example: end-to-end latency

Suppose a request has TTFT = 400 ms and TPOT = 25 ms, and the model generates 200 output tokens.

end-to-end = TTFT + TPOT × (output_tokens − 1)
           = 400 ms + 25 ms × 199
           = 400 ms + 4 975 ms
           = 5 375 ms ≈ 5.4 s

Notice what dominates: the prompt’s prefill bought you 0.4 s, but the 200-token answer cost ~5 s of decode. For any non-trivial answer length, TPOT × output_tokens is the bulk of the wall-clock time — which is exactly why decode efficiency (batching, the KV cache) is where the dollars and the seconds both live. Halving TPOT here saves ~2.5 s per request; shaving TTFT can only ever save 0.4 s.

Why both matter, separately

A chatbot with great TPOT but a 3-second TTFT feels broken — the user stares at nothing. A voice agent with a snappy TTFT but sluggish TPOT sounds like it is buffering mid-sentence. They are different complaints with different fixes (queueing/prefill vs batch size/decode), so never collapse them into one “latency” number.

Throughput: the system’s side of the ledger

Latency is what one user feels; throughput is what the GPU bill cares about. Two flavors:

• Output tokens/second — total tokens the server emits across all concurrent requests. This is the number that divides into your GPU-hour cost to give dollars-per-token.
• Requests/second (RPS) — completed requests per second, the capacity-planning unit.

Throughput and latency pull against each other. Bigger batches read the weights once and serve more tokens — throughput up, dollar-per-token down — but each decode step now does more arithmetic, so each user’s TPOT rises. This is the continuous-batching trade, now restated as a measurement: you cannot report one without the other.

Goodput: the metric that actually matters

Here is the trap. You can push raw throughput arbitrarily high by cramming the batch — and serve every one of those requests too slowly to be useful. A request that violates its latency SLO is not revenue; it is a timeout, a bounced user, a refund. So the metric that actually matters is goodput: throughput counting only the requests that meet their SLO.

Goodput = requests/second that satisfy both TTFT ≤ target and TPOT ≤ target.

Worked example: throughput vs goodput

Say your SLO is TTFT ≤ 500 ms and TPOT ≤ 30 ms. You run two configurations:

Config	Raw RPS	% meeting SLO	Goodput (RPS)
A: batch 16	40	95%	38
B: batch 64	70	50%	35

Config B looks 75% faster on the spec sheet — but half its requests breach the SLO, so it delivers fewer useful requests per second than A. Optimizing raw throughput chose the worse system. Goodput is the number that keeps you honest.

Percentiles, not averages

Never report a mean latency. Latency distributions are right-skewed — a few requests hit a cold cache, a long prompt, or a full queue — and the average hides them. Report percentiles: p50 (the median, typical experience), p95, and p99 (the unlucky 1%, who are often your most active users and loudest complainers).

A mean of 800 ms can mean either:
  p50 = 780 ms, p99 = 900 ms   → tight, healthy
  p50 = 400 ms, p99 = 6 000 ms → most users fine, a tail on fire
The mean (800 ms) is identical. Only percentiles tell them apart.

SLOs are written on percentiles — “p99 TTFT < 500 ms” — precisely because the tail is what users remember and what your goodput calculation enforces.

The cost lens on metrics

Each metric maps to a cost lever. TTFT → prefill/queue cost (latency). TPOT → decode/batch cost (latency and dollars). Throughput → GPU utilization (dollars). Goodput → the only one that ties latency and dollars to delivered value. When someone proposes an optimization, ask which of these four it moves — and whether it quietly moves another the wrong way.

Under the hood — TPOT is an average; ITL is the jitter

“TPOT” and “ITL” get used interchangeably, but the precise definitions differ in a way that matters when you read a dashboard. TPOT is almost always a per-request average: total decode time divided by output tokens, (end_to_end − TTFT) ÷ (output_tokens − 1). ITL is the gap between two specific consecutive tokens — a whole distribution per request, not one number. A request can post a healthy mean TPOT while hiding ugly ITL spikes: one token arrives 200 ms late because a chunked-prefill step or a batch admission stole that scheduler slot. Average it away and the stutter the user actually felt vanishes from the metric. This is why standardized benchmarks pin the tail: MLPerf Inference’s Server scenario, for one, holds submissions to per-token latency constraints rather than a mean, precisely so a smooth average can’t launder a jittery stream.

With the vocabulary fixed, the next question is how to generate these numbers honestly — which is load testing.

Check your understanding

1. Which phase dominates TTFT, and which dominates TPOT? Why?
2. Using TTFT = 300 ms, TPOT = 20 ms, and 150 output tokens, compute end-to-end latency.
3. Define goodput, and explain why a system with higher raw throughput can have lower goodput.
4. Why must you always report TTFT and TPOT as percentiles rather than averages?
5. A change cuts TPOT by 30% but, by enlarging the batch, pushes p99 TTFT past the SLO. What happened to throughput, and what happened to goodput?

Show answers

1. Prefill dominates TTFT — the model must process the whole prompt before emitting the first token. Decode dominates TPOT — each subsequent token is one memory-bound decode step reading the weights from HBM.
2. end-to-end = 300 + 20 × (150 − 1) = 300 + 2 980 = 3 280 ms ≈ 3.3 s.
3. Goodput is the rate of requests that meet their latency SLO (both TTFT and TPOT targets). Cramming the batch raises raw throughput but pushes many requests past their SLO, so they no longer count as useful — raw throughput rises while goodput falls.
4. Latency distributions are right-skewed; the mean hides a heavy tail. Percentiles (p50/p95/p99) expose the unlucky requests, and SLOs are defined on the tail (e.g. p99) because that is what users remember.
5. Raw throughput likely rose (bigger batch, faster decode), but because p99 TTFT now violates the SLO, the breaching requests stop counting — so goodput fell. The “win” was a net loss.