Observability for AI Systems

Load testing gave you steady-state numbers under controlled traffic. Production is the uncontrolled version: traffic you did not choose, prompts you did not write, at hours you are asleep. Observability is how you keep reading the metrics from the last two pages on live traffic — and it requires instrumenting things a classic web dashboard never thought to measure.

Why classic infra metrics are not enough

A standard service emits CPU, memory, request rate, and HTTP error rate. For an LLM service those are necessary but blind to the things that actually drive cost and quality. CPU at 30% looks healthy while the GPU is melting; a 200 OK hides a request that streamed for 40 seconds and burned 8 000 tokens. The unit of cost here is the token, and tokens are invisible to ordinary monitoring. So the cost throughline becomes a literal instrumentation task: if you cannot see tokens and GPU state, you cannot see dollars.

The signals that matter

Layer these on top of the usual infra metrics.

Per-request economics

• Tokens in / tokens out per request — the raw driver of cost. Histogram them; the tail (giant prompts, runaway generations) is where surprise bills come from.
• Cost per request — derived: (prompt_tokens × prefill_cost) + (output_tokens × decode_cost). This is the number finance and product both want.
• TTFT / TPOT histograms — the latency metrics, now as live distributions so you can watch p99, not just p50.

Engine internals (the GPU’s vital signs)

• GPU utilization & HBM memory used — is the silicon busy, and how close to OOM. An idle GPU is pure dollar loss; a near-full one is about to start rejecting or preempting.
• KV-cache utilization — what fraction of the KV blocks are in use. This is the real capacity gauge: when KV is full, no new request can be admitted regardless of spare compute.
• Queue depth & waiting time — how many requests are parked before prefill, and for how long. Rising queue depth is the leading indicator that you are past the knee and TTFT is about to breach SLO.
• Cache hit rate — for prefix/prompt caching, the fraction of prompt tokens served from cache. A hit skips prefill entirely, so this number maps directly to TTFT and dollars saved.
• Batch size (running) — the live average; if it collapses, your amortization (and dollar-per-token) just got worse.

Healthy vs sick, read off the engine signals:

           GPU util   KV util   queue depth   batch   p99 TTFT
healthy      85%        70%         ~0          28      410 ms
saturated    99%        99%        rising       64      3 200 ms  ← shed load
underused    20%        15%         0            3       120 ms   ← scale down, save $

That last row is the one teams miss: low utilization is not “everything’s fine,” it is money on fire — you are paying for a GPU at 20% load. Observability is what lets you catch it and scale down.

Tracing a request end to end

Aggregate metrics tell you the system is slow; a trace tells you where. Instrument each request as a span tree so you can attribute its latency to a stage.

trace: request 9f2a  (total 5.4 s)
├─ retrieve (RAG)        220 ms   embed query + vector search + fetch chunks
├─ queue wait            140 ms   parked before admission
├─ prefill               380 ms   ← becomes TTFT boundary
│   └─ kv-cache: miss (cold prefix)
├─ decode  (200 tok)    4 600 ms   ← TPOT 23 ms × 200
└─ post-process / stream  60 ms

Now a “this request was slow” complaint is debuggable: was it the vector DB (retrieve), a full queue (admission), a long prompt (prefill), or just a long answer (decode)? Each points to a different fix — and a different cost lever. Standard tracing (OpenTelemetry-style spans) plus LLM-specific tooling (LangSmith, Langfuse, Phoenix, Helicone, and similar) capture these; the key is that prefill and decode are separate spans, because they have separate bottlenecks.

Cost attribution: per-feature, per-customer

Total spend is not actionable; attributed spend is. Tag every request with customer_id, feature, and model, and you can slice the token/cost metrics to answer the questions that change decisions:

• Which customer is 60% of your inference bill? (Maybe their plan is mispriced.)
• Which feature has the worst cost-per-successful-action? (Maybe it should use a smaller model.)
• Did last week’s prompt change quietly double tokens-per-request? (A regression you would otherwise eat silently.)

Without attribution you optimize blind. With it, you point every efficiency lever from earlier parts at the line item that is actually bleeding.

Cost lens

Every signal here is a dollar gauge in disguise: tokens = the bill, KV/GPU utilization = how much of what you rented you actually use, cache hit rate = prefill you avoided paying for, queue depth = SLO breaches about to cost goodput. Dashboards that show only latency hide three-quarters of the cost story.

Logging prompts and responses — and the privacy trade

The richest debugging signal is the actual prompt and response text. It is also the most dangerous to store: prompts contain PII, secrets, proprietary data, and whatever a user pasted in. The trade is explicit.

• For debugging/evals you want full prompt+response logs (they become your eval set — the next page).
• For privacy/compliance you want to log as little raw content as possible.

Common middle grounds: log metadata always (token counts, latency, model, hashes) but gate raw text behind consent and short retention; redact or tokenize PII before storage; sample (log 1% of raw bodies, 100% of metadata); and keep raw content in a separately-access-controlled store. There is no free lunch — more logging is better observability and worse privacy posture, and you must choose the point on that line deliberately rather than by default.

Under the hood — a standard schema for LLM telemetry

Tracing only pays off if every service tags spans the same way; otherwise “tokens out” means three different things across three teams. The industry’s answer is the OpenTelemetry GenAI semantic conventions — an emerging, standardized set of attribute names for LLM spans and metrics: gen_ai.request.model, gen_ai.usage.input_tokens, gen_ai.usage.output_tokens, gen_ai.operation.name, and friends. Adopt them and a generation span carries its model and token counts in fields any backend already understands, so cost-per-request becomes a query instead of a bespoke parser per provider. The conventions are still marked experimental and shift release to release, but standardizing the names is what lets the token — the unit of cost — flow into the same dashboards as latency and errors rather than living in a private side-channel.

The logs you do keep are not just for debugging — they are the raw material for measuring whether your optimizations hurt quality, which is the whole subject of the next page.

Check your understanding

1. Why are CPU, memory, and HTTP status insufficient for an LLM service? What is the missing unit of cost?
2. KV-cache utilization hits 99% while GPU compute has headroom. What happens to new requests, and why?
3. Why is low GPU utilization a problem rather than a relief?
4. In the trace example, which spans set TTFT and which set the rest of end-to-end latency?
5. State the logging trade in one sentence, and name two middle-ground techniques.

Show answers

1. They miss the GPU and the token. CPU can look idle while the GPU saturates, and a 200 OK hides token-heavy, multi-second requests. The unit of cost is the token, which ordinary monitoring does not see — so dollars are invisible without token/GPU instrumentation.
2. New requests cannot be admitted: a request needs free KV blocks to hold its cache, and with KV full there is nowhere to put it regardless of spare compute. They queue (or get preempted), so TTFT rises.
3. An idle or under-loaded GPU still costs full price per hour, so low utilization is money wasted — it signals you should scale down or consolidate, not that the system is healthy.
4. retrieve + queue wait + prefill set TTFT (everything up to the first token); decode and post-process/stream make up the remainder of end-to-end latency.
5. More logging means better debugging/evals but worse privacy exposure, so you must choose deliberately. Middle grounds: log metadata always but gate/redact raw text, sample raw bodies, tokenize/redact PII, and use short retention with separate access control.