The Serving Stack: vLLM, TGI, TensorRT-LLM, SGLang

You already understand the hard parts. PagedAttention and continuous batching are not things you implement by hand for production — you get them from a serving engine that bundles every Part 3 technique behind a clean API. This page is a tour of the four that matter, and the one question that picks between them: what is this engine built to make cheap — throughput, latency, or developer time?

What an inference server actually does

Strip away the branding and every serving engine does the same five jobs:

   HTTP request ──► [ 1. API layer        ] OpenAI-compatible /v1/chat/completions
                    [ 2. Tokenize         ] text → token ids
                    [ 3. Scheduler        ] who runs this GPU step? (continuous batching)
                    [ 4. Model executor   ] forward pass + KV-cache manager (paged)
                    [ 5. Detokenize/stream ] token ids → text, streamed back
   ◄── tokens ──────┘

It loads the model once into HBM (optionally quantized, per Part 4), manages the KV cache (paged blocks, not contiguous reservations), batches requests at the iteration level, schedules every GPU step, and exposes an OpenAI-compatible API so your application code does not care which engine is underneath. The differences between engines are almost entirely in how well they do jobs 3 and 4 — and on which hardware.

The four engines

vLLM — the open-source default. It is the project that introduced PagedAttention, and it pairs that with continuous batching, FlashAttention kernels, and automatic prefix caching. Optimizes for throughput (tokens/sec/GPU, hence dollar-per-token) while staying easy to run and supporting a huge range of models. If you have no strong reason otherwise, you start here.

TGI (Text Generation Inference) — HuggingFace’s server. Tightly integrated with the HuggingFace ecosystem and model hub, production-hardened with metrics, tracing, and safe-tensor loading out of the box. Optimizes for operational smoothness inside the HF stack; performance is competitive and it carries the same core tricks (paged KV, continuous batching). Note the momentum, though: by late 2025 HuggingFace put TGI into maintenance mode and its own Inference Endpoints now default to vLLM/SGLang backends, so treat TGI as the ecosystem-fit choice rather than the leading edge.

TensorRT-LLM — NVIDIA’s engine. Instead of running the model through a general framework, it compiles the model into fused, hardware-specific CUDA kernels ahead of time. On NVIDIA GPUs this is typically the lowest-latency, highest-throughput option — but the compile step is per-model, per-GPU, per-config, so it is less flexible: changing the model or shape means rebuilding the engine. Optimizes for raw NVIDIA performance at the cost of iteration speed.

SGLang — built around RadixAttention, which generalizes prefix caching into a radix tree so many requests sharing overlapping prefixes all reuse the same KV automatically. That makes it strong for structured and agentic workloads — multi-turn chats, tool-calling loops, branching prompts, constrained/JSON decoding — where shared context is the norm. Optimizes for complex, prefix-heavy programs, not just one-shot completions.

Comparison

Engine	Built by	Signature trick	Optimizes for	Trade-off
vLLM	community	PagedAttention	throughput, broad model support	jack-of-all-trades default
TGI	HuggingFace	HF-native serving	ops smoothness in HF stack	tied to HF conventions
TensorRT-LLM	NVIDIA	compiled fused kernels	lowest latency on NVIDIA	rigid: recompile per change
SGLang	community	RadixAttention	agentic/structured, shared prefixes	newer, narrower sweet spot

They all implement Parts 3 and 4

Paged KV, continuous batching, FlashAttention, and quantization are table stakes — every engine here has them. The differentiators are the extra idea on top (compiled kernels, radix-tree prefix sharing) and the ecosystem fit. Choosing an engine is not choosing whether to use these techniques; it is choosing which additional optimization and which operational style you want.

The cost lens: a worked comparison

Suppose you serve a 13B model on one 80 GB GPU renting at $2/hour, and an interactive chat workload where most requests share a long system prompt.

Baseline (no prefix reuse), engine sustains  2,000 tok/s
  cost  = $2/hr ÷ 3600 s = $0.000556 /s
  per token = $0.000556 ÷ 2000 ≈ $0.00000028  → $0.28 per 1M tokens

Prefix-heavy traffic on a RadixAttention engine that skips
80% of the shared prefill, lifting effective throughput to 3,200 tok/s:
  per token = $0.000556 ÷ 3200 ≈ $0.00000017  → $0.17 per 1M tokens

Same GPU, same hourly rent, same model — ~40% lower dollar-per-token purely from the engine matching the workload’s prefix structure. That is the whole point of choosing a stack: the techniques are fixed, but how well an engine exploits your traffic is not.

How to choose, in one breath

General workload, want it working today → vLLM. Already all-in on HuggingFace → TGI. Squeezing the last millisecond out of fixed NVIDIA hardware → TensorRT-LLM. Agents, tool loops, heavy shared context → SGLang.

Under the hood — RadixAttention is a radix tree of KV blocks

Ordinary prefix caching reuses one shared prefix — a system prompt — across requests. SGLang’s RadixAttention generalizes that to any shared prefix by keying cached KV blocks in a radix tree: a trie of token sequences, compressed so single-child chains collapse into one edge. When a request arrives, the engine walks the tree matching its tokens as far as they go, reuses the KV for that whole matched path, and computes only the divergent tail. Branches share their common ancestor automatically — a tool-calling agent that forks into five candidate continuations pays for the shared context once and the five tails separately — and an LRU policy evicts cold branches when KV memory tightens. That is why it shines on the prefix-heavy agentic traffic in the comparison above: the data structure turns “requests that happen to overlap” into “requests that provably share a path.”

The architect’s lens

The serving stack is the build-vs-adopt decision for everything in Parts 3–4. The five questions:

• Why does it exist? Because PagedAttention, continuous batching, and FlashAttention are not things you hand-implement for production — an engine bundles every Part 3 and Part 4 technique behind a clean, OpenAI-compatible API.
• What problem does it solve? Turning a model into an always-on service: load weights once, manage the paged KV cache, batch at the iteration level, schedule every GPU step, and stream tokens back — so your application code never touches the internals.
• What are the trade-offs? Each engine optimizes a different axis — vLLM for throughput and broad model support, TGI for HuggingFace ops, TensorRT-LLM for lowest latency on NVIDIA (at the cost of a rigid per-config recompile), SGLang for prefix-heavy agentic traffic — and matching engine to workload can swing dollar-per-token ~40% on the same GPU.
• When should I avoid it? Don’t reach for TensorRT-LLM if you change models often (rebuilds are slow), and SGLang’s RadixAttention buys little on one-shot completions with no shared prefixes — pick the engine your traffic actually exercises.
• What breaks if I remove it? You hand-build the paged KV manager, scheduler, and batcher yourself — months of systems work — and a bare model.generate() serves one request at a time and crashes when two arrive together.

Check your understanding

1. List the five jobs every inference server does between receiving a request and returning tokens.
2. Why does an OpenAI-compatible API let you swap engines without touching application code?
3. What does TensorRT-LLM do differently from vLLM, and what flexibility does it give up for that?
4. What workload is SGLang’s RadixAttention especially good for, and why?
5. In the worked example, the GPU and its hourly rent never changed — so why did dollar-per-token drop?

Show answers

1. Expose an API, tokenize, schedule (decide which requests run each GPU step), run the model forward pass while managing the KV cache, and detokenize/stream the result back.
2. Because the request/response contract is identical across engines, so the application only depends on the API shape, not on the engine internals behind it.
3. It compiles the model ahead of time into fused, hardware-specific CUDA kernels for top latency/throughput on NVIDIA, giving up flexibility — any change to model, shape, or config requires rebuilding the engine.
4. Structured/agentic workloads (multi-turn chat, tool loops, branching prompts) where many requests share overlapping prefixes; the radix tree lets them all reuse the same cached KV automatically.
5. The RadixAttention engine matched the prefix-heavy traffic and skipped most redundant shared prefill, raising effective throughput from 2,000 to 3,200 tok/s, so the same per-second cost was divided over more tokens.