Revision · Part 2 · The Transformer, Mechanically

Part 2 opens the box, tracing a single request from raw text to a generated token and stopping at each component to ask where the latency, memory, and dollars go. The throughline: parameters set the cost of decode, and sequence length sets the cost of attention and the KV cache.

What this part covered

• Token count is the bill — a model sees integer token IDs, not words, and you pay per token in and out; every downstream cost (KV cache, attention, latency) scales with token count, set before any GPU math runs.
• BPE and embeddings — Byte-Pair Encoding merges frequent byte pairs into subword tokens (~4 chars/token in English), and each ID is a cheap row lookup into a large embedding table (~1 GB for Llama-3-8B), so non-Latin scripts and code pay a structural token tax.
• Attention is O(n²) — every token’s Query dots against every Key to build an n×n score matrix per head per layer; doubling context quadruples attention’s compute and memory, which is why long context is the expensive regime.
• Attention scores dwarf the weights — at n=8,192 the transient matrices across 32 heads × 32 layers would be ~137 GB versus ~16 GB of weights, motivating FlashAttention (never materialize the matrix) and the KV cache (don’t recompute K/V).
• The FFN is where the parameters live — the feed-forward block’s expand-activate-project matrices are ~8d² of each layer’s 12d² (two-thirds of the weights), so it dominates weight-memory and decode-time bandwidth, and sets up quantization and Mixture of Experts.
• Parameter arithmetic — N ≈ n_layers × 12 × d_model² estimates a model’s size from its shape (Llama-3-8B ≈ 6.4B in layers, ~8B with embeddings ≈ 16 GB in FP16), the floor under every latency and dollar figure.
• Prefill vs decode have opposite bottlenecks — prefill processes the whole prompt in one parallel, compute-bound pass (sets TTFT); decode emits one token at a time, memory-bound, reading all weights per token (sets TPOT) and dominating wall-clock time (~97% in the worked example).
• Batched decode is a tiny prefill — stacking B requests into one weight-read raises arithmetic intensity toward compute-bound, the seed of continuous batching in Part 3.
• The KV cache trades compute for memory — since a past token’s K and V never change, caching them turns naive O(n²) decode into O(n); the cost is memory that grows with sequence × batch × layers and often caps concurrency more than the weights do.

The takeaway

The transformer’s cost has two levers: parameters (the memory-bound weight-loading that decode pays every step, concentrated in the FFN) and sequence length (the quadratic attention cost and the linearly growing KV cache). Every serving optimization ahead is a targeted attack on one of these lines — read the mechanics right and Part 3’s Inference Efficiency Core reads as obvious consequences rather than tricks.