The Cost Problem: Why a Token Costs Money

The Overview promised that every page returns to one question: what does this cost, and how do we make it cheaper? Before we can make anything cheaper we have to understand where the cost comes from. A token feels free — you type, tokens stream back. But each one is the output of a staggering amount of arithmetic, run on hardware that is rented by the second. This page traces the chain from “produce one token” to “pay this many dollars.”

A token is a forward pass

To generate one token, the model runs a forward pass: your input flows through every layer, and the result is a probability distribution over the vocabulary. The model picks one token from it. To get the next token, it runs another forward pass. A 500-token answer is 500 forward passes. Each pass touches every parameter in the model at least once — and modern models have billions of them.

That is the heart of the cost problem: work scales with model size, and you pay it once per token.

FLOPs per token: the 2× rule

A FLOP is one floating-point operation (a multiply or an add). The dominant operation in a transformer is matrix multiplication, and there is a clean rule of thumb:

FLOPs per token ≈ 2 × (number of parameters)

Why 2×? Each parameter is used in a multiply-accumulate: one multiply + one add = 2 FLOPs. Nearly every parameter participates once per token in the matmuls, so the total is ~2 × params. (Attention adds a bit more at long context, but the parameter term dominates for typical lengths.)

Plug in real numbers:

Model	Params	FLOPs/token (≈ 2 × params)
7B	7 × 10⁹	~14 GFLOPs
70B	70 × 10⁹	~140 GFLOPs
405B	405 × 10⁹	~810 GFLOPs

So a 70B model burns ~140 billion floating-point operations to produce a single token. A 200-token answer is ~28 trillion FLOPs. This is why bigger models cost more in a way you can compute, not just assert.

The three currencies

Cost is never one number. It has three faces, and optimizing one often worsens another:

Currency	What it measures	Set by
Latency	Time to the answer (and time per token)	Model size, sequence length, batch contention
Memory	GPU RAM to hold weights + activations + KV cache	Params × bytes, context length, batch size
Dollars	What you actually pay	GPU-hours × utilization, or $/1M tokens

Hold this triad in your head for the whole book. Every technique we cover moves one or more of these, usually by trading against another.

From FLOPs to dollars

GPUs are rented by time, and each GPU has a peak FLOPs/second. The bridge is utilization — you rarely hit peak, so assume some fraction (~30–50% is realistic for decode). A back-of-envelope:

$ per token  = FLOPs/token ÷ (GPU FLOPs/sec × utilization) × ($/GPU-hour ÷ 3600)

Take a 70B model on a GPU renting at ~$2/hour with ~300 TFLOP/s effective (peak derated by utilization):

time/token  = 140e9 FLOPs ÷ 300e12 FLOP/s ≈ 4.7e-4 s   (~0.47 ms of pure compute)
$ per token = 4.7e-4 s × ($2 ÷ 3600 $/s) ≈ 2.6e-7 dollars
$ per 1M tokens ≈ 0.26 dollars  (single-stream, compute-only floor)

That looks almost free — and it would be, if you ran one request at a time at full efficiency. In reality decode is memory-bandwidth-bound (Part 1), utilization is low, and you must batch many users to amortize the cost. That gap between this floor and real published prices (often a few dollars per 1M output tokens) is exactly the territory this book explores.

Why $/1M tokens is the lingua franca

APIs price in dollars per million tokens, split into input (prefill, cheap — processed in parallel) and output (decode, expensive — generated one at a time). The asymmetry is real, not a markup quirk, and we’ll see exactly why when we reach prefill vs decode.

Why this becomes your biggest line item

A web request might do a few microseconds of CPU work. An LLM request does trillions of FLOPs on hardware that costs 10–50× a normal server. At scale — millions of requests, long contexts, big models — inference routinely becomes the largest single line item in an AI product’s budget, dwarfing storage, bandwidth, and the rest of the stack combined. That is why “make the token cheaper” is not a micro-optimization; it is often the difference between a viable business and a money furnace.

It actually happened — “an LLM exchange costs ~10× a web search” (2023)

In early 2023, as search engines raced to bolt LLMs onto their results, Alphabet’s chairman John Hennessy told Reuters that an exchange with a large language model “likely costs 10 times more than a standard keyword search.” That one comparison made the cost problem legible to the whole industry: a web search is a cheap lookup, while every LLM reply is the trillions-of-FLOPs forward-pass chain this page traced. Analysts spent months trying to price what it would take to serve LLM answers at search scale — and the numbers were large enough to reshape product roadmaps. The lesson is that the per-token math above isn’t a curiosity; multiplied by billions of queries it decides what is even economically possible to ship.

Check your understanding

1. Why does producing a 500-token answer require roughly 500 forward passes?
2. State the FLOPs-per-token rule of thumb and explain where the factor of 2 comes from.
3. A 70B-parameter model produces one token — roughly how many FLOPs is that, and how did you get it?
4. Name the three currencies of cost and what each is primarily set by.
5. The compute-only floor for a 70B model came out near $0.26 per 1M tokens, yet real APIs charge several dollars. Give two reasons for the gap.

Show answers

1. Each forward pass produces exactly one token; the next token needs another full pass, so an N-token output is N sequential forward passes.
2. FLOPs/token ≈ 2 × params. Each parameter does one multiply and one add (a multiply-accumulate) per token, and that’s 2 FLOPs per parameter.
3. ~140 GFLOPs: 2 × 70 × 10⁹ = 1.4 × 10¹¹ floating-point operations.
4. Latency (set by model size, sequence length, batch contention), memory (params × bytes, context length, batch size), and dollars (GPU-hours × utilization, or $/1M tokens).
5. Decode is memory-bandwidth-bound rather than compute-bound, so real GPU utilization is far below peak; and the floor assumed an idealized single stream — real serving must cover low utilization, batching overhead, idle capacity, and margin. (Output tokens are also generated one at a time, unlike parallel prefill.)