Tokenization & Embeddings

The overview promised we’d trace a request from raw text to a token. The very first step is also the one that shows up on your invoice: turning text into a sequence of integers. This page is where the cost meter starts ticking.

Why this matters first: token count is the bill

A model never sees characters or words. It sees tokens — integer IDs from a fixed vocabulary. Two facts make this the most financially important page in the part:

• You are billed per token, input plus output. Every API price (e.g. “$3 per million input tokens”) is denominated in tokens, not words or requests.
• Every cost we study later — KV cache size, attention compute, latency — scales with token count, not character count.

So before any GPU does any math, your bill is already determined by how many tokens your text becomes. Controlling token count is the cheapest optimization that exists, and it’s entirely in your hands at the application layer.

From text to tokens: BPE

Modern LLMs tokenize with Byte-Pair Encoding (BPE) or a close variant. The idea: start from individual bytes/characters, then repeatedly merge the most frequent adjacent pair into a new token, building a vocabulary of common subword chunks.

"tokenization"  →  ["token", "ization"]
"GPT-4o"        →  ["G", "PT", "-", "4", "o"]
" the"          →  [" the"]        (leading space is part of the token)
"日本語"         →  multiple byte-level tokens (non-Latin costs more)

The result: common English words are often a single token; rare words, code, whitespace, and non-Latin scripts fragment into many. A leading space is usually part of the token, which is why "the" and " the" are different IDs.

Note

A durable rule of thumb for English: 1 token ≈ ¾ of a word, or about 4 characters. So 1,000 tokens ≈ 750 words ≈ a page and a half. This is approximate — it varies by text and tokenizer.

The vocabulary and token IDs

The tokenizer carries a fixed vocabulary — a lookup table mapping each subword string to an integer ID. Vocab sizes typically run from tens of thousands to a few hundred thousand entries (GPT-2: ~50k; Llama-3: ~128k; many models: 32k–256k). A larger vocabulary means each token carries more text on average (fewer tokens per document), at the cost of a bigger embedding table.

Tokenization is just: scan the text, greedily match the longest known vocabulary entries, emit their IDs.

" Attention is all"  →  [220, 9784, 374, 682]   (illustrative IDs)

From IDs to vectors: embeddings (a lookup)

The model can’t multiply integers like 9784 meaningfully, so each ID indexes into an embedding matrix of shape [vocab_size, d_model]. Row 9784 is the embedding vector for that token. That’s the entire operation — a table lookup, not a matrix multiply. It’s cheap in compute but the table itself is large.

embedding matrix  E : [vocab_size × d_model]
token id 9784  ──►  E[9784]  ──►  vector of length d_model   (e.g. 4096 floats)

Worked: how big is the embedding table?

For Llama-3-8B: vocab ≈ 128,000, d_model = 4,096, weights in FP16 (2 bytes).

128,000 × 4,096 × 2 bytes = ~1.05 GB just for the input embedding table. A bigger vocabulary buys shorter sequences but spends GPU memory here — a real latency/memory/dollars trade-off, decided before a single attention score is computed.

Context window = max tokens

A model’s context window is its maximum sequence length in tokens — 8k, 128k, 1M, etc. Everything (prompt + generated output) must fit. Note what this means: a “128k context” model holds ~96,000 English words, but far fewer if your input is dense code or Japanese, because those tokenize into more tokens per character.

Different tokenizers count differently

The same string produces different token counts under different tokenizers. So “how many tokens is this?” has no universal answer — it’s model-specific. Switching model families can change your token count (and bill) by 10–30% for the same text, especially for code and non-English.

Worked example: estimate a paragraph

Take this paragraph (the one you’re reading counts too). Suppose a sample is 510 characters and 88 words.

• By the 4-chars-per-token rule: 510 ÷ 4 ≈ 128 tokens.
• By the ¾-word rule: 88 ÷ 0.75 ≈ 117 tokens.

The two estimates bracket the truth (~120 tokens). At $3 / 1M input tokens, that paragraph costs about 120 × $3 / 1,000,000 = $0.00036 to read — and you pay again, at output rates, for whatever the model writes back. Multiply by millions of requests and token-counting becomes a budget line, which is exactly why Part 6 obsesses over trimming context.

Under the hood — byte-level BPE never says “unknown”

How does a 128k-entry vocabulary handle an emoji, a Korean sentence, or a typo it has never seen? Older tokenizers had an <UNK> token for out-of-vocabulary words and simply lost that information. Modern LLMs avoid this with byte-level BPE: the base alphabet isn’t characters, it’s the 256 possible bytes. Any text encodes to UTF-8 bytes first, and since every byte is in the vocabulary, every possible string is representable — there is no out-of-vocabulary case, ever. The merges then glue common byte sequences into bigger tokens.

That’s also why cost is so uneven across languages. An English word like ” the” is one learned merge — one token. The character “日” is three UTF-8 bytes, none of which merged into a Japanese-specific token if the training corpus was mostly English, so it can cost several tokens by itself. Same model, same vocabulary, but non-Latin scripts and code pay a structural token-count tax — and since token count is the bill, that is a real, often-overlooked cost gap between an English and a multilingual workload.

It actually happened — the “glitch tokens” (2023)

In early 2023, researchers probing the GPT-2/GPT-3 tokenizer found a cluster of tokens — the most infamous being SolidGoldMagikarp — that made the models behave bizarrely: asked simply to repeat the word back, the model would dodge, hallucinate, or insult the user. The cause was pure tokenization economics. These strings (mostly scraped usernames and forum artifacts) appeared often enough in the tokenizer’s training data to earn their own dedicated token, but were almost absent from the model’s training data. So the token existed with a barely-trained embedding row — a vector the model had never learned to interpret — and feeding it in steered the model into undefined behavior. The lesson: the tokenizer and the model are trained separately, and a token is only as meaningful as the embedding the model managed to learn for it.

The architect’s lens

Tokenization is upstream of every other cost — the bill is set here before a GPU does any math. The five questions:

• Why does it exist? A model can’t multiply raw characters or words; it needs a fixed integer vocabulary. Byte-Pair Encoding builds one by merging frequent adjacent pairs into subword chunks, and the embedding matrix turns each ID into a vector by a simple row lookup, not a matmul.
• What problem does it solve? Mapping arbitrary text into a finite vocabulary the model can process — and doing it compactly, since you are billed per token and every downstream cost (KV cache, attention, latency) scales with token count, not characters.
• What are the trade-offs? Vocabulary size is the knob: a larger vocab (Llama-3’s ~128k) packs more text per token so sequences are shorter, but the embedding table grows — ~1 GB in FP16 for Llama-3-8B — and byte-level BPE makes non-Latin scripts and code cost several tokens per character.
• When does the choice bite? When token count balloons — spending fewer tokens is the cheapest optimization there is — or when you switch model families and the same text re-tokenizes 10–30% larger, silently changing your bill.
• What breaks if I remove it? There is no path from text to the model without it; and since token count is the invoice, it is the first lever every application-layer optimization (context trimming, caching) pulls.

Check your understanding

1. Why is “token count is the bill” the financially central idea of this page?
2. What does Byte-Pair Encoding actually do to build a vocabulary?
3. Roughly how many tokens is a 1,000-word English document, and why is that only approximate?
4. The ID-to-vector step is described as “a lookup, not a matmul.” What is the operation, and what is its main cost?
5. Why can two models report different token counts — and therefore different prices — for the identical input string?

Show answers

1. Because you are billed per token (input + output) and every downstream cost — KV cache, attention, latency — scales with token count, so the bill is largely fixed before any GPU math runs.
2. It starts from individual characters/bytes and repeatedly merges the most frequent adjacent pair into a new token, building a vocabulary of common subword chunks.
3. About 1,300 tokens (1,000 ÷ 0.75 ≈ 1,333). It’s approximate because the true count depends on the specific words, whitespace, punctuation, and the tokenizer used.
4. It’s a table lookup: the token ID indexes a row of the [vocab_size × d_model] embedding matrix. Compute is trivial; the main cost is the memory the large embedding table occupies (≈1 GB for Llama-3-8B).
5. Different tokenizers have different vocabularies and merge rules, so the same string splits into different numbers of tokens — diverging most on code and non-Latin text.