Multimodal & Diffusion Efficiency

Tokenization & embeddings showed that for text, token count is the bill — characters become tokens, and tokens are what you pay for. Multimodal and image-generation models obey the same law, but the unit changes. An image fed into a model becomes a surprising number of tokens. An image generated by a diffusion model is built up over many iterative steps, and the step count is the cost driver. This page traces both: what it costs to let a model see, and what it costs to let a model draw — and the techniques that shrank each by 2024–2025.

Images become tokens too

A vision-language model can’t feed raw pixels into a transformer that expects a sequence of token vectors. So it runs the image through a vision encoder — typically a Vision Transformer (ViT) — which slices the image into a grid of fixed-size patches, embeds each patch into a vector, and hands that sequence of patch-vectors to the model as if they were tokens.

   image ─▶ cut into patches ─▶ embed each patch ─▶ sequence of "image tokens"
                                                      │
                                                      ▼
                                          fed into the model like text tokens

The consequence: an image costs image tokens, and a high-resolution image costs a lot of them. More pixels means more patches means more tokens — and those tokens are billed and prefilled exactly like text. A common design pattern is tiling: a big image is split into tiles, each tile encoded separately, so doubling resolution can multiply the token count. Vision is not free context; it is a large, resolution-dependent chunk of your prompt budget.

By the numbers — an image is a lot of tokens

Suppose a vision encoder uses 14×14-pixel patches with no overlap. A modest 336×336 image becomes a grid of (336 / 14) × (336 / 14) = 24 × 24 = 576 patches — so roughly 576 image tokens for one small image.

336 × 336 image, 14×14 patches  →  24 × 24  =   576 tokens
672 × 672 image (2× per side)   →  48 × 48  = 2,304 tokens   (4× the area, 4× the tokens)

Doubling each side quadruples the area and quadruples the token count. A single high-resolution screenshot can therefore cost as much, in tokens, as several pages of text — which is why high-detail image inputs visibly raise both latency and the bill. The lever is the same as for text: send the resolution the task needs, not the maximum the model accepts.

Diffusion: the cost is the step count

Image generation with diffusion models works backwards from noise. The model starts from pure random noise and iteratively denoises it, a little at a time, over many steps, until an image emerges. Each step is one full forward pass through a large network.

   noise ─▶ denoise ─▶ denoise ─▶ ... ─▶ denoise ─▶ image
            step 1     step 2           step T

   total compute  ≈  T  ×  (cost of one network forward pass)

That makes the number of steps, T, the dominant cost driver — and the dominant latency driver, because the steps are sequential (each denoising step consumes the previous step’s output). The original diffusion formulation (DDPM, Ho et al., 2020) used on the order of a thousand steps. Faster samplers like DDIM (Song et al., 2020) cut that to roughly tens of steps with little quality loss. But “fewer steps” was the central efficiency battle of image generation, because halving the steps roughly halves both cost and time-to-image.

Latent diffusion: denoise in a compressed space

The second big lever attacks the cost per step rather than the number of steps. Running diffusion directly on full-resolution pixels is enormously expensive — every step processes millions of pixel values. Latent diffusion (Rombach et al., 2022 — the basis of Stable Diffusion) first uses an autoencoder to compress the image into a much smaller latent representation, runs the entire denoising loop in that compact latent space, and only at the very end decodes the final latent back into pixels.

   pixels ─encode─▶ small latent ─▶ [ denoise loop runs HERE ] ─▶ latent ─decode─▶ pixels
            (once)        ↑ every step is cheap because the space is small        (once)

Because the latent is far smaller than the pixel image, every one of the T denoising steps is dramatically cheaper. You pay the full-resolution cost only twice — once to encode, once to decode — instead of on every step. This is the same cost-shifting instinct as the rest of the book: do the expensive work in the smallest representation you can get away with.

Few-step distillation: consistency and turbo models

The frontier move was to collapse T itself — from tens of steps down to a handful, or even one. Consistency models (Song et al., 2023) are trained so that any point along the denoising trajectory can jump more or less directly to the final result, enabling one- or few-step generation. The same idea shows up as latent consistency models (LCM) and as adversarially-distilled “turbo” models (e.g. SDXL Turbo, 2023) that generate a usable image in one to four steps instead of dozens.

The mechanism is distillation: a slow, many-step “teacher” diffusion model trains a fast “student” to reproduce its output in far fewer steps. As with reasoning distilled back into small models, the expensive iterative work is moved into training, paid once, so inference becomes cheap.

Lever	Attacks	Effect
Fewer-step samplers (DDIM)	step count T	~1000 → tens of steps
Latent diffusion	cost per step	denoise in a small compressed space
Few-step distillation (consistency / turbo)	step count T	tens → 1–4 steps
Lower input resolution	image-token count	fewer patches per image

It actually happened — latent diffusion and Stable Diffusion (2022)

High-Resolution Image Synthesis with Latent Diffusion Models (Rombach and colleagues, 2022) introduced running the diffusion process in a compressed latent space rather than on raw pixels. Released publicly as Stable Diffusion in 2022, it cut the compute per step enough to bring high-quality image generation from datacenter-only down to a single consumer GPU, and it set off the open image-generation wave. It is one of the cleanest demonstrations of this book’s recurring move: the model didn’t get fundamentally simpler — the team just refused to do the expensive iterative work at full pixel resolution, doing it in a small latent space instead, and the cost (and the hardware bar) collapsed.

The throughline, in pixels

Whether a model sees or draws, ask the same question: what does this cost in latency, memory, and dollars, and how do we make it cheaper? For seeing, the lever is resolution — fewer image tokens. For drawing, the lever is iterations — fewer, cheaper denoising steps. Both answers are versions of “use the smallest representation, and do the fewest passes, that the task can tolerate.”

The architect’s lens

Whether a model sees or draws, the levers shrink the representation and the passes:

• Why does it exist? Because images obey the “token count is the bill” law in a new unit — a vision encoder cuts an image into patches that cost image tokens — and diffusion builds an image over many sequential denoising steps, each a full forward pass.
• What problem does it solve? The two dominant cost drivers: resolution on the input side (a 336×336 image ≈ 576 tokens; doubling each side quadruples it) and step count T on the generation side (DDPM’s ~1000 steps → DDIM’s tens → turbo/consistency’s 1–4).
• What are the trade-offs? Latent diffusion does the denoising in a small compressed space (cheap per step, full-res cost only at encode/decode); few-step distillation moves the expensive iteration into one-time training, so the student rarely matches the slow teacher on the hardest outputs.
• When should I avoid it? Don’t send max resolution when the task doesn’t need the detail, and don’t reach for one-step turbo models when image fidelity matters more than latency.
• What breaks if I remove it? Full-pixel diffusion over ~1000 steps was datacenter-only; high-detail image inputs at full resolution cost as much as several pages of text in tokens, raising latency and the bill on every call.

Check your understanding

1. How does a vision-language model turn an image into something a transformer can process, and why does resolution drive the token count?
2. In the by-the-numbers example, why does doubling each side of an image roughly quadruple its image-token count?
3. For a diffusion model, what is the single dominant cost-and-latency driver, and why are the steps unavoidably sequential?
4. What does latent diffusion change — the number of steps or the cost per step — and how?
5. How do consistency / “turbo” models achieve one-to-few-step generation, and what does that have in common with distilling a reasoning model?

Show answers

1. It runs the image through a vision encoder (a ViT) that cuts it into fixed-size patches and embeds each patch into a vector; the sequence of patch-vectors is fed in like tokens. More pixels means more patches means more image tokens, so resolution directly sets the token count and thus cost.
2. Token count scales with the number of patches, which scales with image area; doubling each side quadruples the area (and patch count), so a 336×336 image at 576 tokens becomes about 2,304 tokens at 672×672.
3. The number of denoising steps, T — total compute is roughly T times one forward pass. The steps are sequential because each denoising step takes the previous step’s partially-denoised output as its input, so they cannot be parallelized away.
4. The cost per step. It compresses the image into a small latent with an autoencoder, runs the whole denoising loop in that compact space, and only encodes/decodes to full pixels once at each end — so every step is cheap, while the step count is unchanged.
5. They are distilled: a slow many-step teacher trains a fast student to reproduce the result in 1–4 steps. Like distilling reasoning into a small model, it moves the expensive iterative work into one-time training so that inference becomes cheap.