Prompt & Context Economy

The overview named five levers; this is the first and most immediate. Before you cache anything, route anything, or build a retriever, the single most direct cost knob is the prompt itself — because you pay per input token, on every call, for as long as that feature ships. This page is about spending that token budget like it’s money, because it is.

Why context is never free

It’s tempting to treat the context window as free headroom — “the model can take a million tokens, so I’ll give it everything.” Three separate costs say otherwise.

• Dollars. Every input token is billed every call. A prompt you bloat once is a tax you pay forever.
• Latency. Before the model emits a single output token, it must prefill — run attention over your entire input. More input tokens means more prefill, means a slower first token. Long prompts feel sluggish even when the answer is short.
• Quality. Models attend unevenly across a long context. Facts buried in the middle of a huge prompt get retrieved worse than facts at the start or end — the well-documented “lost in the middle” effect. Past a point, more context makes answers worse, not just costlier.

Bigger context window ≠ use all of it

A 200K or 1M token window is a ceiling, not a target. Filling it costs prefill latency and dollars on every call and can degrade accuracy. Send what the task needs and almost nothing more.

Where the tokens hide

Three repeat offenders, all paid on every call:

[ system prompt ]   ← fixed, re-sent every call
[ few-shot examples ]  ← often the biggest, most overlooked block
[ retrieved context ]  ← can dwarf everything if you over-retrieve
[ the actual question ]  ← usually the smallest part

The user’s actual question is frequently the cheapest line in the request. The fat lives in the scaffolding around it.

Worked example: fat prompt vs lean prompt

Suppose a classification feature runs N = 1,000,000 calls/month. Each output is a short label — say 20 output tokens. The input is where the design choice lives.

Fat version: a 600-token system prompt, eight few-shot examples at ~150 tokens each (1,200 tokens), and the item to classify (~200 tokens).

input_fat = 600 + 1200 + 200 = 2,000 tokens/call

Lean version: tighten the system prompt to 200 tokens, drop to three sharp few-shot examples (450 tokens), same 200-token item.

input_lean = 200 + 450 + 200 = 850 tokens/call

Now scale by N. We’ll price in relative token-units (input = 1 unit, output ≈ 5 units, matching the ~5× output premium):

cost_fat  = (2,000 × 1 + 20 × 5) × 1,000,000 = 2,100,000,000 units
cost_lean = (  850 × 1 + 20 × 5) × 1,000,000 =   950,000,000 units

The lean prompt costs ~55% less — and it also prefills faster, so latency drops too. Nothing about the model changed. You deleted tokens. (Check current per-token pricing to turn units into dollars.)

Techniques, cheapest first

Technique	What it does	Saves
Trim the system prompt	cut throat-clearing, redundancy, dead instructions	input tokens/call
Fewer, better few-shots	3 sharp examples often beat 8 mediocre ones	input tokens/call
Summarize history	replace a long chat log with a running summary	input tokens/turn
Cap output (max_tokens)	hard ceiling on the expensive side	output tokens (~5×)
Stop sequences	end generation the instant the answer is done	output tokens

A note on the last two: because output tokens carry the ~5× premium, capping them is the highest-leverage trim per token removed. Setting max_tokens to a tight-but-sufficient value, and adding stop sequences so the model halts at the natural end of a structured answer, prevents the model from rambling past the useful content on the priciest tokens you buy.

Trimming the conversation, not just the prompt

In a multi-turn chat, the entire history is re-sent every turn — turn 20 pays for turns 1–19 again. Cost grows quadratically with conversation length if you do nothing. Two fixes: summarize older turns into a compact running summary once they’re no longer needed verbatim, or drop turns that no longer matter. (Providers also offer server-side help here — compaction and context editing — but the application-layer instinct is the same: don’t re-send tokens that have stopped earning their place.)

The discipline

Before adding a sentence to a prompt that runs at scale, ask: does this change an answer the model would otherwise get wrong? If not, it’s pure recurring cost. The leanest prompt that hits your quality bar is the correct prompt — every token past that is a bill with no benefit.

It actually happened — “Lost in the Middle” (Liu et al., 2023)

The middle-of-context penalty this page leans on isn’t a hunch — it was measured. In Lost in the Middle: How Language Models Use Long Contexts (Nelson Liu and colleagues, 2023), researchers placed the single relevant document at different positions inside a long context and tracked accuracy. The result was a striking U-shaped curve: models answered best when the needed fact sat at the very beginning or the very end of the context, and noticeably worse when it was buried in the middle — sometimes worse than giving the model no documents at all. The finding held across multiple models and is why “the window is a ceiling, not a target” is more than a cost slogan: padding a prompt with marginal context doesn’t just cost prefill dollars, it can actively bury the one passage that mattered.

The architect’s lens

The leanest lever of all — you delete tokens, and nothing about the model changes:

• Why does it exist? Because every input token is re-billed on every call for as long as the feature ships, so a prompt you bloat once is a tax you pay forever — before you cache, route, or retrieve anything.
• What problem does it solve? Three costs at once: dollars (a fat 2,000-token prompt vs a lean 850-token one is ~55% cheaper at 1M calls/month), prefill latency, and quality (the “lost in the middle” U-shaped accuracy curve).
• What are the trade-offs? Trimming risks cutting context the model actually needs — and capping max_tokens/stop is the highest-leverage trim because output carries the ~5× premium, but cut too tight and you truncate useful content.
• When should I avoid it? Never skip it, but don’t over-trim below the quality bar to save a few tokens — the leanest prompt that still hits the bar is the correct one, not the shortest possible.
• What breaks if I remove it? You treat the context window as free headroom — paying prefill dollars and latency on every call for marginal tokens that can actively bury the one passage that mattered.

Check your understanding

1. Give the three distinct costs of a long prompt, beyond just dollars.
2. What is the “lost in the middle” effect, and why does it mean a bigger context window isn’t always better?
3. In the worked example, why does capping max_tokens save more per token removed than trimming the system prompt?
4. Why does a long multi-turn conversation get expensive even if each individual message is short?
5. You inherit a feature with eight 150-token few-shot examples running at high volume. What’s your first move and why?

Show answers

1. Dollars (every input token re-billed each call), latency (more input means more prefill before the first output token), and quality (models attend unevenly and can lose facts buried in a long context).
2. Models retrieve information placed in the middle of a long context less reliably than information at the start or end; so past a point, adding context degrades accuracy rather than helping — a bigger window can make answers worse and always costs more.
3. Output tokens carry roughly a 5× price premium over input tokens, so removing an output token saves about 5× what removing an input token saves; capping max_tokens trims the expensive side.
4. The full history is re-sent and re-billed on every turn, so cost grows with the square of conversation length even when each message is small.
5. Cut to ~3 sharp, high-signal examples and measure quality — fewer good few-shots often match or beat many mediocre ones, and you remove ~750 input tokens from every call.