Structured Output & Constrained Decoding

Routing chose the model; this last lever shapes what you demand back from it. When your code needs a machine-readable answer — JSON for a database, a label for a switch statement, arguments for a function — a free-text reply that’s almost valid is worse than useless: you parse it, it fails, you call the model again. That retry is a second full call. The throughline lands hard here: a wasted re-ask costs the full input tokens, the full output tokens, and a full round-trip of latency. Structure is how you stop paying it.

Why structure is an efficiency lever, not just a correctness one

Free text:   call ─▶ parse ─▶ FAILS ─▶ call again ─▶ parse ─▶ ... (each retry = full cost)
Constrained: call ─▶ guaranteed-valid output ─▶ parse ─▶ done (one call)

Two savings, both direct:

• No retries. Every retry is a duplicate call — input re-billed, output re-billed, latency doubled. Eliminating a 10% retry rate is a ~10% cost cut on that path, for free.
• Shorter outputs. A schema forbids the preamble. No “Sure! Here’s the JSON you asked for:” wrapping the payload. Since output tokens carry the ~5× premium, cutting the chatter off the front of every reply is real money.

So structure improves reliability (it parses) and efficiency (no re-asks, shorter replies) at the same time. That’s why it earns a place among the cost levers, not just the quality ones.

Constrained decoding: how the guarantee works

A language model picks each token from a probability distribution over the whole vocabulary. Constrained decoding intervenes at that step: given the schema and the tokens generated so far, it masks out every token that would make the output invalid, so the model can only sample from tokens that keep it on a valid path.

generating a JSON object, just emitted  {"age":
   valid next tokens:  digits, maybe a minus sign
   masked (forbidden): letters, quotes, braces
   └─▶ the model literally cannot emit "twenty" — only a number can come next

Because invalid tokens are never sampled, the output is guaranteed to match the schema. There’s no “validate then maybe retry” — validity is enforced during generation. That’s the mechanism behind a provider’s JSON mode and grammar-constrained output.

The approaches

Approach	What you give	What you get
JSON mode	”give me JSON”	syntactically valid JSON (not necessarily your shape)
JSON schema	a full schema (fields, types, enums)	output that conforms to your exact shape
Tool / function calling	tool definitions with input schemas	the model emits validated arguments for the tool
Grammar	a formal grammar (e.g. for a DSL)	output matching an arbitrary structure, not just JSON

On the Messages API, two features cover most needs. Structured outputs constrain the response itself to a JSON schema (output_config.format), and strict: true on a tool definition guarantees the tool’s input validates exactly. Tool use is the natural fit when the model’s job is to call something: you describe the tool’s input schema, and the model returns conforming arguments instead of prose you have to parse out of a paragraph.

Enum the labels

For classification, don’t ask for a free-text label and hope. Give a schema with an enum of the exact allowed values. The model can then emit only one of your labels — no “Positive!” vs “positive” vs “The sentiment is positive” normalization, no retries on a typo’d category.

Worked example: the retry tax

A pipeline extracts a JSON record. Input ~500 tokens, output ~150 tokens. Free-form prompting yields valid JSON 90% of the time; the other 10% need one retry (a full second call). Cost per attempt, in relative units (output ≈ 5×):

per_call = 500 × 1 + 150 × 5 = 500 + 750 = 1,250 units

Free-form:    1,250 × (1 + 0.10 retries) = 1,375 units/record  (avg)
Constrained:  1,250 × 1                   = 1,250 units/record

Constrained decoding saves the ~10% retry tax outright — and that’s before counting the shorter outputs (no preamble) and the latency you don’t spend on a second round-trip. At a 30% failure rate the gap widens to 1,625 vs 1,250 — a ~23% saving. The worse your raw reliability, the more structure pays.

The trade-offs

Structure isn’t free of cost:

• Rigidity. A schema that’s too tight can force the model into a box that doesn’t fit a genuine edge case (where does “unknown” go?). Design schemas with explicit null/unknown options for the cases reality will throw at you.
• First-call schema compilation. A brand-new schema can carry a one-time setup latency before it’s cached; steady-state calls are fast.
• Expressiveness limits. Constrained-output schemas typically support core JSON-schema features but not every exotic constraint, so keep schemas within the supported subset.

None of these outweigh the win for machine-consumed output. The rule of thumb: if your code parses the model’s answer, constrain it. Free text is for humans; structure is for programs — and structure is cheaper.

Under the hood — what “schema compilation” actually does

Masking the logits at every step sounds expensive: re-checking a JSON schema against the whole vocabulary on each token could cost more than the generation it guards. The trick that makes constrained decoding nearly free is precompiling the schema (or regex, or grammar) into a finite-state machine before generation starts. Each state of that machine stores exactly which vocabulary tokens are legal next, as a precomputed mask. At generation time the decoder just looks up the current state’s mask and applies it — an O(1) operation per token, not a fresh grammar evaluation. This is the insight behind libraries like outlines and guidance (and the “efficient guided generation” work of 2023): pay a one-time compilation cost up front — that’s the first-call latency this page warns about — and every constrained token after it is essentially as fast as an unconstrained one. The guarantee of valid output stops being a tax you pay per token and becomes a fixed setup cost you amortize away.

The architect’s lens

Structure earns its place among the cost levers, not just the correctness ones:

• Why does it exist? Because when your code parses the model’s answer, an almost-valid free-text reply is worse than useless — you re-ask, and that retry costs full input + full output + a full round-trip.
• What problem does it solve? The retry tax and bloated replies: constrained decoding masks every token that would break the schema, guaranteeing valid output in one call (eliminating a 10% retry rate is a ~10% cost cut) and forbidding the “Sure! Here’s the JSON:” preamble.
• What are the trade-offs? Rigidity (a too-tight schema can’t represent a genuine edge case — design in explicit null/unknown), a one-time schema-compilation latency on the first call, and a limited expressiveness subset.
• When should I avoid it? For human-facing free text where there’s nothing to parse — structure is for programs; over-constraining open-ended generation boxes the model in for no benefit.
• What breaks if I remove it? Parsing reverts to “validate then maybe retry,” so failures recur as full duplicate calls (at a 30% failure rate the gap is 1,625 vs 1,250 units), and classification labels drift into normalization headaches.

Check your understanding

1. Why is a structured-output guarantee an efficiency lever and not only a correctness one?
2. Mechanically, how does constrained decoding guarantee valid output instead of validating after the fact?
3. In the worked example, why does a 10% free-form failure rate translate into roughly a 10% cost penalty per record?
4. When is tool/function calling the natural way to get structured output, versus a JSON-schema response?
5. Give one trade-off of constraining output and how you’d design around it.

Show answers

1. Because invalid output forces a retry, and every retry is a full duplicate call — input re-billed, output re-billed, latency doubled. Guaranteeing valid output removes those retries (and a schema also forbids preamble, shortening the expensive output), so it cuts cost as well as fixing correctness.
2. It masks out, at each generation step, every vocabulary token that would make the output violate the schema, so the model can only sample tokens that keep it on a valid path — validity is enforced during generation, so there’s nothing to fail and retry.
3. Each retry is a full second call costing the same ~1,250 units; if 10% of records need one retry, the average cost is 1,250 × 1.10 ≈ 1,375 units — about 10% more than the constrained path’s single call.
4. Tool/function calling fits when the model’s job is to invoke something — you define the tool’s input schema and get validated arguments back. A JSON-schema response fits when you just need a structured data object back, not a call to a tool.
5. Rigidity — a too-tight schema can’t represent a genuine edge case (e.g., an unknown value). Design around it by including explicit null/unknown/other options in the schema for the cases reality will produce.