A depressing amount of production LLM code is still regex over model prose, with a retry when the parse fails and a metric nobody looks at counting how often that happens. Constrained decoding removed the need for all of it: the decoder is restricted at each step to tokens that can still produce a valid document, so schema conformance stops being a probability and becomes a property.
That changes the engineering posture. The interesting question is no longer 'did it return JSON' but 'is the schema shaped so the model gives good answers', which turns out to be a design skill with real leverage.
The schema is a prompt the model cannot ignore
Field names, enums and descriptions are read on every generation, and they constrain behaviour more reliably than instructions in the system prompt. Replacing a free-text status field with an enum of five values does not only guarantee a valid value — it measurably improves the choice, because the model is selecting from a set rather than inventing a label.
The highest-value trick is ordering. Fields are generated in sequence, so a field placed before the answer becomes reasoning the later fields are conditioned on. Putting evidence before verdict is a small change that consistently improves the verdict.
from pydantic import BaseModel, Field
from typing import Literal
class TicketTriage(BaseModel):
# Ordered deliberately: evidence first, decision second.
evidence: list[str] = Field(
description='Direct quotes from the ticket supporting the classification',
min_length=1, max_length=3)
category: Literal['billing', 'outage', 'access', 'feature_request', 'other']
severity: Literal['low', 'medium', 'high', 'critical']
# An explicit escape hatch beats a confident guess on an ambiguous ticket.
needs_human: bool = Field(description='True if the ticket is ambiguous or out of scope')
resp = client.messages.create(
model=MODEL,
messages=[{'role': 'user', 'content': ticket}],
response_format={'type': 'json_schema', 'schema': TicketTriage.model_json_schema()},
)
triage = TicketTriage.model_validate_json(resp.content) # cannot fail on shapeWhat still goes wrong
Schema validity is not semantic validity. The model will return a perfectly typed reference to an invoice number that does not exist, a date in the wrong century, or a set of fields that are individually valid and collectively contradictory. Cross-field and referential checks remain your responsibility, and they belong in code, not in the prompt.
- Always provide an explicit uncertainty path — a needs_human flag or a nullable answer. Without one, a constrained model must pick something, and it will.
- Keep schemas shallow. Deep nesting degrades quality and makes failures harder to localise.
- Describe fields in the schema rather than restating them in the prompt; the schema is closer to the decoding.
- Validate references against your own data after parsing. Type-correct and factually wrong is the new failure mode.
- Watch for truncation: a cut-off generation can still be schema-valid and semantically empty.
Guaranteed JSON solved the parsing problem. It did not solve the being-right problem, and conflating the two is how bad data gets into good pipelines.
One caution worth stating: an over-constrained schema can hurt. Force a single-label classification onto genuinely multi-topic input and you get an arbitrary choice with no signal that it was arbitrary. The schema should be able to express the real world, including the parts of it that are ambiguous — otherwise you have not removed uncertainty, only the evidence of it.