In a chat window, a language model can answer however it likes. In a business process it cannot. When its output feeds another system, such as a tool call, a CRM record or an invoice line, every brace, quote, field name and allowed value is fixed in advance, and one missing quotation mark breaks the whole object. So how do you make a model that writes one token at a time always return valid JSON?
Why prompting and retries are not enough
The obvious first step is to ask for JSON in the prompt. That makes the right structure more likely, but it cannot make every wrong continuation impossible. The usual defence is a pipeline: generate, parse, validate against a schema, and retry or repair on failure. Take a simple weather lookup to see its limits. One attempt drops a quote and fails the parser. Another is valid JSON but writes units instead of unit, so it fails the schema. Both are caught, but only after the tokens were generated and paid for. Repair can fix punctuation, but it becomes guesswork once the mistake is bigger than a comma.
Constrained decoding: block illegal tokens before they are chosen
At every step the model scores thousands of candidate next tokens, and a sampler picks one. Constrained (grammar-based) decoding places a filter between those two steps. Any token that would make a valid completion impossible is masked: its probability is set to zero. The sampler can only choose among the tokens that survive. Legal tokens keep their relative scores; the mask does not make them equally likely and does not change the model’s weights.
The loop is mask, sample, advance. After each token the constraint moves on, because what is legal next depends on everything generated so far. The model is not even allowed to stop early: the end-of-output token is masked until the object is complete. The same idea works for other formal languages, such as SQL, XML or program syntax, not just JSON.
The hard part: tokens are not characters
A model token can contain a quote, a comma, part of a field name, or several JSON boundaries at once. Suppose the output so far is {"city": "Paris", "unit": " and the only allowed values are C or F. The tokens C, C" and even C"} all keep a valid completion possible. Celsius starts with the right letter but cannot be completed validly, so it is masked, however highly the model scored it. A reliable constraint therefore tests the whole token against the grammar, not just its first character.
Simple choices such as C-or-F can be tracked with a small finite-state machine. Nested objects and arrays need parser state that remembers every bracket still to be closed. In practice an engine compiles the supported part of the JSON Schema, together with the model’s tokenizer, into a recognizer that can answer “which tokens are legal now?” quickly at every step.
How well does it work?
Three questions are easy to blur, and deserve separate answers:
- Validity. When OpenAI introduced strict structured outputs in 2024, it reported that its model matched the supplied JSON schemas in 100% of its internal tests, against under 40% for an earlier model relying on instructions alone.
- Coverage. Engines enforce different subsets of JSON Schema. The JSONSchemaBench study (2025) found marked differences in which schema features the popular engines support.
- Speed. Overhead depends on the engine, hardware, schema and caching. It can be negligible or significant, so measure your own setup.
What it guarantees, and what it does not
Constrained decoding guarantees form: every emitted token kept a valid completion possible, and output ended only when the structure was complete. It does not guarantee:
- Meaning. A call for Paris is perfectly valid even if the user asked about Prague.
- Completion. A token limit, a cancellation or a dropped connection can still cut the output short.
- Permission. A well-formed request is not an authorised one.
Those remain separate layers: semantic checks against the request, permission gates with least privilege, and validation of the effect in the downstream system. In short: the decoder can guarantee the brackets, not the weather, and not the permission to fetch it.
Why this matters for AI in business processes
Assistants that trigger actions and models that automate transactions and workflows, part of our AI Model Development service, live or die by machine-readable output. Constrained decoding removes a whole class of failures, such as broken JSON, missing fields and invented enum values, before they happen, instead of catching them with retries. It works best alongside:
- schemas kept deliberately simple and within what your engine fully supports;
- business-rule checks on the values themselves;
- explicit permissions on every action a model can trigger.
Structured, predictable output also combines well with speculative decoding for speed; see serving LLMs on your own GPUs. If you are planning an AI application that has to talk to your systems, see our AI Model Development and AI Model Training services, or talk to our experts.