Ask an inference engine like vLLM or SGLang for structured output, either a JSON schema or a plain JSON mode, and a technique called constrained decoding does the work. The more familiar way to get JSON out of a model is to ask for it in the prompt, parse whatever comes back, and try again when parsing fails. Constrained decoding works earlier than that, while the tokens are still being picked.

What happens at each step

At every step a model gives each token in its vocabulary a score, called a logit, and the decoder turns those scores into probabilities and samples one token. Nothing in that step checks for JSON. When the prompt is the only thing asking for JSON, a trailing comma or a sentence of prose can still come out. A constrained decoder adds one thing before the sampling. It holds a grammar, either a general one for JSON or one compiled from your schema, and it keeps track of where the output currently is inside that grammar. After {"name":, for example, a value has to come next, and a closing brace is not allowed yet.

From that position the decoder marks every token that could legally come next, and it sets the logit of every other token to negative infinity. The softmax turns negative infinity into a probability of exactly zero, and a token with zero probability is never sampled, however high the model had scored it. The tokens that are still legal keep their scores untouched.

Once a token has been sampled, it is fed back into the grammar, the position moves forward past it, and the next step gets a fresh mask. Tokens are usually a few characters long, while the grammar is written one character at a time. A token passes only if all of its characters fit where the output is now, and true can go through as a single token where a value is expected.

This is the approach xgrammar takes. Its README currently lists the library as “the default structured generation backend for most LLM inference engines”, vLLM, SGLang and TensorRT-LLM among them, and describes the result as “100% structural correctness of the output”. That holds for a reply that finishes; one cut off by the token limit still stops partway through an object.

Next, running it on twelve tokens

Here is xgrammar with its built-in JSON grammar and a vocabulary of twelve tokens, writing {"name": 42} one token at a time:

import torch
import xgrammar as xgr

# A deliberately tiny vocabulary, so the whole mask is printable.
vocab = ['{', '}', '"', 'name', 'age', ':', ',', '42', 'true', ' ', 'hello', 'xyz']
info = xgr.TokenizerInfo(vocab, vocab_size=len(vocab))
compiler = xgr.GrammarCompiler(info)
matcher = xgr.GrammarMatcher(compiler.compile_grammar(xgr.Grammar.builtin_json_grammar()))

bitmask = xgr.allocate_token_bitmask(1, len(vocab))

def allowed_now():
    matcher.fill_next_token_bitmask(bitmask)
    logits = torch.zeros(len(vocab))          # the model's scores; all equal here
    xgr.apply_token_bitmask_inplace(logits, bitmask)
    survivors = [vocab[i] for i in range(len(vocab)) if logits[i] != float('-inf')]
    return survivors, logits

survivors, logits = allowed_now()
print("start:  ", survivors)
print("logits: ", logits.tolist())
for token in ['{', '"', 'name', '"', ':', '42']:
    matcher.accept_string(token)
    print(f"after {token!r}:", allowed_now()[0])

start:   ['{']
logits:  [0.0, -inf, -inf, -inf, -inf, -inf, -inf, -inf, -inf, -inf, -inf, -inf]
after '{': ['}', '"', ' ']
after '"': ['{', '}', '"', 'name', 'age', ':', ',', '42', 'true', ' ', 'hello', 'xyz']
after 'name': ['{', '}', '"', 'name', 'age', ':', ',', '42', 'true', ' ', 'hello', 'xyz']
after '"': [':', ' ']
after ':': ['{', '"', '42', 'true', ' ']
after '42': ['}', ',', '42', ' ']

At the start only { is left, and the logits line shows how: one 0.0 and negative infinity for all the other choices, although every score started out equal, which means the brace there comes from the grammar alone.

What about inside a string?

But look at the two lines after the opening quote of the key. Every token in the vocabulary is allowed again, including {, : and xyz (a little counterintuitive at first), because inside a string they are only characters, and all the grammar checks there is that the string stays a valid string.

Running the same twelve tokens against a one-field schema, an object with a required string name, leaves name as the only token allowed after {", and after the colon 42 is no longer allowed, only a quote or a space. The run above used the general JSON grammar, and there even the key was free. Once the opening quote of the value arrives, though, everything is allowed again.

So among the tokens that survive, the choice still comes from the model’s own scores, which the mask leaves untouched. When those scores favour a wrong name or a made-up number, nothing in the grammar stops them. The JSON still parses, but from its shape alone you can’t tell whether the values in it can be trusted.

This was put together from the xgrammar README and the two runs above, the general one and the schema one, both done with Python 3.12, xgrammar 0.2.8 and CPU-only torch. Every score in them was equal. They show the mask itself; how much the mask changes what a real model would have gone on to write is a separate question. Models and inference engines move fast, and each engine handles this a little differently, so the details here may date quickly; take it as one simple look at how it works.