What happens at each step

At every step a model gives each token in its vocabulary a score, called a logit, and the decoder turns those scores into probabilities and samples one token. Nothing in that step checks for JSON. When the prompt is the only thing asking for JSON, a trailing comma or a sentence of prose can still come out. A constrained decoder adds one thing before the sampling. It holds a grammar, either a general one for JSON or one compiled from your schema, and it keeps track of where the output currently is inside that grammar. After {"name":, for example, a value has to come next, and a closing brace is not allowed yet.

From that position the decoder marks every token that could legally come next, and it sets the logit of every other token to negative infinity. The softmax turns negative infinity into a probability of exactly zero, and a token with zero probability is never sampled, however high the model had scored it. The tokens that are still legal keep their scores untouched.

Once a token has been sampled, it is fed back into the grammar, the position moves forward past it, and the next step gets a fresh mask. Tokens are usually a few characters long, while the grammar is written one character at a time. A token passes only if all of its characters fit where the output is now, and true can go through as a single token where a value is expected.

This is the approach xgrammar takes. Its README currently lists the library as “the default structured generation backend for most LLM inference engines”, vLLM, SGLang and TensorRT-LLM among them, and describes the result as “100% structural correctness of the output”. That holds for a reply that finishes; one cut off by the token limit still stops partway through an object.

Next, running it on twelve tokens

Here is xgrammar with its built-in JSON grammar and a vocabulary of twelve tokens, writing {"name": 42} one token at a time:

import torch
import xgrammar as xgr

# A deliberately tiny vocabulary, so the whole mask is printable.
vocab = ['{', '}', '"', 'name', 'age', ':', ',', '42', 'true', ' ', 'hello', 'xyz']
info = xgr.TokenizerInfo(vocab, vocab_size=len(vocab))
compiler = xgr.GrammarCompiler(info)
matcher = xgr.GrammarMatcher(compiler.compile_grammar(xgr.Grammar.builtin_json_grammar()))

bitmask = xgr.allocate_token_bitmask(1, len(vocab))

def allowed_now():
    matcher.fill_next_token_bitmask(bitmask)
    logits = torch.zeros(len(vocab))          # the model's scores; all equal here
    xgr.apply_token_bitmask_inplace(logits, bitmask)
    survivors = [vocab[i] for i in range(len(vocab)) if logits[i] != float('-inf')]
    return survivors, logits

survivors, logits = allowed_now()
print("start:  ", survivors)
print("logits: ", logits.tolist())
for token in ['{', '"', 'name', '"', ':', '42']:
    matcher.accept_string(token)
    print(f"after {token!r}:", allowed_now()[0])

start:   ['{']
logits:  [0.0, -inf, -inf, -inf, -inf, -inf, -inf, -inf, -inf, -inf, -inf, -inf]
after '{': ['}', '"', ' ']
after '"': ['{', '}', '"', 'name', 'age', ':', ',', '42', 'true', ' ', 'hello', 'xyz']
after 'name': ['{', '}', '"', 'name', 'age', ':', ',', '42', 'true', ' ', 'hello', 'xyz']
after '"': [':', ' ']
after ':': ['{', '"', '42', 'true', ' ']
after '42': ['}', ',', '42', ' ']

At the start only { is left, and the logits line shows how: one 0.0 and negative infinity for all the other choices, although every score started out equal, which means the brace there comes from the grammar alone.

What about inside a string?

But look at the two lines after the opening quote of the key. Every token in the vocabulary is allowed again, including {, : and xyz (a little counterintuitive at first), because inside a string they are only characters, and all the grammar checks there is that the string stays a valid string.

Running the same twelve tokens against a one-field schema, an object with a required string name, leaves name as the only token allowed after {", and after the colon 42 is no longer allowed, only a quote or a space. The run above used the general JSON grammar, and there even the key was free. Once the opening quote of the value arrives, though, everything is allowed again.

So among the tokens that survive, the choice still comes from the model’s own scores, which the mask leaves untouched. When those scores favour a wrong name or a made-up number, nothing in the grammar stops them. The JSON still parses, but from its shape alone you can’t tell whether the values in it can be trusted.

This was put together from the xgrammar README and the two runs above, the general one and the schema one, both done with Python 3.12, xgrammar 0.2.8 and CPU-only torch. Every score in them was equal. They show the mask itself; how much the mask changes what a real model would have gone on to write is a separate question. Models and inference engines move fast, and each engine handles this a little differently, so the details here may date quickly; take it as one simple look at how it works.

模型產生文字的方式，是每一步替詞彙表裡所有的 token 計算分數，再從裡面挑出下一個 token。constrained decoding 則是在選擇挑取之前先把「選了就會讓格式壞掉」的 token 分數改成負無限大，這些 token 就不可能被選到，模型就被限縮在剩下的選項中挑選。

參考之前寫的相關文章：大語言模型 LLM：其實做的事情比你想像中更單純

剩下幾個 token

xgrammar 是專門做這件事的開源函式庫，而且可以拿 tokenizer 來跑，不需要載入模型，因為要劃掉哪些 token 只跟 schema、詞彙表和目前進度到哪裡有關，跟模型的權重無關(感覺很反直覺)。下面用 Qwen2.5-0.5B-Instruct 的 tokenizer 來測試。

import json
import torch
import xgrammar as xgr
from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-0.5B-Instruct")
info = xgr.TokenizerInfo.from_huggingface(tok)
schema = {
    "type": "object",
    "properties": {"name": {"type": "string"}, "age": {"type": "integer"}},
    "required": ["name", "age"],
    "additionalProperties": False,
}
compiled = xgr.GrammarCompiler(info).compile_json_schema(json.dumps(schema), any_whitespace=False)
matcher = xgr.GrammarMatcher(compiled)
bitmask = xgr.allocate_token_bitmask(1, info.vocab_size)

print("vocab_size", info.vocab_size)
for t in tok.encode('{"name": "Ada", "age": 36}', add_special_tokens=False):
    matcher.fill_next_token_bitmask(bitmask)          # 算出還能用的 token
    logits = torch.zeros(info.vocab_size)
    xgr.apply_token_bitmask_inplace(logits, bitmask)  # 不能用的分數改成 -inf
    print(f"{tok.decode([t])!r:8} {int(torch.isfinite(logits).sum()):>7}")
    matcher.accept_token(t)

環境是 Python 3.12.13、xgrammar 0.2.8、torch 2.14.1、transformers 5.19.0。左邊是送進去的 token，右邊是當下還可以使用的 token 數量：

vocab_size 151665
'{"'           2
'name'         4
'":'           2
' "'         121
'Ada'     147057
'",'      147057
' "'           2
'age'          3
'":'           2
' '            2
'3'           11
'6'           11
'}'           11

一開始只能從兩個 token 裡選，{ 或是 {"，詞彙表裡其他選擇都被無視，因為 JSON 物件只能用左大括號開頭。接下來是 key，xgrammar 會照 schema 裡欄位寫的順序產生，我們看到第一個是 name，所以剩下的只有 n、na、nam、name 這四種組合；拼寫處理完之後是 " 或 ":，此時又回到兩個選擇。可以看到在這一階段，模型其實沒有什麼需要決定的，也沒有甚麼可選的，因為不管怎麼走，最後都會是 {"name":。

但接下來到了 name 的值，可選的數字跳到 147,057 個，瞬間變多了，也就代表詞彙表裡大部分的 token 都可以用。而被劃掉的四千多個裡面，其中多數是含有換行或 tab 的 token。也就是說，這裡要填 Ada、Bob 還是一整句中文，都是模型自行挑選的，也不會被劃掉。後面 age 的位置，能用的只剩 0 到 9 的數字和負號，但到底要寫 36 還是 360，對規則來說都一樣。所以仔細看看可以發現，schema 負責規範的是格式，括號、引號、欄位名稱和型別。

schema 和 prompt

那模型怎麼知道 name 要填名字、age 要填年紀？從剛剛過程來看，處理的都是劃掉不合格式的 token，並沒有告訴模型每個欄位是什麼意思。llama.cpp 的 grammar 文件寫到，JSON schema 只用來限制輸出，不會放進 prompt，模型本身看不到 schema；但工具呼叫（tool calling）則是例外，schema 會被放進 prompt。

因此在用這類功能的時候，還是記得要在 prompt 裡把每個欄位的意思寫清楚，像是 age 是實際年齡、單位是年，不然模型也會照格式輸出整數，只是這個數字你也不知道可不可信。至於其他服務會不會另外把 schema 放進 prompt，當成一併處理的資訊，就要看各自的文件以及其發展。

不過還是說明現今的AI模型發展很快，每家廠商的做法和服務都不一樣，可能這個資訊很快就會過時了，就當作一個簡易分享囉。