模型評測
Asking for JSON Makes 44 Models’ Answers Converge, Suggesting Structured Output May Alter Model Judgment
An experiment spanning 44 language models found that merely adding “respond in JSON” reduced answer diversity, without schema enforcement or constrained decoding. The findings remind developers that structured output is not just a serialization layer—it may also change the answers models actually choose.
A new study reran the “One-Word Census” across 44 language models. Its 31 questions each allow many equally reasonable one-word answers. The experiment only added “respond in JSON only” to the prompt, without enabling JSON Schema, function calling, or constrained decoding. For the most open-ended “pick a word” questions, the share of models selecting the most common answer rose from 41% to 64%, while the number of distinct answers fell from 52 to 36. Mean answer surprisal declined from 1.80 to 1.58 bits.
The researchers also compared formats. JSON and XML both significantly compressed answer distributions, while YAML and CSV did not have the same effect. Wrapping answers in arbitrary brackets even made the distributions slightly more diverse. Enforcing a schema through the decoder did not produce noticeably greater compression than prompting alone. The authors therefore infer that the shift occurs mainly when models recognize a “tool-use or data-interchange register,” rather than because the decoder filters out invalid tokens. Some models that had originally chosen more distinctive answers moved back toward the group mode, suggesting that tool-use post-training may also instill content preferences.
For agents and batch data pipelines, this means model comparisons conducted in chat interfaces may not directly represent production API behavior. If classification, recommendation, brainstorming, or synthetic-data workflows collect results in JSON, formatting requirements may reduce candidate coverage and increase correlated errors across multi-model systems. Engineering teams can treat free text, prompted JSON, and decoder-enforced schemas as separate evaluation conditions, comparing answer distributions and accuracy field by field. However, the study used deliberately designed word-choice questions and has not yet shown that convergence causes a meaningful decline in quality for extraction tasks with a single correct answer, function calling, or complex reasoning tasks.