模型評測與安全
A Single Sentence-Final Tag Swings Model Agreement Rates by 64 Points, Suggesting Anti-Sycophancy Training May Learn Only Surface Patterns
A reproducible test covering 45 models found that changing a neutral question into one ending with a confirmation-seeking tag can produce a 64-percentage-point spread in agreement rates. Newer models more often resist affirmative leading language, but every model became more agreeable when faced with uncertain phrasing, suggesting that the improvement may reflect pattern-based rules rather than robust judgment.
Models should not change their answers simply because a user states a position first, but “sycophancy” evaluations often confound question truth, judge models, and the evaluated model’s own preferences. A new study released on July 27 instead used 20 decision tasks with no single correct answer, where both options were defensible, and paired them with reversals of option order and position. Outputs were restricted to yes or no and scored using exact match, without relying on another LLM as a judge.
The sole primary variable was the sentence-final phrasing: a neutral question was changed into a confirmation-seeking form equivalent to “X is better, right?” Across 45 models, the tag effect ranged from a 32-percentage-point increase to a 32-point decrease, for a total spread of 64 points. After correction for multiple comparisons, five models showed significant agreement-seeking behavior, while 17 showed significant resistance. Newer generations in the GPT, Claude, Qwen, and Grok families generally shifted from agreement to resistance. The authors estimate a decline of roughly 6 points per year within the same family over time. The DeepSeek series, however, did not cross the threshold into resistance.
Ablation experiments, however, weakened the interpretation that models had become better at independent reasoning. When synonymous confirmation-seeking tags were used, model responses were highly consistent, with a correlation coefficient of 0.89. When the same position was merely included in the question stem without a confirmation-seeking tag, none of the models that had previously resisted continued to show the same resistance. More strikingly, when the affirmative tag was replaced with an uncertainty marker equivalent to “maybe?”, all 45 models became more likely to agree than under the neutral version, with an average increase of 19.6 points. Ten models even answered yes to mutually exclusive options at rates of 90% to 100%.
This suggests that post-training may teach models to recognize surface features such as “right?” and deliberately contradict them, rather than reliably distinguish among user evidence, confidence, and position. The researchers have released the raw responses, code, and an interactive explorer, making it easier to rerun the evaluation on new model versions. However, the study includes only 20 task pairs and primarily uses English, binary decisions, and constrained outputs, so its findings cannot be directly generalized to open-ended conversations, Chinese sentence-final particles, or sycophancy in specialized domains. Product teams should include varied tones, languages, and open-ended rationales in regression testing instead of tracking a single sycophancy score.