AI 評測與代理可靠性
Study Reveals Referential Drift in Multi-Model Review Pipelines: Validators May Misclassify Correct Rollbacks as Errors
New experiments show that relative terms such as `previous` in “draft–validate–revise” pipelines may change referents as model roles shift. Increasing reasoning effort can dramatically improve some models, but model selection matters more than simply adding test-time compute.
Multi-model “draft–validate–revise” workflows are often considered more reliable than single-pass generation: the first model proposes an answer, the second looks for errors, and the third adjudicates and corrects it. However, a new study finds that the pipeline itself can introduce “referential drift.” For example, deployment instructions may list both a “Previous Version” and a “Current Version.” The drafting model interprets the rollback target as the current version that was running before the upgrade, while the validator—following the field labels—insists that the system should revert to the older previous version. Both linguistic interpretations are reasonable, but only one matches the actual sequence of operations.
The researchers turned these conflicts into 10 base cases with three conditions each, producing 30 synthetic stimuli, and tested six models across 21 reasoning settings. Balanced accuracy ranged from 0.156 to nearly perfect. GPT‑5.2 improved from 0.156 with reasoning disabled to 0.942 at the highest reasoning effort. Gemini 3 Pro scored above 0.94 across all settings; even at a low reasoning setting, it outperformed GPT‑5.2 at its highest reasoning setting while costing roughly 5% as much per run. Error analysis showed that models were often swayed by field labels and the validator’s incorrect classification. Even after articulating the correct operational logic, they still accepted the wrong verdict.
The immediate engineering lesson is that content passed between stages should not retain perspective-dependent terms such as “previous,” “current,” or “original response.” These should be replaced with version numbers, timestamps, object IDs, and explicit state transitions. Validators should also identify the objects they are referring to, rather than outputting only error labels. The researchers have released the stimuli, aggregate results, and code for reproducing the charts. However, the sample covers only 10 scenarios, and some raw provider responses and reasoning traces are not included in the repository, so the findings cannot yet establish that all agentic workflows fail at the same rates.