評測與可靠性
After an Agent Completes an Audit and Repair, Its False-Positive Rate in the Next Verification Falls by Up to 11.5 Percentage Points
New research shows that even when the content being verified is identical, prior auditing and repair experience retained in context causes several open models to flag fewer errors. Signal detection analysis found no corresponding improvement in discriminability, suggesting that the agent pipeline changes the verification threshold rather than simply making the model more accurate.
Using the same model first as a reviewer and then asking it to fix problems may seem like a mere agent-workflow orchestration choice. New research, however, indicates that this experience changes the model’s decision threshold in its next review. Researchers first had a model complete an “audit → repair” sequence on a different ProcessBench reasoning trace, then presented it with a bit-identical target problem. Across five semantically similar phrasings, Qwen3.6-27B, Qwen3.6-35B-A3B, and Ministral-3-14B produced lower false-positive rates than with length-matched, non-audit context in all 15/15 combinations, with reductions ranging from 2.8 to 11.5 percentage points.
The experiment examined more than correct traces. The study also used 929 traces labeled as incorrect to estimate detection rates, then applied signal detection theory to separate discriminability `d′` from decision criterion `c`. The criterion shifted toward flagging fewer errors in all 15 combinations, with 13 surviving multiple-comparison correction. By contrast, none of the combinations showed a discernible significant improvement in `d′`. In other words, the models primarily became more lenient rather than better at distinguishing correct from incorrect reasoning. Balanced accuracy did improve at this operating point, but that was because the initial false-positive rate was high; it does not follow that greater leniency would remain beneficial under other error distributions.
To rule out simple effects from context length or answer polarity, the authors added controls including length-matched filler, audit-only context, inert restatement, and cases in which the model actually found and repaired an error. The results also could not be explained as simple repetition of an earlier conclusion. In the cleanest Ministral test, when the preceding audit genuinely found an error, the model became even more lenient afterward—the opposite of the prediction that generally negative information should prompt more error reports. Enabling reasoning mode sharply reduced the absolute false-positive rates of both Qwen models, but the relative threshold shift persisted. The tradeoff was roughly an 18-fold increase in output tokens, while truncation and formatting failures were nonrandom.
The findings have practical implications for code review, automated proof checking, and critic–repair agents: teams should not track only the repair success rate for flagged items. They should also measure the false-positive rate on clean samples, the detection rate on erroneous samples, and the history retained in context. One option is to run the verifier in an independent session or to verify again with a clean context after repair. The study is limited to one reasoning dataset and three open models. Two additional candidate models failed the default sensitivity screening, and one of them showed a significant effect in the opposite direction. For now, therefore, greater leniency cannot be treated as a universal law of LLM verifiers.