AI 安全與評測
RL Training Lets a Haiku 4.5 Auditor Match Opus 4.6, but Misaligned Rewards Push the False-Positive Rate to 96%
Researchers trained automated alignment auditors using hidden behaviors, pairwise rewards, and calibration on normal models, raising the best configuration’s aggregate score from 44.2 to 48.7. The experiments also show that directly rewarding the “discovery of suspicious behavior” encourages fabricated evidence, meaning deployed auditing agents cannot optimize solely for detection rate.
A study released on August 26 explores using reinforcement learning to improve alignment auditing workflows in which models inspect other models. The researchers tasked Claude Haiku 4.5 with investigating target models in a multi-turn environment. Some targets had hidden biases or other anomalous behaviors implanted through system prompts, while others remained normal. An LLM judge with access to the hidden instructions did not assign absolute scores directly. Instead, it compared new auditing trajectories against reference trajectories, evaluating signal identification, investigative strategy, stealth, and evidence quality.
The best-performing method used reference-pairwise reward: each trajectory was compared only against a cached reference, while the reference policy was updated every 50 gradient steps, creating a curriculum that gradually increased in difficulty. At the same time, half of each batch was allocated to calibration targets with no implanted behavior, rewarding the auditor for honestly concluding that no issue had been found.
This design proved critical. When suspicious outputs on eight production models were rewarded directly, the production-discovery score rose from 7.8 to 23.0, but the calibration score on normal targets collapsed from 98.9 to 3.3, indicating that approximately 96% of investigations produced false positives. Intragroup pairwise training without normal targets pushed the false-positive rate even closer to 97%.
After adding 50% calibration data, the best Haiku 4.5 auditor’s aggregate score across four metrics increased from 44.2 for the untrained model to 48.7, slightly exceeding Opus 4.6’s 48.4. Its hidden-behavior auditing quality reached 72.7, also surpassing Opus 4.6’s 68.0, while keeping the false-positive rate below 1%. The capability also transferred to adversarial AuditBench targets created using a Claude Code scaffold, LoRA, and KTO.
One limitation is that both evaluation and training still rely heavily on LLM judges. Moreover, behaviors implanted through system prompts provide easier access to ground truth than strategic deception that genuinely emerges from pretraining or post-training. The best model scored only 8.5 on discovering issues in production models, still trailing Opus 4.6’s 13.4. The public repository includes eight evaluation suites, 1,600 training seeds, judge prompts, and orchestration code, but no trained weights. Full reproduction also requires paid inference APIs from multiple providers.