代理訓練與檢索
CIPO Assigns Step Rewards by Masking Retrieved Evidence, Raising Qwen2.5-7B’s Average F1 Across Seven QA Benchmarks to 0.504
CIPO compares an agent’s action probabilities with and without access to the latest retrieval results, directly rewarding next steps influenced by external evidence. In the authors’ experiments, it outperformed the strongest baseline by 4.7 F1 points, but the additional scoring increases training costs, and the method has so far been validated on only two Qwen2.5 models.
Even when search agents call retrieval tools, they may first guess an answer from the model’s memory and then use the search results merely as confirmation. Rewards based only on the final answer or reasoning progress cannot distinguish such “conclusion-first” trajectories. CIPO (Contextual Information Policy Optimization), submitted on August 6, instead assigns per-turn rewards using the Evidence-Access Log-Likelihood Ratio. For each reasoning or search action following retrieval, it separately calculates the generation probability when the evidence is visible and when it is masked. A larger difference in log-likelihood indicates that the action depends more heavily on the newly retrieved evidence.
This signal requires neither human process annotations nor a separate reward model, although training still requires an additional scoring pass with the evidence masked. Using verl and eight A100 80GB GPUs, the team trained Qwen2.5-3B-Instruct and Qwen2.5-7B-Instruct with batches of 32 questions, 16 trajectories per question, and a maximum of five interaction turns. Across seven question-answering benchmarks, the 7B model achieved a macro-average F1 of 0.504, compared with IGPO’s 0.457. The 3B model likewise improved from the strongest competitor’s 0.409 to 0.456. The authors’ diagnostics show that, for the 7B model, the proportion of cases in which evidence supporting the answer was used rose from 21.9% with terminal-only rewards to 60.7%, while the use of irrelevant evidence fell from 11.3% to 9.1%.
Ablation experiments also reveal a limitation: rewarding evidence sensitivity alone raises the irrelevant-evidence usage rate to 28.9%, so it must be jointly optimized with answer correctness. The additional scoring increased the 7B model’s per-step training time from 417.2 seconds to 436.5 seconds, but introduced no extra inference cost. From an engineering perspective, it remains important to determine whether the results generalize across models, real-world web search, interactions longer than five turns, and multi-tool tasks. The paper also does not currently link to a CIPO-specific code repository.