AgentOPSD Assigns Agent Rewards Through Recursive Belief Updates; Qwen2.5-7B Reaches 89.1% on ALFWorld

Long-horizon agents typically receive a success or failure signal only at the end of a task. Methods such as GRPO broadcast a trajectory-level advantage to every turn, potentially rewarding critical actions, irrelevant steps, and even mistakes made before recovery. AgentOPSD reframes this credit assignment problem as a recursive belief-update process without training a separate critic.

During training, the system first calculates the token-level log-probability difference between a “teacher” with access to additional skill information and the original agent, then aggregates those differences into turn-level evidence. This evidence updates a Bayesian belief about the trajectory’s eventual success in log-odds form. The change in belief between adjacent turns—after its sign is calibrated using the final verification outcome—is then used to reshape the original advantage. The method requires no additional rollouts and introduces no separate distillation loss; the signal takes effect solely through the modified policy objective.

The authors evaluated Qwen2.5-3B and Qwen2.5-7B on ALFWorld, WebShop, and seven Search-QA datasets. The 7B model achieved an average success rate of 89.1% on ALFWorld, outperforming GRPO and several self-distillation baselines in the same experiments. For each additional interaction turn, its success rate declined by 0.54 percentage points in the regression analysis, compared with 2.91 points for GRPO and 3.59 points for RLSD. In ablation experiments, replacing recursive correction with isolated local differences reduced the success rate from 89.1% to 82.8%. Removing outcome-based directionality or the initial success-rate prior lowered it to 80.5% and 78.9%, respectively.

This design is particularly relevant to training search, browsing, and embodied agents: the terminal verifier can remain simple, while the training process does not need to assume that every step is equally important. However, the experiments cover only two Qwen2.5 model sizes and simulated environments, and the teacher signal depends on privileged skill information available during training. More importantly, the GitHub repository currently contains only documentation and labels the code as “coming soon.” Until the complete code, data-processing pipeline, and multiple independent reruns become available, the 89.1% figure should be treated as an author-reported result rather than an externally reproduced record.