代理強化學習
FlashREINFORCE Replaces Same-Prompt Group Sampling in Agentic RL With Asynchronous Single-Trajectory Updates
NVIDIA researchers propose a critic-free, single-rollout training method that lets tool-use trajectories of varying lengths enter the update process as soon as they finish. In small-model experiments, it outperformed the reported GRPO baseline using half as many rollouts, but its stability across environments and model scales remains insufficiently validated.
Long-horizon agent rollouts can stall on tools, simulators, or reasoning chains of varying lengths. Group-relative methods such as GRPO also require multiple trajectories to be generated from the same prompt before rewards can be compared, creating a synchronization barrier. FlashREINFORCE, submitted on September 14, instead samples only one trajectory per prompt and immediately feeds completed samples into asynchronous training. A batch of size B can therefore cover B different prompts, rather than spending the rollout budget on sibling samples for the same prompt. The [project repository](https://github.com/yifanzhang-pro/FlashREINFORCE) provides an Apache 2.0-licensed PyTorch reference loss, a CPU update example, and NVIDIA Molt-based configurations for inference, Python tools, and ALFWorld.
The method stabilizes updates through three components. One-Batch REINFORCE uses the batch mean reward as its baseline, eliminating the need to train a separate critic; token importance sampling corrects for probability differences between the rollout policy and the current policy; and Sequence Trust Region rejects excessively off-policy trajectories using a sequence-level condition. Finally, Sample-Mean Optimization first averages token losses within each trajectory and then averages across the batch, preventing very long failed responses from receiving disproportionately large negative weights merely because they contain more tokens. Each batch is used for only one update and then discarded, which also reduces the bias caused by repeatedly reusing stale samples.
The [paper page](https://www.alphaxiv.org/abs/2609.flashreinforce-asynchronous-rl-agentic-models) reports that Qwen2.5-Math-1.5B achieved an average score of 38.0% across five mathematics benchmarks, 1.7 percentage points above the reported GRPO baseline, while using 256,000 rollouts versus 512,000. DeepSeek-R1-Distill-Qwen-1.5B continued training for 6,000 steps with a policy lag of approximately four updates. On ALFWorld, Qwen2.5-7B-Instruct achieved seen/unseen success rates of 98.3%/96.5% and avoided the collapse observed in comparison runs, where the model stopped calling tools.
However, importance sampling corrects only actions along observed histories and cannot recover the state distribution altered by an older policy; the sequence-level trust metric is also merely a KL proxy evaluated on sampled actions. Current evaluations focus on mathematics, Python tools, and ALFWorld, with no testing in more complex environments such as OSWorld. The 30B MoE experiment mainly demonstrates training stability rather than a comprehensive quality advantage. Engineering teams should also verify whether the public launcher can reproduce the paper’s curves instead of treating the included CPU toy example as a complete reproduction.