代理訓練
GRSD Assigns Rewards by Comparing an Agent’s Own Success and Failure Trajectories; ALFWorld Unseen Task Success Rate Reaches 83.6%
Group-Reflective Self-Distillation no longer extracts fixed skills from an external teacher. Instead, it compares successful and failed rollouts from the same policy and converts their differences into turn-level training signals. It delivers overall improvements across three agent environments, but increases training time by 17% over GRPO and still requires an external model to score reflection text.
When training agents with verifiable rewards, methods such as GRPO typically know only whether an entire trajectory ultimately succeeded or failed. This terminal reward cannot identify which step actually advanced the task, and it assigns the same advantage to incidental actions and critical decisions. Existing self-distillation methods add external skill libraries or use a stronger model to analyze individual trajectories, but the resulting recommendations may exceed the current policy’s capabilities or overfit to a particular path.
Group-Reflective Self-Distillation (GRSD), submitted on July 30, instead derives guidance from the policy’s own on-policy rollouts. For each problem, it generates eight trajectories, has a verifier label their outcomes, and then asks the policy to reflect separately on successful and failed cases. A stop-gradient snapshot of the policy compares the two sets of reflections and synthesizes group-level privileged guidance. The self-teacher then uses this guidance to reassess each turn and adjust the outcome-based advantage without reversing the overall learning direction determined by the verifier. Reflection tokens are trained with a separate loss, so task tokens and reflection tokens receive different forms of supervision.
The researchers evaluated Qwen3-1.7B, 4B, and 8B on ALFWorld, seven search-based question-answering datasets, and WebShop. Compared with the stronger Skill-SD or SDAR baseline for each backbone, GRSD improved the overall ALFWorld success rate by an average of 4.2 percentage points and the WebShop success rate by 5.5 points. On the ALFWorld unseen split, it achieved an average success rate of 83.6%, outperforming GRPO and SDAR by 8.2 and 6.0 points, respectively. The gains were not consistent across every subtask, but the aggregate results were more stable across the three model sizes.
The tradeoff appears during training: each prompt requires eight reflections, one group summary, and calls to an external judge to score the reflections. In the authors’ ALFWorld experiment using Qwen3-1.7B on eight H100 GPUs, each update took 333.4 seconds, compared with 284.1 seconds for GRPO—a relative cost of 1.17×. These additional steps are not needed during deployment-time inference, but the code and checkpoints have not yet been released. Reliance on an external judge also means the results may be affected by biases in the scoring model. Future work should compare variants that do not use an external judge and test whether GRSD can still assign credit correctly over long trajectories, amid real-world tool errors, and with verifiers of varying quality.