模型訓練
ReflectRL Turns Expert Error Trajectories into Training Signals, Improving GRPO Cross-Domain Evaluation by an Average of 19.1 Points
Instead of discarding reasoning trajectories where a strong model produces an incorrect answer, ReflectRL asks the student model to identify localized errors and then gradually removes this hint. The study reports consistent gains across nine benchmarks, although the results come primarily from mathematical reasoning settings, and the listed GitHub project link is currently inaccessible.
On-policy post-training of large language models typically retains only the expert model’s correct “golden trajectories.” When the expert answers a difficult problem incorrectly, the entire reasoning trace is often discarded as a negative sample. ReflectRL argues that these trajectories may still contain long correct prefixes, with errors usually concentrated in only a few steps. It may therefore be easier for a student model to locate and correct errors in existing reasoning than to solve the problem from scratch.
The system calls this type of data Golden Negative Trajectories (GNT). In the RLVR setting, each rollout group contains both direct-answer inputs and inputs that ask the model to “reflect after reading an incorrect trajectory.” It reuses the existing verifier and objectives such as GRPO and DAPO, without adding an extra loss. During training, a cosine schedule gradually reduces the proportion of reflection samples until only direct answering remains, preventing the model from depending on external expert trajectories at inference time. For on-policy distillation, GNT is provided only to the teacher, while the student continues generating tokens from the original question.
The researchers released OpenR1-GNT-69k, which contains 69,000 failed expert trajectories, and evaluated four model backbones, four training methods, and nine benchmarks. Using Qwen2.5-Math-7B as an example, adding ReflectRL to GRPO increased the average score across six in-distribution tests from 37.0 to 42.4. The cross-domain average across ARC-c, GPQA, and MMLU-Pro rose from 20.9 to 40.0. DAPO and OPD experiments with a 3B student model also showed improvements.
From an engineering perspective, the notable point is that the method changes the rollout configuration rather than the inference interface, making it potentially compatible with existing RLVR pipelines. However, all reported figures are self-reported by the authors, and the primary training domain remains automatically verifiable math problems. Whether the cross-domain gains reflect genuinely transferable reasoning ability, data overlap, or benchmark-specific characteristics still requires independent replication. The GitHub project listed in the paper currently returns a 404 error. Although Hugging Face pages for the model weights and dataset are available, the reproducibility of the complete training code remains unconfirmed.