AI 研究與代理訓練
PlanPhys Dissects Long-Horizon Agent Training: On-Policy Distillation Is More Stable Than GRPO Under Sparse Rewards
Using controllable synthetic environments, PlanPhys separately examines how pretraining, GRPO, single-teacher distillation, and multi-teacher distillation develop long-horizon planning capabilities. Experiments show that a small number of long trajectories and explicit state transitions improve generalization, but distillation can instead damage existing capabilities when teacher knowledge or planning patterns are incompatible.
The research team introduced PlanPhys to study agents’ long-horizon planning capabilities separately from the difficult-to-trace data used in web-scale pretraining. Its controllable environment spans three domains, each with five difficulty levels. Tasks require models to progressively combine base objects into a target according to AND/OR composition rules. Researchers can independently vary trajectory length, error rate, planning knowledge, and policy patterns. The student model was trained on a 1.2-billion-token corpus, and the associated model weights were released under the Apache 2.0 license.
Pretraining results show that learning individual atomic skills does not mean a model can independently compose them into unseen long workflows; stronger compositional generalization emerged only after adding a small number of long-horizon examples. Requiring the model to first describe state-transition rules and then select an action also clearly outperformed directly predicting the next step. At the highest difficulty setting, the internal-world-model variant improved avg@8 and pass@8 by 45.8 and 42.9 percentage points, respectively. However, suboptimal or erroneous trajectories amplify mistakes over long sequences, making data quality more critical than in shorter tasks.
For post-training, the study compares GRPO with on-policy distillation (OPD). When the base model is already sufficiently capable, neither may be necessary. When tasks are longer and pretraining trajectories are lower quality, OPD has a broader effective regime than GRPO, which relies on sparse terminal rewards, because the teacher can provide more consistent token-level update directions. This does not mean distillation is always better: if the teacher possesses procedural knowledge that the student lacks and that is incompatible with the student’s existing knowledge, then even with a 100% teacher success rate, the student may fail to learn the new procedure while also damaging its existing world model.
Multi-teacher OPD exhibits similar boundaries. When teachers share a planning pattern, the student can generalize across environments; when their patterns conflict completely, the score on a previously learned domain fell from 90.00 to 15.62 in one experiment. These findings come from controlled synthetic tasks and small research models, so they cannot yet be directly extrapolated to browser or coding agents. Key questions for future work include whether the code, data, and complete training recipe will be released, and whether the same effects can be reproduced with real-world tool errors, imperfect teachers, and models at different scales.