Environment Evolution’s Off-Policy Strategy Raises Terminal Training Difficulty, Improving Two Qwen Agents by 14.4 and 18.0 Percentage Points

Researchers from Tencent Hunyuan and collaborators have introduced Environment Evolution, an approach intended to address environment saturation in terminal-agent training. Existing co-evolution methods typically have the target model perform rollouts on tasks, then generate harder environments based on its failures. As the model improves, however, failure signals become scarcer, and the generated difficulty can become tied to a particular model. The new method instead derives off-policy difficulty from multi-stage learning objectives, separating the contributing factors into scenario novelty, skill rarity, and expected execution length, without relying on the current training agent’s pass rate.

The implementation uses multiple agents and two feedback loops. The first loop proposes and reviews an environment-evolution plan. The second modifies the environment according to that plan, then repeatedly repairs it until it passes solvability and quality checks. Each accepted environment becomes the starting point for the next generation, creating a lineage that grows progressively harder. If one direction exhausts its repair budget, the system switches to another difficulty dimension. Unlike synthesizing a large batch of tasks in a single pass, the focus is on continuously supplying the training process with verifiable interactive environments that retain discriminative value.

The authors ran rollouts using Hy4 preview, Claude Opus 5, and GPT-5.6 Sol, reporting that all three models experienced increasing difficulty across generations. They then conducted 200-step, long-horizon reinforcement learning runs on Qwen3.6-27B and Qwen3.6-35B-A3B. On Terminal-Bench 2.1, the two models improved by 14.4 and 18.0 percentage points, respectively, reaching peak scores of 71.5% and 64.9%. They also outperformed the co-evolution and environment-composition baselines reported in the paper.

These figures cannot yet be directly generalized to software-engineering agents broadly. The paper is a non-peer-reviewed preprint, and its appendix acknowledges that further details and experiments in which the same model both generates and learns from environments remain for future versions. A complete release of the code, environment images, and training data also does not appear to be available yet. Key areas to watch include cross-model and cross-benchmark reproduction, as well as the additional inference and compute costs introduced by iterative synthesis and verification across generations.