代理與強化學習
SPADE Has the Same LLM Write Executable Training Environments, Improving a 30B Model by an Average of 5.3 Points Across Eight Benchmarks
SPADE trains the environment designer using the return gap between runs with and without hints, allowing the curriculum to become harder automatically as the agent improves. Qwen3-30B-A3B gained as much as 13.9 points on tool-use benchmarks, though environment validators may still accept underspecified or unsolvable tasks.
Fixed-task reinforcement learning loses its training signal once a model masters the existing tasks. [SPADE](https://arxiv.org/abs/2608.19197) instead has the same language model alternate between the roles of “environment designer” and “reasoning agent.” Drawing on excerpts from pretraining corpora and a memory of previous environments, the designer generates complete Python programs containing state transitions, reward functions, validation logic, and `reset()` and `step()` interfaces. A program is added to the agent’s training pool only after passing structural and execution checks.
The key signal is hint-based regret: the agent runs each environment both with and without privileged hints, and the difference in returns is used to update the designer. A small gap usually means the task is too easy, while failure in both settings may indicate that it is unsolvable. The objective is therefore to push new environments toward the boundary of the agent’s capabilities. The agent itself is updated from task returns, with advantages normalized separately for the two roles within the same policy. The public release includes 4,976 synthetic games; the tool-use variant additionally generated 4,023 simulated backends and multistep workflows.
On Qwen3-30B-A3B-Instruct, SPADE outperformed the strongest fixed-environment baseline by an average of 5.3 points across eight held-out benchmarks covering mathematics, science, programming, and procedural reasoning. It improved BFCL-v4 multi-turn and ACEBench-Agent by 5.7 and 13.9 points, respectively. The authors also released 4B, 8B, and 30B-A3B checkpoints, environment data, and [MIT-licensed training code](https://github.com/spade-rl/spade), with backend integrations for SGLang, Megatron-LM, Ray, and Tinker.
One limitation is that the paper is still labeled a work in progress, and its main results cover only the Qwen3 family. The project page also acknowledges that some programs contain state errors and that most randomly generated Sokoban boards cannot be solved within the step limit. Tool validators sometimes check only the primary state change without covering every constraint in the instruction. Future work should measure the proportion of erroneous or unsolvable environments, reproducibility across model families, and the additional generation and dual-rollout costs of the adaptive curriculum.