代理強化學習
Agent Lightning v1.0 Brings Real-World Agent Frameworks Directly Into Reinforcement Learning, Raising Qwen3.5-9B to 56.4% on SWE-bench Verified
Microsoft rewrote Agent Lightning to capture complete interactions from existing agent frameworks through a model API proxy layer, eliminating the need to move tool loops into the training engine. Using 6,000 training examples, the team reports that Qwen3.5-9B improved from 41.8% to 56.4% on SWE-bench Verified.
Microsoft and researchers from several universities have released Agent Lightning v1.0, bringing the agent frameworks actually used in deployment into model reinforcement learning instead of recreating simplified environments inside the trainer. The new version comprises roughly 3,500 lines of code and consists of a Trainer, an API Gateway, and a Rollout Controller. The Gateway proxies model endpoints and records requests and responses; the Controller runs existing agents locally or in Kubernetes Jobs; and the Trainer uses verl and vLLM to process trajectories and update the policy. Tools, context compression, and control flow remain managed by frameworks such as OpenHands.
The challenge with this decoupled architecture is that a single agent run can generate a variable number of model calls, while reserializing or compressing context may break token-prefix continuity. The paper therefore moves advantage computation and loss normalization from the individual training-sample level to the full-rollout level. The best configuration achieved a validation reward of 38.2%, compared with 35.0% for the sample-level baseline, while also maintaining more stable policy entropy.
For the coding-agent experiments, the researchers first cleaned SWE-smith. Of its 59,136 records, 18,033 lacked problem descriptions. They also removed tasks with missing branches or more than 200 tests, then filtered the remaining data by model success rate to produce approximately 6,000 examples. To prevent agents from directly accessing answers, the environment hides `.git`, disables Git commands, and uses Kubernetes network policies to block arbitrary outbound connections. Qwen3.5-9B ultimately improved from 41.8% to 56.4% on SWE-bench Verified.
The engineering value extends beyond the benchmark score: the system provides a reproducible framework for studying how retokenization, trajectory merging, and asynchronous scheduling affect RL. The next step is to determine whether the gains can be reproduced across models, agent frameworks, and programming languages. The current results also do not imply that simply adopting this framework will yield the same improvement.