代理訓練／強化學習
OpenForgeRL Brings Real-World Agent Frameworks into Reinforcement Learning, Decoupling Training and Execution with an Agent Server
Researchers from Microsoft Research and other institutions have introduced OpenForgeRL, enabling Codex, OpenClaw, and GUI agents to undergo end-to-end reinforcement learning within their native execution frameworks. The system uses a model-call proxy server and Kubernetes to isolate rollouts, but its code, data, and weights have yet to be fully released.
When agent models are deployed, they typically do not operate on their own. Instead, they are wrapped in harnesses such as Codex and OpenClaw, which manage multi-turn context, tools, subprocesses, and external environments. Existing SFT and RL training stacks generally assume that rollouts can be generated synchronously within the training process, making it difficult to reproduce these stateful, multi-process, real-world execution paths. If researchers instead build simplified agents, they create a mismatch between training and deployment.
At the core of OpenForgeRL is a lightweight proxy server: the harness continues issuing model requests as usual, while the proxy layer forwards those requests to the inference engines of training systems such as veRL. At the same time, it records prompts, responses, and final rewards, then reconstructs them into trainable trajectories. A separate Kubernetes orchestrator creates an isolated remote container for each rollout, handling environment configuration, timeouts, and cleanup. This decouples agent execution from GPU training nodes. In theory, the design allows the harness, environment, or RL algorithm to be replaced without rewriting the entire agent.
Using an approximately 30B-A3B MoE model, the authors trained OpenForgeClaw, which achieved 31.7 pass³ and 55.9 pass@3 on ClawEval, and 33.7 on QwenClawBench. The 8B OpenForgeGUI scored 37.7, 63.0, and 72.3 on OSWorld-Verified, Online-Mind2Web, and WebVoyager, respectively. The study also reports that RL improves self-verification, tool coverage, and multi-step plan completion rates, while error recovery remains a weakness.
Its engineering value lies in bringing the agent framework itself into the post-training distribution, rather than training only the bare model. However, all current results are self-reported by the authors, and the arXiv page does not yet provide a directly accessible official code repository, training data, or model weights. Discarding incomplete trajectories when containers fail may also alter the data distribution. Key areas to watch include the scope of the open-source release, the cost of reproducing results across different harnesses, and security mechanisms for isolating malicious tools and protecting training credentials.