ClawGym II Captures Trajectories at Agent Framework Boundaries, Enabling Even Black-Box Claude Code to Participate in Reinforcement Learning

Agent reinforcement learning typically assumes that researchers control the entire agent loop. However, harnesses such as Claude Code and OpenClaw manage prompts, tools, subagents, and context themselves, making their internal workflows unsuitable for direct integration with traditional RL trainers. ClawGym II, submitted on August 17, instead operates at the model-serving boundary: each task and harness runs in a temporary sandbox, while all model requests from the agent pass through a serving proxy. This allows the training system to record prompts, responses, and branching relationships without needing to understand the framework’s internal implementation.

The captured calls do not naturally form a linear rollout. The research team organizes shared prefixes and subsequent branches into a prefix tree, and modifies PPO and critic-free GRPO so that policy updates can be computed over the reconstructed tree-structured trajectories. At the same time, the same harness is used during both training and inference. The system also supports mix-harness training, allowing a single model to learn jointly from the control flows of different agent frameworks rather than adapting only to one prompt template.

Using Qwen3-30A3B as the base model, the paper reports that ClawGym-Bench Pass@1 improved by 9.98 percentage points when trained through OpenClaw and by 14.81 percentage points when trained through Claude Code. Training remained stable for 200 to 400 update steps, with gains also recorded on JobBench and OfficeQA. This makes it feasible to use commercial or rapidly evolving agent frameworks as black-box training environments, allowing engineering teams to preserve the tool semantics and context management of their production harnesses.

However, the current results still come from combinations of models, sandboxes, and evaluations controlled by the authors, and do not yet demonstrate that improvements will generalize across harnesses from different vendors. Key issues to watch include whether the complete ClawGym II training code, compute costs, and failed trajectories will be released, and whether the serving proxy can maintain reproducible policy gradients after agent frameworks update their prompts or tool protocols.