AI 代理與強化學習
Harness-R1 Learns to Modify Execution Frameworks from Agent Failure Trajectories, Boosting Qwen3.5-9B Success Rate by 9.3 Points
Rather than adjusting the target agent’s model weights, Harness-R1 trains a separate 9B model to modify its context, tools, and validation code. The average success rate across three environments rose from 44.3% to 53.6%, but no public code is available yet to assess patch safety.
Most AI agents accumulate large numbers of failure trajectories after deployment, yet improvements still typically require either fine-tuning the model again or having engineers manually modify prompts and tool code. Harness-R1 instead makes the agent’s “harness” itself a learnable target. This executable layer assembles context, integrates tools, validates actions, and handles error recovery, while the target agent’s weights can remain frozen.
The system separately trains a 9B “harness engineer” that reads a batch of the target agent’s failure trajectories and generates executable framework patches. After validation, each patch is used to rerun the frozen target agent on the same batch of tasks; whether those tasks actually succeed becomes the reward signal. Training begins with supervised fine-tuning as a cold start, followed by group-relative policy optimization, allowing the editing policy to update based on a patch’s ultimate effects rather than merely learning to imitate human edits.
The study evaluated Harness-R1 on WebShop, ALFWorld, and DBBench. With Qwen3.5-9B as the target agent, its average success rate increased from 44.3% to 53.6% without fine-tuning the model—a gain of 9.3 percentage points. Even when the target agent was first fine-tuned for the tasks and then paired with a dedicated harness engineer, its average score still rose from 59.2% to 64.2%. This suggests that model weights and the agent execution framework may be two complementary optimization axes, while offering a concrete path toward agents that automatically repair themselves based on incidents during continuous deployment.
The risks are equally clear: a model capable of automatically rewriting tool validation and recovery logic could also remove safety checks, overfit to a particular test batch, or introduce behavioral drift that is difficult to audit. Using the same set of failure cases for both patch generation and feedback could further amplify overfitting. The paper currently provides no public code, patch dataset, or independent reproduction results. Engineering teams should watch for whether future work adds permission sandboxing, diff reviews, held-out test sets, and rollback mechanisms.