代理工程與評測
HarnessOpt-Bench Turns Agent Harness Optimization Into a Benchmark; Model Choice Has About 1.8× the Impact of the Coding Harness
HarnessOpt-Bench asks models to modify prompts, tools, memory, and control flow within a fixed evaluation budget, then measures actual gains on an unseen test set. Across 111 experiments, the model itself accounted for more variation than the outer coding harness, while native tools did not consistently outperform shared ones.
Agent capabilities depend not only on model weights, but also on the harness comprising prompts, tool definitions, retry strategies, memory, and control flow. Released on August 6, [HarnessOpt-Bench](https://arxiv.org/abs/2608.06301) frames “having one model improve another agent” as constrained program optimization: the optimizer may modify the entire executable harness, but it cannot replace the target model, environment, or verifier. The development set provides per-case results and trajectories, the validation set reveals only the aggregate score, and the test set is held on an isolated server and run only after the model submits its final version.
The benchmark covers OfficeQA, BrowseComp-Plus, Terminal-Bench, and GAIA, while limiting each development and validation split to at most four full case passes and imposing a target-model token budget. The study compares Claude Opus 5, Claude Sonnet 5, GPT-5.6 Sol, GPT-5.6 Terra, and Kimi K3, with each model working through both OpenCode and its native coding harness. Across 111 scored units, changing the model while holding the task and harness fixed produced an average normalized-gain difference of 0.142; changing the harness while holding the model fixed produced a difference of 0.079. The former was about 1.8 times the latter. Among 20 comparisons between native and shared harnesses, shared tools won 11 and native tools won nine, showing no general “first-party pairing” advantage.
The results also reveal a search bottleneck for optimizers: engaging with more design levers—including prompts, context, tool schemas, retrieval, and timeouts—was positively correlated with higher gains, whereas extensively reading full trajectories showed no comparable relationship. Most submitted versions scored below the best validation score observed during the search when evaluated on the hidden test set, indicating that reporting only peak validation performance overestimates self-improvement. This echoes [Lilian Weng’s overview of harness engineering](https://lilianweng.github.io/posts/2026-07-04-harness/): the harness is now an executable system-design space, not merely a prompt template.
Engineering teams should next examine whether the benchmark can incorporate dynamic tasks, perturbations to tool behavior, and mature production-grade seeds. The current seeds deliberately leave substantial room for improvement, and the GAIA starting point is not even functional. The optimizer’s own inference cost is also uncapped, so the scores cannot be translated directly into cost-effectiveness in production environments.