AI 代理與評測
HarnessOpt-Bench Tests Models at Rewriting Agent Harnesses; Model Choice Has About 1.8× the Impact of Coding Tools
Scale AI treats prompts, tools, memory, and control flow as model-modifiable code, then measures real-world gains using a hidden test set. Across 111 experiments, native coding agents did not consistently outperform a shared harness, while the best validation scores generally proved overly optimistic.
Agent capabilities depend not only on model weights, but also on the prompts, tool definitions, memory, retry strategies, and control flow wrapped around the model. Scale AI introduced HarnessOpt-Bench, turning this “harness engineering” into an end-to-end evaluation. An optimization model receives an initial agent program, per-example trajectories from development cases, an aggregate validation-set score, and a fixed evaluation budget. It can directly add, remove, or modify code, but may ultimately submit only one candidate version, which is scored on an undisclosed test set in an isolated, trusted execution environment.
The benchmark covers OfficeQA, BrowseComp-Plus, Terminal-Bench, and GAIA. It tests five frontier models, pairing them with both the shared OpenCode harness and native tools including Claude Code, Codex, and Kimi CLI, for a total of 111 scored runs. Scores use “normalized gain,” which measures the proportion of the initial version’s remaining room for improvement captured by the new harness. After controlling for task and harness, changing the optimization model shifted scores by an average of 0.142. After controlling for model and task, changing the coding harness produced an average shift of 0.079, making the former effect about 1.8 times larger. The strongest combinations captured roughly two-thirds of the remaining improvement on OfficeQA and about half on BrowseComp-Plus, but differences among mid-tier models were often smaller than run-to-run variance.
The search trajectories also exposed practical issues. Using a broader range of optimization levers—including prompts, context, step limits, tool schemas, and retrieval strategies—was positively correlated with higher test-set gains. However, complete failure trajectories were requested in only 16 of the 111 experimental units. Most of the best validation scores were not reproduced on the hidden test set, showing that repeatedly exposing an agent to the same evaluation can cause selection bias to be mistaken for genuine progress.
Engineering teams can borrow the benchmark’s designs for isolated testing, resource metering, and version retention to build auditable agent-optimization workflows. However, each combination was rerun only twice, and the optimization models’ own inference budgets were not capped. The current results are therefore better suited to comparing search behavior than to directly estimating production benefits at a fixed cost.