AI coding agents
Harness-of-Harness Extends Coding Agent Work Sessions Through Planning, Development, and Independent Testing Loops, but Its Core Implementation Is Not Yet Public
A team at the Shanghai AI Laboratory has introduced Harness-of-Harness, which adds a recurring control loop for planning, implementation, and verification around existing coding agents. Across three-round experiments, it delivered an average relative gain of 52.25%, but each condition used only a single generation, and the executable HoH-lite is still marked as coming soon.
Harness-of-Harness (HoH) is not a new foundation model, but a long-horizon control framework wrapped around existing coding agents such as Codex, OpenCode, and Pi. In each round, a Planner generates a small, verifiable development plan based on the requirements and evidence from the previous round; a Developer implements it in a shared workspace and runs targeted tests; and a read-only QA Tester then uses white-box and black-box methods to assess completeness, functionality, and usability. The test report becomes input for the next round, while hidden evaluation results are not fed back into the loop, with the aim of reducing the risk of directly overfitting to the benchmark.
The framework does not require agents to follow fixed reasoning steps. Instead, it constrains the output formats for plans, software artifacts, test reports, and related deliverables; noncompliant outputs are retried. Long-term state is preserved through a versioned workspace and file index, with detailed information loaded on demand rather than placing the entire history into the context window. This design turns “memory” into inspectable engineering artifacts and allows major regressions to be rolled back to a previously validated version.
The study tested Codex + GPT-5.5, OpenCode + DeepSeek-V4-Pro, and Pi + MiniMax-M3 on GameCraft-Bench, FrontierSWE, and ProgramBench. Three-round HoH improved performance by an average of 52.25% relative to a single native-agent run. Codex’s dominance score on FrontierSWE rose from 44% to 71%, while in a ten-round test it increased from 22% to 72.67%. The team also demonstrated a playable first-person shooter completed over more than 70 rounds, although this was not a controlled benchmark.
Engineering teams should initially treat these results as evidence that an iterative protocol may be more effective than a single run, rather than proof that unattended software development is mature. In the main experiments, each task and condition was run only once; the models did not share a reproducible seed; and token statistics cannot be compared across models because vendors account for caching differently. More importantly, the GitHub repository currently contains only the paper, assets, assets materials, and demos, while HoH-lite—the executable implementation of the core workflow—has not yet been released. The real deployment costs, failure-recovery behavior, and security security boundaries therefore remain to be independently reproduced.