代理評估
Task-CoEvolve Samples Just 20% of Validation Tasks, Cutting Harness Search Token Costs by 67% to 80%
Task-CoEvolve dynamically selects validation tasks based on historical disagreement among candidate harnesses, then estimates overall performance using sampling probabilities. On Terminal-Bench 2.1, it approaches full-set search performance, but the paper uses the same tasks for search and evaluation, and the code has not yet been released.
When automatically optimizing an agent harness, an outer-loop model repeatedly rewrites memory, retrieval, and prompting workflows, then runs validation tasks to decide whether to retain each candidate version. The problem is that long-horizon sandbox tasks can each take tens of minutes. Running the complete validation set in every round can quickly make evaluation more expensive than generating the new code itself.
[Task-CoEvolve](https://arxiv.org/abs/2608.20169) does not rely on a fixed small subset. Instead, it assigns sampling weights based on the Bernoulli variance of each task’s historical success rate. Tasks on which candidate harnesses sometimes succeed and sometimes fail receive higher weights, while exploration terms and minimum weight floors prevent sparsely observed or as-yet-unsolved tasks from disappearing permanently. Because the sampled difficulty varies between rounds, the system uses each task’s inclusion probability with either a Hájek estimator or an anchored difference estimator to convert partial results to a common full-validation-set scale.
On the 89 tasks in Terminal-Bench 2.1, evaluating only about 20% per round produced an average pass rate of 51.7% with GPT-5.6-Luna and Qwen3.6-35B-A3B, compared with 52.8% for the full-set Meta-Harness. The former ran about 180 task executions, versus 890 for the latter. Input token costs for the two models fell by 80% and 67%, respectively, while total search time dropped from 22.2 to 11.5 hours and from 38.0 to 20.5 hours. In the text-classification experiments, the 20% budget achieved an average test accuracy of 49.3%, slightly above the 48.6% recorded by full-set search.
The figures still require cautious interpretation. Terminal-Bench used the same tasks for search and final evaluation, with only a single rollout, and a difference of roughly one percentage point corresponds to just one task. High-information tasks also tend to run longer, so reducing the number of tasks to one-fifth does not produce a proportional reduction in elapsed time. The [GitHub project page](https://github.com/Agent4Science-UTokyo/Task-CoEvolve) currently contains only documentation and charts, and its README still labels the code as “coming soon.” Independent reproduction will have to wait for the actual implementation and configuration files.