AI 評測
Reassessing Five Computer-Agent Benchmarks: 15.3% of Failure Judgments Were Actually Wrong
Researchers manually reviewed 150 public trajectories labeled as failures and found that 10.7% were grader false negatives, while another 4.7% resulted from impossible tasks. Genuine failures also occurred mainly in planning and outcome verification—not mouse targeting or tool execution.
Computer-use agent success rates are commonly treated as a measure of capability, but new research suggests that these figures actually conflate agent errors, environment failures, and grader limitations. The team sampled 150 public trajectories that had originally received scores of zero from OSWorld-Verified, WebArena, VisualWebArena, WorkArena, and AssistantBench, then reexamined their reasoning traces, actions, screenshots, and final states. They found that 16 trajectories had actually completed the task but were marked as failures because answer-format, URL, DOM, or state checkers did not accept alternative solutions. Another seven tasks were impossible to complete because of outdated reference answers, search-service failures, CAPTCHAs, or virtual machines lacking required hardware states. In total, 15.3% of FAIL judgments were incorrect, with a 95% Wilson confidence interval of 10.4% to 22.0%.
The problems differed across benchmarks. WebArena had a 21.7% misjudgment rate in the sample, mainly because valid answers failed to match. AssistantBench also had a 21.7% rate, but its errors primarily stemmed from website drift. Among OSWorld’s 57 trajectories, state checkers failed to recognize eight valid alternative solutions, while two tasks were broken. No misjudgments were found among WorkArena’s 24 sampled trajectories. Because the study examined only trajectories already classified as failures, it cannot estimate false positives among successful cases or directly correct the benchmarks’ full model leaderboards.
The team then classified the 122 genuine failures by their earliest decisive cause. Verification and feedback issues accounted for 39.3%, including repeated actions caused by failing to notice that an operation had not changed the screen, which alone represented 29.5%. Planning issues accounted for 35.2%, while execution, parameter selection, and visual targeting together made up only 13.9%. This suggests that engineering teams focused solely on improving grounding may not be addressing the main bottlenecks. The next step should be to preserve replayable screenshots, tool responses, and state transitions; version tasks, environments, and graders separately; and report broken tasks, grader uncertainty, costs, and failure categories before publishing a single success-rate figure.