OSReward Finds That Computer-Agent Judges Commonly Let Failed Trajectories Pass, While Visual Evidence Has Surprisingly Limited Impact on Verdicts

Computer-use agent training and leaderboards often feed complete sequences of screenshots, actions, and reasoning traces into a vision-language model, which then determines whether a task was completed. OSReward shifts the focus from the agent itself to this “judge model,” creating 1,019 human-validated gold-standard trajectories spanning the web, Windows, Ubuntu, and mobile devices. Each trajectory was independently annotated by three people; when opinions diverged, two senior reviewers discussed the case instead of relying directly on a majority vote.

The study tested 27 VLMs under a standardized protocol. The best binary accuracy on the full dataset approached 90%, but the models were generally lenient: across every model family, the most common error was accepting a task that the agent had not actually completed. These “false success” judgments accounted for roughly two-thirds of all errors and were more than three times as frequent as false failure judgments. Reliability declined further on OSReward-Hard, which deliberately concentrates disputed cases. Cloud-based judges that maintained around 70% accuracy cost approximately $45 to $100 to evaluate the full dataset once.

The input ablation results are even more notable. Removing step-by-step reasoning and action text reduced accuracy by an average of 7.2 percentage points and flipped 22.7% of verdicts, while changing seemingly critical screenshot configurations produced a smaller change in aggregate accuracy. This suggests that judges often infer success from the agent’s textual narrative rather than verifying the final interface state. Even majority voting among the top three judges improved accuracy by only about one percentage point because they often made the same mistakes on the same difficult samples.

The team also created OS-Shepherd-100K and trained 9B and 35B reward models based on Qwen3.5, aiming to approach the performance of commercial judges at a lower cost. However, the paper is still labeled as a work in progress, and the project page says that the code, data, and model weights are being released. Some cost figures are also presented differently in the abstract and on the project page. For now, engineering teams should treat judges as potentially biased measurement components and retain environment-state verification, failure audits, and platform-specific calibration instead of treating a single VLM verdict as ground truth for reinforcement learning.