OSReward Evaluates 27 Computer-Agent Judges, With Hard Trajectories Exposing a Widespread “False Success” Bias

The training and evaluation of computer-use agents increasingly rely on vision-language models as automated judges: the models examine screenshots, actions, text histories, and reasoning trajectories, then determine whether a task was completed successfully. OSReward notes that this critical reward signal has historically lacked independent calibration. If a judge labels a failed trajectory as successful, erroneous data may flow directly into filtering pipelines, reinforcement learning, and leaderboards.

The research team therefore built its own execution environments spanning the web, Windows, Ubuntu, and mobile devices, collecting 1,019 trajectories generated by four types of agents. Each trajectory was evaluated by three annotators, with disagreements undergoing further review, for a total investment of roughly 800 hours of human labor. In addition to the full test set, OSReward-Hard focuses on difficult examples, while OSReward-Multi separately evaluates task alignment and execution efficiency. Under a fixed protocol, all 27 VLM judges exhibited a shared leniency bias: genuinely failed executions were more likely to be accepted. Overall performance that appeared close to 90% on the general set declined markedly on the hard examples. The analysis also showed that judgment signals often came primarily from textual action histories rather than final screenshots, suggesting that an agent could conceal the fact that the environment had not actually reached the target state simply by leaving behind a plausible-sounding narrative.

The team further generated 321,631 ensemble judgments from its own rollouts and four public trajectory datasets, distilling them into OS-Shepherd-100K, which includes judge reasoning, and using it to train open 9B and 35B reward models. This provides a self-hostable baseline for teams unable to continuously afford commercial frontier judges, but it should not replace deterministic verifiers. When the environment state can be queried directly—whether a file was created, a setting was applied, or a transaction was completed—programmatic verification should remain the preferred approach. Another issue warrants caution: the paper’s abstract claims costs are 30% to 60% lower, while the project page states they are 30 to 60 times lower. Until the authors clarify the billing assumptions, the latter figure should not be cited as a basis for procurement decisions.