代理評測
OmegaUse-OfficeVal Adds Per-Task Cost Labels for Office Agents, but the Best Model Still Trails Humans by More Than One-Third in Quality
Baidu’s research team has released 100 long-horizon document, spreadsheet, and presentation tasks, along with human labor time, price proxies, and executable validators. The best model achieved an average score of 17.91, compared with the human baseline of 27.79, showing that cheap and fast inference does not necessarily produce deliverables that are ready to use.
Released on July 29, OmegaUse-OfficeVal attempts to shift office-agent evaluation from “did it click the right button?” to “is the deliverable’s quality worth paying for?” The dataset contains 100 long-horizon tasks adapted from real-world work requirements. Its inputs comprise 220 DOCX, PPTX, XLSX, PDF, image, audio, and video files, and the tasks require the production of 115 editable deliverables. Each task takes humans an average of 2.32 hours to complete, with a median of 2.03 hours and a maximum of 8.35 hours.
The benchmark includes human completion time and a price proxy for each task. About 20% of the prices come from contributors’ real-world outsourcing experience, while the remainder were estimated by three experts. The average price across all tasks is $6.86. The scoring system does not restrict agents to a GUI, code, or a hybrid workflow; instead, it evaluates the final files. The research team converted granular criteria into 219 usability checks and 2,009 completeness checks. If a file cannot be opened, edited, or used, the entire task receives a score of zero. Other errors incur deductions based on the effort required to fix them.
Under a fixed agent framework, GLM-5.2, Kimi K2.6, DeepSeek-V4-Pro, MiniMax M3, and Qwen3.7-Plus were all faster and cheaper than humans, but the best average score was only 17.91, compared with 27.79 for a human baseline that selected the best human deliverable for each task. Notably, the model with the highest average score did not necessarily generate the most value on higher-priced tasks, meaning deployers cannot choose a model based solely on its macro-average score.
The benchmark’s engineering value lies in its full release of inputs, criteria, validators, and economic labels, which can be used to measure whether model upgrades genuinely reduce human rework. However, the tasks were de-identified and reconstructed, most prices are expert estimates, and the human baseline uses the best submission for each task—all factors that affect how the results should be interpreted. The next step is to examine whether the validators can resist targeted gaming and whether the results can be reproduced across different office software suites, languages, and regional wage levels.