AI 研究
Why an Agent’s Estimated 40-Hour Capability Isn’t a Guarantee of Autonomous Research
METR’s new evaluation adds research tasks that are hard to verify, revealing a gap in judgment behind long-task scores. Human work-hour estimates are also uncertain, limiting what the numbers can tell us.

Users preparing to hand extended research and development work to AI agents have a new metric to consider—and one that is easy to misread. In an evaluation released on November 20, METR estimated GPT-6 Astra’s “50% task time horizon” at 40 hours and Opus 5.5’s at 39 hours. But the same study added work that is difficult to verify and found that agents still have weaknesses in certain judgment tasks. Original METR evaluation
Forty hours is a difficulty measure, not a delivery promise
A “time horizon” expresses task difficulty in terms of the hours a human expert would need, then fits an agent’s test results to estimate its success rate. A 50% time horizon is the point at which the curve predicts a 50% success rate. It does not mean the agent can operate autonomously for that long, nor that it has a one-in-two chance of succeeding at every task of that duration. METR methodology
That distinction directly affects how work should be assigned. For a week-long research project, the 40-hour metric does not tell you whether an agent can choose the right question, handle new evidence that emerges along the way, or recognize whether its findings are valuable. METR’s existing tasks focus mainly on software engineering, machine learning, and cybersecurity, and are deliberately designed to have complete context and clear success criteria. Real jobs also depend on project context, collaboration, and quality requirements that are difficult to score automatically. Scope of the metric
Producing an output doesn’t mean judging its quality
One task in the new evaluation asked agents to design capability tests that would be useful to METR. METR said none of the questions the agents produced was useful enough to include in this report. Another test asked models to rate the quality of critiques of philosophical arguments. The tested public models agreed with expert ratings less often than METR estimated a human expert would after spending 15 minutes on the task. Judgment test results
These results are limited to the scenarios tested. METR also cautioned that the test setup may not have elicited the models’ full capabilities. The findings are evidence of specific weaknesses; they do not show that all research work is unsuitable for agents or establish that AI will not accelerate research and development.
Human work-hour estimates are uncertain too
Of the 34 tasks in the new suite with time and score calibration, only three had successful human implementations as a baseline; researchers estimated the hours for the other 31. GPT-6 Astra’s 40-hour point estimate has a 95% confidence interval of 13 to 2,645 hours. This is not a range of observed completion times; it reflects statistical uncertainty in the estimate. Calibration method and Appendix B
The practical takeaway for users: before handing off work, establish who can verify the results, how long it might take errors to surface, and who is responsible for the research direction. That is a better use of this metric than translating 40 hours directly into a week of work that can be safely delegated.