代理評測
Terminal-Bench-Science 0.1 Tests Agents on 70 Real-World Scientific Workflows; Top Resolution Rate Reaches Just 30%
The Stanford-led open benchmark packages scientific work—including data analysis, simulation, theorem proving, and instrument calibration—as reproducible terminal tasks. Claude Opus 5 paired with Claude Code ranks first, but achieves a resolution rate of only 30% across three trials, showing that high scores on general coding benchmarks cannot be directly extrapolated to scientific work.
Terminal-Bench-Science has released version 0.1, featuring 70 research workflows across five domains: life sciences, physical sciences, Earth sciences, mathematics, and engineering sciences. The tasks require more than code generation, covering statistical inference, simulation and optimization, inverse problems, image reconstruction, signal processing, sensor calibration, model fitting, and scientific machine learning. Each task runs in an isolated terminal environment, with outputs verified by automated tests. The datasets, environments, oracle solutions, and versions can all be rerun through Harbor.
For the inaugural leaderboard, every model–agent combination ran each task three times. Claude Opus 5／Claude Code ranked first with a 30.0% resolution rate, followed by GPT-5.6 Sol／Codex at 22.4% and Claude Fable 5／Claude Code at 21.4%. The strongest open model, GLM-5.3, achieved only 8.1%. Every system also evaluated on the general Terminal-Bench scored more than 10 percentage points lower on the science edition, indicating that work involving specialized packages, domain assumptions, and verifiable numerical results is far more difficult than editing files or executing routine shell workflows.
The benchmark also publishes a cost and token Pareto frontier. A full evaluation of GPT-5.6 Sol reportedly costs $4,200, achieving a score close to Claude Fable 5 at less than one-third of the cost. Claude Fable 5, however, uses approximately 6.4 billion tokens, compared with Sol’s 8.4 billion. This highlights that research teams should not compare success rates alone; they must also measure the number of trials, token usage, sandbox costs, and the reproducibility of failures.
The dataset’s 70 official tasks were selected from 920 proposals through three review stages covering domain, technical, and final-threshold criteria. However, the current tasks use binary pass-or-fail scoring for the entire task. Even when a verifier contains multiple subconditions, the score may not capture progress that is close to completion. A dataset of 70 tasks and only three trials per task are also insufficient to eliminate sampling variability. Version 0.2 should address partial credit, environment flakiness, shortcuts and reward hacking, and whether publicly released tasks are subsequently absorbed into training data.