代理評測
SWE-bench Science Releases 119 Scientific Code-Repair Tasks, With the Best Agent Still Below a 50% Full Pass Rate
The new benchmark uses private scientific tests to check units, numerical invariants, file semantics, and cross-module data flow, rather than merely whether code compiles. Claude Code paired with Opus 5 achieved an overall pass@1 of 47.90%, while experiments also showed that adding domain knowledge can sometimes anchor agents in the wrong direction.
SWE-bench Science extends code-agent evaluation to scientific software. Its 119 tasks are drawn from 98 GitHub projects spanning 20 domains and are divided into three categories: fixes for existing issues, expert exploration, and cross-module engineering integration. The tasks involve contracts such as unit conversions, coordinate systems, numerical invariants, and spectral and simulation semantics. Even if a localized change passes the public tests, its pass@1 is still counted as a failure if it breaks equivalent representations, boundary conditions, or data flow in undisclosed scientific cases.
The release packages each task’s environment and verifier as a standalone Docker image pinned by digest. Agents can access only the baseline code, problem description, and public diagnostics; reference patches and private tests never enter the workspace. At test time, the runner can use Codex, Claude Code, or another harness supported by Pier, while recording model settings, image versions, task-selection hashes, and test output. The public dataset includes 96 tasks by default; another 23 tasks must be explicitly enabled before they can be materialized because of upstream licensing terms.
Across eight model-and-harness combinations, Claude Code with Opus 5 achieved the highest overall pass@1 at 47.90%. Codex with GPT-5.6 Sol earned the highest average score on the private tests, at 78.82%, but its full pass rate was 46.22%. Failures primarily involved incorrect scientific abstractions, fixes that addressed only surface symptoms, insufficient cross-module coverage, and an inability to generalize principles to unseen cases. A paired experiment on 91 tasks further showed that providing scientific explanations increased DeepSeek-V4-Flash’s pass@1, but reduced GPT-5.6 Sol’s score from 36.26% to 31.87%. This suggests that adding more papers or expert text through RAG does not guarantee better repairs; whether the information aligns with executable evidence may matter more than the amount of knowledge provided.