SWE-bench Science Covers 20 Scientific Domains, but the Best Coding Agent Solves Only 47.90% of 119 Tasks

The OpenMOSS team has released [SWE-bench Science](https://arxiv.org/abs/2608.19799), extending coding-agent evaluation to scientific software. Correctness in this kind of software depends on more than whether tests pass: it also involves physical units, coordinate systems, numerical invariants, data-format semantics, and simulation assumptions. A seemingly reasonable patch may compile successfully while silently altering research results.

The benchmark contains 119 tasks drawn from 98 GitHub repositories across 20 scientific domains. Tasks are divided into three categories: existing issue-driven tasks, expert exploration tasks, and cross-module engineering integration tasks. For each task, the [public toolkit repository](https://github.com/OpenMOSS/SWE-bench-Science) provides a digest-pinned environment image and a separate verifier image. The agent modifies the code in the former, after which the evaluator applies the patch to a clean baseline, rebuilds the project, and runs held-out tests. Pier installs the harness at runtime, allowing users to switch among Codex, Claude Code, and mini-swe-agent without baking a specific agent into the image.

On the [leaderboard](https://swescience.github.io/) as of August 16, Claude Code with Claude Opus 5 (max) ranks first with a 47.90% pass@1 score, followed by Codex with GPT-5.6-sol at 46.22%. Claude Code with Opus 5 reaches 65.31% on expert exploration tasks, but only 38.46% on issue-driven tasks and 27.78% on engineering integration tasks. Codex with GPT-5.6-sol scores 38.89% on engineering integration tasks. The paper attributes failures to missing domain abstractions, pursuing the wrong investigative direction, fixing only surface-level symptoms, and failing to account for every dependent module. Results after removing scientific hints from task descriptions also show that more domain knowledge is not always better: well-aligned information can narrow the search space, while misplaced hints can create anchoring bias.

The barrier to reproduction remains substantial. Because of upstream licensing restrictions, 23 tasks require explicit opt-in, leaving only 96 enabled by default. The images are fixed to linux/amd64, so Apple Silicon systems require emulation. Private verifier tests and reference solution patches are not publicly available; although this can reduce contamination, it also limits external auditing. Engineering teams should next examine whether improvements from different harnesses, token budgets, and domain-retrieval methods can be reproduced independently using the same pinned images.