AI 編程與評測
SWE-Bench ProMax Expands Refactoring Tests to Seven Languages, but the Best Agent Still Solves Only 41.2%
SWE-Bench ProMax selected 170 multi-file refactoring tasks from 29,782 candidate commits, with experts rewriting the requirements and reviewing the tests. GPT‑5.2 paired with OpenHands achieved the highest solve rate, but its trajectories show that agents often fail to update related call sites, configuration, and test files.
Released on August 10, SWE-Bench ProMax shifts the focus of coding-agent evaluation from localized debugging to large-scale refactoring that must preserve existing behavior. The dataset covers Python, Java, TypeScript, Go, C, C++, and Rust, comprising 170 tasks across 70 repositories. Each task includes a base commit, reference patch, test patch, evaluation script, and container image information, and can be loaded directly as a Hugging Face dataset.
The construction process began by automatically collecting candidates from 29,782 real-world commits, followed by environment validation. Experts then rewrote the problem statements and manually reviewed the tests. This step was intended to prevent two common flaws: tests that are too narrow and incorrectly reject functionally correct patches with different implementations, and tests that are too broad and check behavior not required by the task. Tables in the paper show that each task modifies an average of 15.9 files, including 11.4 non-test files, with an average code change size of 261.6 lines.
Six models were run using both the mini-swe-agent and OpenHands harnesses. With OpenHands, GPT‑5.2 ranked first at 41.2%; Claude Sonnet 4.6 reached 38.8%, while GLM‑5 and Qwen3.5 each achieved 36.5%. The same model’s performance could also vary substantially by harness: GPT‑5.2 rose from 21.8% with mini-swe-agent to 41.2% with OpenHands, indicating that tools, context management, and execution loops remain important variables in evaluation results.
Failure trajectories typically involved more reading and retry attempts, yet changed fewer files than the reference patch. Agents often modified the core interface correctly but failed to propagate the change to related call sites, documentation, configuration, and test fixtures. Engineering teams can use the dataset to measure multi-file planning and dependency tracking, but should not treat 41.2% as a measure of pure model capability: the benchmark contains only 170 tasks, and its reference patches and tests are already public, so future evaluations will still need to monitor contamination and reproducibility across reruns.