AI 評測／程式代理
Real-SWE Tests Coding Agents on Private Enterprise Repositories, but the Highest Solve Rate Is Still Just 38.8%
The new benchmark places real-world tasks involving taxation, billing, identity migration, and more in isolated sandboxes. The best pass@1 across eight model-agent framework combinations was 38.8%. The results more closely reflect enterprise development, but the repositories, evaluators, and complete trajectories have not been released, so the rankings cannot currently be reproduced independently.
Specific Labs has released Real-SWE, an effort to bridge the gap between SWE-bench-style evaluations and enterprise software development. The benchmark currently contains ten tasks taken or adapted from licensed private production repositories. They cover tax-rate calculations, API token metering, cross-region data scanning, billing schedules, and customer identity migrations. Each task may require not only code changes but also interaction with PostgreSQL, MongoDB, Kubernetes, Linear MCP, or simulated cloud services. The reference solutions modify a median of 11 files, compared with a median of six in the team’s statistics for FrontierCode and DeepSWE.
The team ran each of eight “model + native harness” combinations eight times, producing 640 rollouts in total. Claude Fable 5.1 with Claude Code achieved the highest solve rate at 38.8%; GPT-6 Astra with Codex CLI reached 33.8%, while Gemini 3.8 Flash with Gemini CLI reached 31.2%. Six of the ten tasks had an overall solve rate below 15%, and no model passed the hardest task, which involved a streaming reducer. Failures were not limited to syntax errors: the classification results identified missed requirements, unverified system assumptions, and correctly understood concepts connected to the wrong existing components as major failure modes. Short execution time was also not a reliable proxy for success; 71.4% of rollouts lasting less than ten minutes still failed.
The significance of this work is that it explicitly evaluates combinations of models and tool frameworks instead of pretending that performance can be attributed entirely to the model. For engineering teams, a more practical approach may be to extract tasks from their own Git histories and build internal regression suites using fixed commits, sandboxes, and hidden tests.
At this stage, however, Real-SWE remains a closed benchmark: its complete private source code, tests, and trajectories cannot be released, preventing outsiders from examining data contamination, prompt fairness, or the evaluators. Discussion on Hacker News has also noted that providing private repositories to cloud model vendors creates data-governance concerns. These figures are best treated as an initial stress test of enterprise workloads, not as a general-purpose model leaderboard.