AI 評測
Audit of 254 Submissions Shows Top SWE-bench Verified Rankings Are No Longer Statistically Distinguishable
A new study analyzing per-instance results across four SWE-bench splits found that none of the 29 adjacent pairs among the top 30 Verified systems passed a paired significance test. Interactions between models and agent scaffolds can also exceed the gap between leaderboard leaders, so engineering procurement decisions should no longer treat decimal-point differences as stable rankings.
A study released on September 15 and accepted by ADMA 2026 reanalyzed per-instance outcomes from 254 submissions across four public SWE-bench splits, rather than rerunning the models. Its conclusion is not that coding agents now have “equal capabilities,” but that the current leaderboard lacks sufficient resolution among top-performing systems to reliably support the strict rank-order interpretations commonly applied to it. [Paper](https://arxiv.org/abs/2609.17394)
On [SWE-bench Verified](https://www.swebench.com/), which contains 500 human-validated, solvable issues, the two leading submissions each solved 396 tasks. The top 10 collectively solved 285 tasks and collectively failed 51, leaving only 164 tasks that actually distinguish them. The median nesting coefficient of their solved-task sets was 0.935, higher than the 0.774 implied by aggregate scores. This indicates that stronger systems mostly cover tasks that weaker systems can already solve, rather than demonstrating substantial complementary capabilities.
The authors used paired McNemar tests to compare the 29 adjacent submission pairs among the top 30 on Verified. At α=0.05, none could be distinguished statistically. By contrast, on the larger Test split of 2,294 tasks, 14 of 23 adjacent pairs were distinguishable. Simply removing every task that all leading systems either solved or failed did not fix the problem: after rescoring the top 30 on 209 non-degenerate tasks, the visible score range widened from 8.8 to 18.7 percentage points, yet no adjacent pair reached statistical significance.
Another warning is that leaderboard scores belong to a “model plus scaffold” combination, not to the model alone. For the same model paired with different agent frameworks, the observed range reached as high as 29.8 percentage points—far greater than the total score spread among the top 30. However, this was an observational study reconstructed from public metadata, so it cannot establish that the scaffold was the causal source of the differences.
The study recommends that leaderboards publish structured metadata on models, scaffolds, versions, attempt counts, and harness modifications, and replace overly precise rankings with descriptive tiers. For engineering teams, the next step should be to test the exact model–agent combinations they plan to deploy, tracking cost, stability, and the tasks on which candidate systems genuinely disagree, rather than making procurement decisions based on differences of 0.2 or 0.4 percentage points.