程式代理評測
SWE-Gate Adds Review-Constraint Tests for Coding Agents, Finds One-Third of Green Patches Still Fail Requirements
SWE-Gate turns review feedback from real pull requests into an executable second test suite, separately measuring functional fixes and engineering constraints. Of 644 patches generated by four models that passed functional tests, 221 still violated explicitly stated review requirements.
The new SWE-Gate benchmark shows that getting all existing tests to pass is still not enough to demonstrate that a coding agent’s patch is ready to merge. The researchers extracted objectively verifiable requirements from review comments on real pull requests—such as preserving existing exception semantics, retaining parameter order, ensuring idempotency, and avoiding breaking public schemas—and transferred them to 75 open-source Python repositories, creating 303 repair cases. Each case includes separate functional and constraint tests, along with a “functionally correct but noncompliant” patch and a gold patch that passes both, confirming that the two sets of requirements are genuinely separable yet can be satisfied simultaneously.
Using the same Mini-SWE-Agent setup with a maximum of 100 interaction steps, the researchers tested GPT-5.5, GPT-5.4-mini, DeepSeek-V4-Flash, and GPT-4o-mini. Even when the agents were shown the natural-language constraints in their prompts, the four models collectively produced 644 patches that passed the functional tests, of which 221—or 34.3%—failed the constraint tests. The strongest model, GPT-5.5, achieved a 74.9% functional success rate but only a 52.8% joint success rate; 29.5% of its functionally valid patches still contained hidden failures. Making constraints visible increased each model’s constraint-adherence rate by 10.2 to 25.6 percentage points, but reduced functional success by 0.7 to 9.9 points, suggesting that agents operating within a limited step budget may sacrifice the original repair objective to satisfy additional rules.
The immediate implication for CI is that teams should not treat a single test suite as a sufficient acceptance gate for agent-generated patches. Compatibility, exception types, resource cleanup, and reentrant behavior can be encoded as independent merge gates, while functional and joint success rates should be tracked separately. The study’s limitations are that its cases were synthesized with LLM assistance, it covers only Python, and each model was run only once per case. The results reveal the gap observed on this benchmark and cannot be directly extrapolated to the rejection rate of all real-world pull requests.