AI 程式開發
55,000 Real-World Review Comments Show That Developers Are More Likely to Adopt Short, Directly Applicable AI Fixes
An empirical study analyzing 54,791 review comments left by five coding agents across 342 Python repositories found that inline code suggestions were the strongest predictor of whether a comment was resolved. Incorrect suggestions and failure to account for a project’s existing design were the main categories in unresolved discussions, but observational data cannot be used to directly rank the quality of different agents.
A new study moves beyond testing AI code review on synthetic vulnerability benchmarks, instead tracking how developers respond to comments that agents actually leave on GitHub pull requests. Researchers collected 54,791 review comments generated by Copilot, Cursor, Codex, Devin, and Claude across 342 Python repositories. They compared whether comments were marked as resolved, their content categories and format, and the subsequent actions of core and peripheral contributors.
The results show that 72.9% of resolved comments came from Copilot. This reflects the composition of the sample and should not be interpreted as a 72.9% resolution rate for Copilot or as evidence that it is necessarily superior to other tools. Regression analysis found that including a directly applicable inline code suggestion was the strongest predictor that a comment would be resolved. Longer, more structurally complex comments were less likely to be acted on. Core developers more often addressed design and evolvability issues, while peripheral contributors responded more frequently to functional defects, indicating that project knowledge and permission levels influence how AI recommendations are implemented.
The team also used open card sorting to analyze 470 unresolved discussion threads, identifying ten patterns. Incorrect suggestions and developers’ deliberate decisions to preserve the existing design were the most common. For teams building review agents, the implication is that the optimization target should not simply be to “find more issues.” Agents must also limit noise, provide minimal directly applicable patches, and incorporate project context such as issues, tests, and architectural decisions.
The study is limited to public Python repositories, and a resolved status does not mean that a suggestion was correct or merged. Differences in agents’ launch dates, usage volumes, default interfaces, and repository selection may introduce confounding factors. Future research will need to incorporate actual diffs, test results, and randomized experiments before the review quality of the tools themselves can be compared.