AI 程式開發／代理評測
DeepRepoQA Uses Monte Carlo Tree Search to Trace Cross-File Code Relationships, but Evaluation Still Relies on LLM Judges
Rather than having coding agents search along a single path, DeepRepoQA uses MCTS to compare multiple repository exploration branches and synthesizes answers from evidence with line-number citations. Its authors report gains of 4 to 7 percentage points on SWE-QA, but data contamination, inference costs, and limited cross-language testing remain to be validated.
DeepRepoQA, introduced by researchers from Shanghai Jiao Tong University and other institutions, reframes repository question answering. Instead of relying on a single vector-retrieval pass or a single-path ReAct process, it uses Monte Carlo Tree Search (MCTS) with iterative branch expansion up to a configured limit. The system first uses Tree-sitter to parse relationships among classes, functions, calls, and modules while also building a semantic vector index. The agent can perform six types of actions: `FindClass`, `FindFunction`, `FindCodeSnippet`, `SemanticSearch`, `ViewCode`, and `Finish`.
Each exploration step passes through four roles: perception, planning, execution, and evaluation. The perception module summarizes the current path and sibling branches; the planner proposes candidate actions; the executor retrieves code snippets and removes duplicates; and the evaluator uses an LLM to generate a utility score and recommendations for the next step, then backpropagates the result to the root of the tree. In the paper’s experiments, each node could expand to at most three child nodes, with the search capped at 15 iterations. Final answers were required to cite supporting files and line numbers. The value of this design is that when the agent enters the wrong directory or finds only a related concept, it can return to another branch instead of carrying an early retrieval bias all the way into the answer.
On SWE-QA, the authors report that configurations using different backbone models outperform existing agents by approximately 4 to 7 percentage points. The GPT-5.1 configuration scored 70.06, close to Cursor’s 70.71 and slightly above Tongyi Lingma’s 69.12. However, this is not a code-repair pass rate but a composite score for repository question answering. The evaluation also averages scores from three LLM judges, which may still favor particular writing or reasoning formats. The paper also acknowledges the risk of pretraining data contamination, while its cross-language validation covered only 30 questions across three Java projects.
The code, baseline scripts, and data have been released, but the current setup requires a custom LLM endpoint and the Voyage embedding API, so it is not a fully offline reproducibility package. Engineering teams should next measure the additional tokens, latency, and API costs introduced by tree search, then retest the system on previously unreleased repositories using human-verified answers and real-world maintenance tasks.