AI 程式代理
SkillForge Synthesizes Failures from Project Tests First, Lifting SWE-bench Verified Scores by More Than 5 Points for Two Coding Agents
Before real-world issues arrive, SkillForge creates verifiable failures from execution paths covered by tests, then distills repair trajectories into project-specific skills. When paired with Mini-SWE-Agent, it raises SWE-bench Verified Pass@1 for DeepSeek-V3.2 and GPT-5-mini to 72.2% and 60.6%, respectively.
When coding agents enter a new codebase, they typically have to repeatedly search files, run tests, and learn project conventions on the fly from errors. SkillForge moves this cold-start cost upstream. It first selects passing tests that cover core functionality and uses execution traces to identify multiple code fragments that jointly implement that functionality. It then masks the original implementation and asks an LLM to reconstruct the code using only the surrounding context. If the rewrite causes a test failure, the system packages the version, failure output, reference patch, and synthetic issue description into an executable training task.
Mini-SWE-Agent attempts to repair these synthetic failures. SkillForge then extracts two layers of knowledge from successful and failed trajectories. Global skills record module purposes, debugging sequences, and relevant APIs, and are retrieved with BM25 before a new issue is handled. Local skills are bound to AST entities and inject modification guidance and common pitfalls in real time when the agent reads the corresponding files. Unlike approaches that stuff a summary of the entire repository into the prompt, this method also does not require waiting for historical issues to accumulate.
In a temporally isolated SWE-bench Verified evaluation, the researchers synthesized 577 failures and allowed agents to use only skills created before the target patch commit. DeepSeek-V3.2's Pass@1 rose from 66.4% to 72.2%, while GPT-5-mini improved from 55.0% to 60.6%. After amortizing synthesis and distillation, the per-task cost increased from $0.049 to $0.074 and from $0.031 to $0.066, respectively. On the more difficult SWE-bench Pro, the scores also increased by 5.8 and 4.1 percentage points, respectively.
The engineering value lies in the ability to build an agent knowledge base in advance for a single large codebase under long-term maintenance. However, the gains are directly constrained by test coverage: legacy modules without executable tests cannot provide sufficient signals, and LLM-generated failure descriptions may not resemble real developer issue reports. The next step is to determine whether skills become obsolete as a codebase evolves rapidly and whether the offline knowledge-base construction cost can be effectively amortized over real-world issue volume.