AI 代理／評測
Relevant Skills Can Also Derail Agents: 307 Failures Mostly Stemmed From Implementation Errors and Over-Validation
Using paired trajectories with and without skills, teams including Microsoft Research identified 125 functional failures and 182 cases of efficiency regression. The problem was usually not that the skill was irrelevant, but that the agent treated generic examples, validation checklists, or environment assumptions as hard task requirements.
Agent skills typically package procedures, examples, and validation rules in a `SKILL.md` file, but relevance to a task does not guarantee that loading a skill will help. Researchers from Microsoft Research, Huazhong University of Science and Technology, and other institutions proposed a differential auditing method: hold the model, agent framework, task, container, and verifier constant while changing only the skill configuration. If a run with a skill fails while a run without it—or with another semantically similar skill—succeeds, the successful run serves as a pseudo-oracle for locating behavioral differences caused by the skill. [Paper](https://arxiv.org/abs/2608.11888)
Starting with [SkillsBench](https://www.skillsbench.ai/) and [SWE-Skills-Bench](https://github.com/GeniusHTX/SWE-Skills-Bench), the researchers retrieved similar candidates from public skill repositories, expanding the pool of potential paired comparisons from 826 to 20,664. After excluding cases with insufficient evidence, overly narrow verifiers, and duplicates, they retained 307 skill-induced problems: 125 functional failures and 182 high-confidence efficiency regressions. Of the functional failures, 86 involved incorrect task implementation; among these, 46 used the wrong API, value, or output structure, while 36 omitted required elements entirely. Only two cases resulted from an incorrect judgment about the skill’s applicability.
An efficiency regression was defined as an increase in both token usage and elapsed time, with at least one exceeding twice that of the reference run. Additional procedures caused 114 cases, including 67 involving excessive testing, rebuilding, or debugging. Another 43 were caused solely by the skill body inflating the context. The team also used SkillTriage to analyze paired trajectories. Its fine-grained classifications matched human annotations in 88.8% of functional-failure cases and 72.5% of efficiency cases.
The engineering takeaway is not to disable skills, but to treat them as execution policies that require versioned evaluation: keep the main body concise, defer loading examples and exhaustive checklists, and scale validation according to the risk of the change and the remaining budget. However, the experiments used only OpenCode 1.15.1 and Claude Opus 4.6, and the root-cause taxonomy also involved human judgment. Whether the findings can be reproduced across models, frameworks, and proprietary enterprise skills remains to be validated.