代理基礎設施
690 Agent Skills Put to the Test: Hybrid Retrieval Reaches 73.5% hit@5, While Adding an LLM Knowledge Graph Lowers It by 11.2 Points
A new study finds that pre-restricting candidate edges using embedding nearest neighbors prevents a knowledge graph from reaching skills the retriever could not already find. BM25–vector fusion still leaves about a quarter of queries unresolved, but outperforms graph neighbors under the same token budget.
As agent skill libraries grow, systems must first decide which `SKILL.md` files to load. Otherwise, placing all 690 skills from the study into context would require about 46,900 tokens per query. A [comparative study](https://arxiv.org/abs/2608.06196) submitted on August 6 tested two approaches: a hybrid ranker combining BM25 with MiniLM vector similarity, and a graph-based method in which Claude Haiku 4.5 generated eight relation types—including prerequisite, data flow, and ordering—among each skill’s eight embedding nearest neighbors, producing 1,421 typed edges.
Across 117 queries deliberately written to avoid reusing wording from skill descriptions, the hybrid ranker achieved a hit@5 of 73.5%±8.0%. Replacing some ranked results with graph neighbors reduced performance by 11.2 percentage points under the same candidate and token budgets, a statistically significant difference. The problem was not that the graph was entirely wrong, but that candidate generation had already been constrained by the same embedding-neighbor set: among 1,022 distinct typed pairs, 98.6% already existed in the similarity graph. Adding 1,421 edges neither connected any new nodes nor merged any components. The LLM could attach semantic labels to existing connections, but it could not expand the reachable set.
This “pre-filtered topology ceiling” appeared directly in the failure cases. Of the 30 queries missed by the hybrid ranker, 73% still could not reach the correct skill within three hops from the top-ranked candidate. When the first-ranked guess was wrong, continuing down the ranked list recovered 48.3% of cases, while graph neighbors recovered only 25.9%. In addition, queries written by the system’s authors inflated BM25 hit@5 by 44.2 points, showing that echoes of skill-description wording can seriously contaminate evaluations.
The findings have practical implications for systems adopting the [Agent Skills progressive disclosure specification](https://agentskills.io/specification): names and descriptions are suitable for first-stage recall, but workflow graphs should be built from actual co-usage logs, human-authored dependencies, or candidate signals that differ from those used by the retriever. The study tested only one enterprise skill library and 117 queries, while 8.5% of directional relationships remained contradictory. The reproduction repository listed in the paper is also not currently publicly accessible, so the results still await independent replication.