AI 安全
MaliciousSkillBench Aggregates 9,740 Agent Skills; Cross-Source Detection Macro-F1 Falls to 0.665
The new benchmark deduplicates agent skills from 13 public sources and splits them by structural family, preventing near-duplicate samples from appearing in both the training and test sets. The best text classifier performed strongly on a random split, but misclassified 62.4% of benign skills as malicious when evaluated on unseen sources.
Agent Skills are more than prompt text: they can also carry scripts, resources, and service configurations. Once distributed through marketplaces, repositories, or team templates, they become a supply-chain interface capable of affecting tool permissions and execution workflows. Released on August 20, MaliciousSkillBench attempts to turn these risks from a collection of isolated cases into a comparable detection problem.
The research team collected data from 13 public sources, normalizing 8,414 raw malicious records into 7,539 unique identities and then organizing them into 4,588 families based on structural similarity. After excluding data conservatively identified as having label conflicts, the main benchmark contains 9,740 Skills: 7,505 malicious and 2,235 benign samples. The dataset provides fixed random, structure-disjoint, and source-disjoint splits. The latter two are particularly important because, with only a random split, a classifier may simply memorize rewritten versions of the same malicious template.
The results reveal a clear generalization gap. The best word-level TF-IDF SVM achieved a Macro-F1 of 0.932 on the random split and 0.916 on the structure-disjoint split, but only 0.665 when evaluated on entirely unseen sources. It still detected 95.6% of malicious samples, but at the cost of classifying 62.4% of benign Skills as malicious. Three off-the-shelf scanners fell at the opposite extreme: some reduced the false-positive rate to around 1%, but their malicious-sample recall dropped to just 2.5% or even zero, making them unsuitable as standalone installation gates.
The public dataset provides the complete static text of 9,735 records under CC BY 4.0. For five samples containing sensitive credential material, only sanitized versions were released, with no executable companion packages included. This makes the benchmark better suited to testing static pre-screening than to representing the dynamic behavior of Skills inside a sandbox. The attack taxonomy also covers only 4,983 malicious identities, while class proportions and the quality of source labels may affect the overall metrics. Engineering teams should prioritize source-disjoint evaluation and combine static scanning, least privilege, signed provenance, and runtime monitoring into a layered defense.