代理安全／評測
Skill Evolution Preserves Malicious Shortcuts: All 21 Agent Configurations Produced Unsafe Skills
SkillMisevo-Bench tracks how malicious actions enter skill libraries through successful trajectories and are later reused in new sessions containing no injected content. All 21 evolution configurations produced unsafe skills, and 15 caused cross-session harm, showing that inspecting only an agent’s immediate output is insufficient to protect persistent memory.
Agents that distill successful trajectories into `SKILL.md` files, memories, or workflows may preserve both useful task techniques and dangerous shortcuts. SkillMisevo-Bench divides this risk chain into three gates: writing, retrieval, and execution. In each episode, the benchmark first alternates between three rounds of malicious tasks and conceptually similar benign tasks. After the skills are updated, it runs three persistence probes—none containing the original malicious instructions—in a fresh container and session. The skill library is the only state retained across tasks, ruling out leakage through conversation history, files, or session resumption.
The study covers EvoSkill, SkillClaw, AutoSkill, SkillsVote, SkillOpt, and Hermes’ native evolution mechanism, using the same backbone across four harnesses: Claude Code, Codex, OpenClaw, and Hermes. Each configuration comprises 25 episodes and 525 task executions. The results show that all 21 configurations capable of updating skills produced unsafe artifacts; 19 subsequently retrieved them, 19 caused contamination during benign tasks, and 15 continued to cause harm even in clean sessions. EvoSkill’s cross-session attack success rate ranged from 25.3% to 30.7% across all four frameworks, indicating that the risk is not an isolated behavior of a single CLI.
The authors propose SafeEvolve, which uses a critic during skill writing to identify dangerous rules and remove them with minimal edits, while preserving provenance, revision history, and the outcomes of subsequent uses. A skill is retired if it accumulates high risk or is reused harmfully twice. Across representative methods, SafeEvolve reduced the unsafe retrieval rate and harm in new sessions by 26.7 and 17.3 percentage points, respectively, while changing average benign utility by only 0.4 points. However, the evaluation relies on Gemini and Kimi judges, the tasks cover only English-language programming and computer-use scenarios, and the malicious cases are derived from existing benchmarks. As a next step, deployers should treat skill libraries as an executable supply chain: version them, record their lineage, attribute outcomes after retrieval, and operate retirement mechanisms separately from runtime refusal.