AI 安全
JailbreakSkill Packages Jailbreaking Strategies as Evolvable Skills, Raising HarmBench ASR by Another 13.4 Percentage Points
JailbreakSkill packages prompt rewriting, failure analysis, and skill evolution behind a unified interface, enabling red-team agents to generate new procedures from unsuccessful attack trajectories. Its second stage raises the cross-model average attack success rate on AdvBench and HarmBench by a further 17.5 and 13.4 percentage points, respectively, although evaluation relies on a GPT-4o judge.
Automated red-teaming tools typically embed jailbreaking methods in prompts, memory formats, or specific workflows, making new attack techniques difficult to replace or reuse independently. The [JailbreakSkill paper](https://arxiv.org/abs/2608.16465) instead uses skills as the unit of execution and evolution: each skill has a `SKILL.md` file describing its applicable conditions and procedure, and can include programs invoked via JSON-over-stdio.
The framework has two stages. The first provides 16 initial rewriting skills covering Base64, character perturbation, cross-lingual masking, code completion, role-playing, and narrative reframing. For each risk category, the planner maintains continuous judge scores for every skill and uses an upper confidence bound to balance exploring unfamiliar skills against prioritizing established strengths. Each original prompt may query the target model up to ten times, stopping after the first success.
Cases that remain unsuccessful are stored in failure memory together with the rewritten content, model response, and score. The second stage first groups them by risk category, analyzes common failure patterns, and then chooses whether to refine an existing skill, combine multiple mechanisms, or discover a new procedure. A candidate skill enters the skill library only if it resolves at least one previously failed case, after which it can also be reused across categories. The system therefore accumulates executable procedures rather than merely preserving one-off successful prompts.
The authors evaluated eight open- and closed-source models using 520 AdvBench prompts and 400 HarmBench prompts. The first stage achieved macro-average ASR@10 scores of 72.0% and 68.1%, respectively; TAP, the strongest overall baseline, achieved 70.9% and 54.6%. Adding skill evolution further increased average ASR by 17.5 and 13.4 percentage points, while GPT-5.4 improved by 48.6 points on AdvBench. UCB ranking also reduced the average number of target-model queries per prompt to 4.14 and 4.63, compared with 5.98 and 6.24 under random ranking.
The [open-source implementation](https://github.com/BattleWen/JailbreakSkill) includes three datasets, a skill registry, and query-budget limits, but it also directly provides offensive transformations, so deployment must be restricted to authorized environments. More importantly, success and harm severity are judged primarily by GPT-4o, while evolution uses Claude Sonnet 4.5. The results may therefore be affected by judge bias, vendor model updates, and the high cost of auxiliary inference. Defenders should watch whether persistent skills allow a single jailbreaking experience to become an attack capability that transfers across models and sessions.