AI 代理可靠性
VaG Blocks Agent Contamination Before Skills Are Written, Reaching 72% pass@1 on a Terminal-Bench Subset
New research finds that agents that accumulate skills indiscriminately begin to regress after the third round, and deleting the original erroneous skills does not remove their derivatives. Verifier-as-Gatekeeper controls skill promotion through format, behavioral, and semantic checks, but the results come from a 50-task subset and no implementation has been released.
Having agents turn successful trajectories into persistent skills is often regarded as a form of continual learning that requires no weight updates. New research suggests, however, that capability does not increase monotonically with the number of skills. Once an erroneous skill enters the execution context, subsequent skill distillation may absorb its reasoning into new entries, creating a cross-round contamination chain. Even if the source is later deleted, the faulty logic remains in descendant skills.
The researchers ran five rounds of evolution on Event-50, a 50-task subset of Terminal-Bench 2.0. The ungated agent peaked in the third round and then declined, losing about 12.3 percentage points between its peak and the fifth round. Removing only the eight source skills classified as harmful recovered roughly 1.7 points. Even when complete provenance information was used to remove the entire derivative chain, nearly half of the decline remained unrecovered. This indicates that a skill library must record the context used during skill generation and its lineage, rather than retaining only the final text.
The team therefore designed Verifier-as-Gatekeeper (VaG). New skills first enter a Cold tier that is not injected into prompts. To be promoted to Warm, a skill must pass a frontmatter schema check, a single-skill A/B replay on held-out tasks, and an LLM-based semantic review for hallucinations, contradictions, and dangerous recommendations. A second gate then uses marginal-gain greedy selection to test skill combinations. Only skills that demonstrably improve joint performance can enter the Hot tier used at execution time.
By the fifth round, VaG achieved 72% pass@1 with 37 skills. Without gating, the skill library grew to roughly five times that size but produced a lower score. In ablation experiments, removing behavioral replay, joint selection, or semantic review reduced performance by 10, 8, and 4 percentage points, respectively. The frozen skill library delivered gains of 8 to 16 points across five backbone models and reached 69.0% on InterCode NL2Bash, compared with 65.5% without gating.
However, the main experiment covered only 50 tasks, and the paper itself reports 95% confidence intervals roughly 30 points wide for the intermediate rounds. Terminal-Bench 2.0, which the study used, also had 28 tasks corrected in the subsequent 2.1 release. VaG's code and complete trajectories have not been released, so for now it is better viewed as a warning for skill-lifecycle design than as a mature solution ready for direct adoption.