代理評測
ContinualSkillBench Finds That Most Progress in Agents’ “Accumulated Skills” Can Be Explained by Contextual Adaptation
ContinualSkillBench evaluates whether agents can organize experience into reusable skills using five domains, each containing 100 progressively dependent tasks. Explicit skill libraries did not significantly outperform simply retaining context on average, showing that growth in the number of skills does not by itself prove that an agent is capable of continual learning.
Many agent frameworks allow models to write prompts, scripts, or operating procedures after completing a task and add them to an external skill library for use in subsequent work. However, existing evaluations often compare only the success rates on the first and last tasks, making it difficult to distinguish among genuine skill abstraction, merely memorizing previous answers, and temporary adaptation based on error feedback.
ContinualSkillBench addresses this issue with a dynamic evaluation: five representative domains each contain 100 interrelated subtasks, ordered by difficulty and deliberately designed to include procedures reusable across tasks. Agents must execute the tasks sequentially, allowing researchers to compare settings with no history, retained interaction context, and explicit skill-library creation and maintenance, and to determine whether newly acquired skills produce transferable gains on different tasks later in the sequence.
The paper finds that sequential execution generally improves overall performance, although the magnitude varies substantially across models and domains. More importantly, in-context learning that simply retains prior context achieves scores comparable on average to explicit skill maintenance. This suggests that much of what is described as “self-evolution” may instead be short-term adaptation enabled by seeing previous attempts and evaluation feedback. Explicit skills still provide selective advantages on tasks requiring fixed procedures or exact output formats, but less capable models tend to accumulate more fragmented skills that apply only to a single task.
These findings suggest that skill-based agents should shift their evaluation unit from “how many files were added to the library” to “whether a skill can independently improve success rates on new tasks after the original trajectories are removed.” The paper’s abstract page currently provides no code link, and details such as the specific models, context lengths, and skill-pruning strategies may also affect the conclusions. Important next steps include releasing public evaluation environments, reproducing the results across models, and determining whether skill merging, retirement, and version regression testing can improve long-term transferability.