AI 代理／評測
SkillProx Validates and Prunes Agent Skills, Raising Qwen3.6-27B Table-Task Accuracy to 54.5%
SkillProx does not retain model-proposed skill modifications outright. Instead, it reruns tasks, rolls back regressive versions, and removes knowledge units with negative utility. All three Qwen models improved on table tasks, but the authors have so far published only a skeletal repository with two commits.
Agent skills typically consist of `SKILL.md` files, examples, and reference materials loaded into context. They can extend behavior without training, but repeated patching can also cause conflicting rules to accumulate. SkillProx likens this problem to forward–proximal optimization, although its actual operations remain discrete text edits rather than differentiable gradient descent.
In the forward stage, the system first generates diagnoses from failed and successful trajectories, modifies the skill, and then reruns the same batch of tasks. If the candidate version’s hard accuracy or cell accuracy declines, the change is rolled back, and the observed failure direction is incorporated into the next round of diagnosis. In the backward stage, the skill is divided into sections and reference files, which are removed one at a time to estimate leave-one-out utility. Low-utility units may be merged, downgraded, or deleted, but every change must still pass a fixed validation set.
The authors trained the skill on SpreadsheetBench Verified and then evaluated it on WikiTableQuestions and HiTab. Qwen3.6-27B’s table-task score rose from 36.7% with a human-authored skill to 54.5%, compared with a no-skill baseline of 45.3%; it also achieved 80.0% on HiTab. Aggregating results across three models and three evaluations, the paper reports an average improvement of 3.0 percentage points over the strongest textual-gradient baseline. In ablation experiments, removing closed-loop diagnosis or the convergence stage reduced the score to 53.0% and 52.0%, respectively.
The engineering value of this design lies in treating skill updates as software assets that require testing and version control, rather than as trustworthy model notes. However, rerunning the same batch may cause modifications to overfit local cases, while repeatedly using a fixed validation set for decision-making can introduce selection bias. The paper does not demonstrate monotonic improvement after deployment. A more immediate limitation is that the official GitHub repository currently contains only a README stating that the code is “coming soon,” with no license, implementation, or reproducibility scripts. The next things to watch are whether the complete code is released and whether the system can still roll back safely for non-table agents, long-term online updates, and contaminated trajectories.