AI coding tools / evaluation
Optimizing Agent SKILL Files Across Three Kotlin Projects: GEPA’s Average Gain Was Just 4.9 Percentage Points and Did Not Exceed the Noise
A new study reconstructed tasks from merged PRs to directly measure whether repository SKILL files help the same coding agent solve problems more effectively. Documents generated by GEPA improved performance by an average of 4.9 percentage points, while SkillOpt produced almost no gain—but the sample remains too small to rule out run-to-run variability.
Repository guidance files for coding agents, such as `SKILL.md` and `AGENTS.md`, are often treated as a low-cost way to improve performance. The problem is that a document can look professional without actually making an agent better at modifying code. A study added to arXiv’s latest listings on September 14 proposes “reverse-PR mining”: it selects merged PRs from Kotest, Ktor, and Koog, reverse-applies the relevant changes to the same frozen baseline version, and turns the resulting test failures into repair tasks. This avoids the saturation problem that occurs when simple tasks are synthesized directly from current code and frontier agents can solve them even without guidance files.
The researchers then used GEPA and SkillOpt to iteratively rewrite SKILL files based on rollout feedback, scoring them by the difference in task-solving performance between the same agent “with documentation” and “without documentation.” Across the three Kotlin repositories, GEPA-generated documents delivered an average gain of 4.9 percentage points. SkillOpt improved on the initial document by only 0.1 percentage points. Human evaluation was more encouraging than the numbers: a Koog maintainer said the output contained build-system and module knowledge that people would normally acquire only through hands-on involvement in the project.
The most important finding is that it is easy to choose the wrong evaluation objective. Optimizing for writing quality, the number of repository facts, CodeBLEU similarity to the original PR, or even token cost can select documents that read as comprehensive but do not improve agent behavior. A cost-based objective may even favor an empty SKILL file. Engineering teams should therefore use frozen versions, executable tests, and paired comparisons against a no-documentation baseline instead of relying solely on another model to review the documentation.
However, the 4.9-point gain still cannot be separated from run-to-run variability in a single agent, and the experiments covered only three JVM/Kotlin projects. Future work needs to include more repositories and tasks, publish per-run trajectories, and test whether the documentation continues to generalize after switching models or agent frameworks.