AI 代理／自動化研究
Across 1,338 Post-Training Trajectories, AI Agents Changed Strategy in Just 2.1% of Adjacent Experiments
The study finds that although agents can independently train, debug, and evaluate models, they lock in their training strategy before the first parameter update. Adding experiment logs, a skill library, and an evaluation agent improves downstream scores but still does not prompt systems to change approaches when the evidence turns against them.
A new study divides automated AI post-training into two capabilities: “execution capability,” which involves adjusting data, learning rates, and output formats within a fixed plan, and “strategy capability,” which involves switching among SFT, PEFT, RL, or multi-stage pipelines based on experimental evidence. The researchers analyzed [1,338 complete trajectories published by PostTrainBench](https://posttrainbench.com/), spanning seven benchmarks, four 1.7B-to-4B base models, and five agent interfaces. Each original run could use one H100 GPU for up to ten hours.
The results show that agents launched an average of 3.82 training runs and 13.8 evaluations per trajectory. Their overall score also rose from 10.41% for the base models to 23.0%, demonstrating that they could assemble datasets, fix code, and deliver checkpoints. Yet among 3,557 pairs of adjacent training experiments, only 74—or 2.1%—changed the training paradigm, data-source type, or stage structure. In Claude Code trajectories, 80.7% began with full-parameter SFT, while Codex CLI selected PEFT in 89.6% of cases. This suggests that strategy was driven more by agents’ default preferences than by the task. The [paper](https://arxiv.org/abs/2608.19072) was released on August 19.
The team also enhanced Claude Code with experiment logs, a post-training skill library, and an independent evaluation agent. With Qwen3-1.7B-Base fixed across three reruns—each using four A800 GPUs for ten hours—GSM8K improved by 12.6 points over the autonomous baseline, while HumanEval improved by 40.8 points. However, even when advised to switch to RL or add an SFT warm-up stage, the primary agent adopted only local changes involving reward shaping, formatting, and hyperparameters. Increasing inference tokens by two to eight times helped on easier tasks but produced almost no gain on AIME 2025.
This does not mean that changing strategies frequently is necessarily better: switching itself consumes compute, and AIME contains only 30 problems, so small score differences may fall within the range of variance. Engineering teams would benefit more from adding explicit strategy checkpoints, alternative hypotheses, and rules for reallocating the remaining budget, while measuring “finishing the plan” separately from “knowing when to abandon the plan.” The next question is whether this lock-in can be reproduced across different hardware budgets, model scales, and research tasks outside post-training.