代理工具
AI4AI-Bench Tasks Agents With Rewriting Ten Types of Training Algorithms; Best System Averages Just 0.250
AI4AI-Bench evaluates hyperparameter tuning separately from genuine changes to learning methods, requiring coding agents to submit training code that can be rerun from scratch. Across 290 trials involving six systems, more than 40% underperformed the original algorithms, while increased reasoning effort mainly made agents more willing to modify core training mechanisms.
Einsia has released AI4AI-Bench, which uses ten frozen research repositories to test whether coding agents can improve “how models learn,” rather than merely extending training, scaling up models, or tuning hyperparameters. The tasks cover supervised fine-tuning, multi-turn agent reinforcement learning, online distillation, reward modeling, preference optimization, reinforcement learning for diffusion models, machine unlearning, graph diffusion, weight merging, and one-shot pruning.
For each task, an agent receives four hours and one NVIDIA B300 to inspect the code, modify the training pipeline, and repeatedly review inexpensive proxy metrics. When the time limit expires, only the source-code patch is retained. Organizers apply the patch in a fresh container, train from scratch for up to 12 hours, and then use fixed evaluators hidden during development to assess up to three checkpoints. This design rules out illusory gains caused by reusing weights or caches from the exploration phase.
Because the ten tasks use different raw scales, researchers map an uninformative model to 0, the repository’s original algorithm to 0.1, and the default best-known result to 1.0. Across six systems and 29 combinations of models and reasoning settings, the 290 evaluation cells achieved an average score of 0.166. The best system, Claude Opus 5, averaged 0.250, while the best individual configuration scored 0.288. A total of 124 evaluation cells scored below 0.1, meaning the modifications made performance worse.
More notably, among the 263 modified submissions, 141 changed only the compute budget, checkpoints, capacity, or hyperparameters without altering the objective function, supervision signal, data, or update rule. Submissions that genuinely modified the learning mechanism averaged 0.226, compared with just 0.126 for the rest. Increasing reasoning effort raised the proportion of such submissions from 8% to 64%, but their average score increased only from 0.094 to 0.196.
Engineering teams can use the publicly available Apache 2.0-licensed harness to replay patches, but the official configuration requires costly, long-running training on B300 GPUs, and there is currently no hosted blind-evaluation service. Local results produced after the benchmark’s release can receive only unofficial receipts. Ten repositories are also insufficient to represent all algorithmic research, so AI4AI-Bench is better treated as a reproducible capability probe than as evidence that “recursive self-improvement is no longer viable.”