AI 代理評測
AI4AI-Bench Asks Agents to Rewrite Ten Types of Training Algorithms; the Best System Averages Just 0.250
AI4AI-Bench has coding agents modify complete research codebases on a single B300, after which an isolated evaluator trains and scores their submissions from scratch. Across 290 experiments involving six systems, more than 40% of the results underperformed the original algorithms, and most submissions did not even change how the models learned.
Existing evaluations of “AI doing AI research” often conflate data collection, hyperparameter search, and code optimization, making it difficult to determine whether an agent has actually designed a new learning method. AI4AI-Bench therefore freezes ten research codebases spanning supervised fine-tuning, multi-turn agent reinforcement learning, online distillation, reward modeling, DPO, reinforcement learning for diffusion models, machine unlearning, discrete graph diffusion, weight averaging, and one-shot pruning. Agents must directly modify the objective function, update rules, data pipeline, or training loop.
For each task, an agent receives four hours and one NVIDIA B300. During that period, it may repeatedly use inexpensive proxy metrics, but its final submission can contain only source code—not weights, caches, or runtime state. The organizers then run the submission from scratch for up to 12 hours in a clean container and score it using a fixed evaluator inaccessible to the agent. To combine different metrics, the benchmark assigns an uninformative model a score of 0, the original codebase’s algorithm a score of 0.1, and the theoretical optimum a score of 1.0.
The study tested six model–agent framework combinations, including GPT-5.6, Claude 5, and Kimi K3, producing 29 configurations and 290 experimental cells. The average score across all experiments was 0.166. When aggregated by system, Claude Opus 5 ranked highest with an average of 0.250, while the best individual configuration was Opus 5 at medium reasoning effort, averaging 0.288. Of the 290 attempts, 124 scored below the original algorithm’s 0.1 baseline. Among the 263 submissions that actually modified the code, 141 only changed the compute budget, checkpoints, capacity, or hyperparameters without altering the learning procedure. Submissions that genuinely modified the algorithm averaged 0.226, compared with just 0.126 for the rest.
This provides a clearer operational distinction between “an agent can write training code” and “an agent can propose an effective training algorithm.” It also shows that a larger reasoning budget mainly increases an agent’s willingness to attempt deeper modifications, without guaranteeing success. The benchmark’s limitations include its extremely high cost, the restriction of each task to a single B300 and a fixed time window, and a scoring scale that depends on researcher-defined uninformative and optimal reference points. It also provides no human-researcher baseline, so it cannot support claims that agents are either approaching or far from human-level algorithm research capabilities.