AI 代理與自動化研究
Iris Drives ML Experiments with a Revisable Knowledge State, Achieving a 64.9% Medal Rate on MLE-Bench in 12 Hours
Rather than repeatedly modifying code along a tree of candidate solutions, Iris first investigates critical unknowns and then organizes evidence across experiments into claims that can be added, updated, or retracted. The paper reports a higher medal rate on the full MLE-Bench in half the time allotted to its direct baseline, but the system’s source code and complete execution traces have not been released.
Autonomous ML engineering agents typically treat each experiment as a candidate solution: modify the code, run an evaluation, and choose the next branch based on the score. The newly proposed Iris instead centers its process on an “information state,” dividing the workflow into inquiry and revision. Inquiry creates a local action plan based on current unknowns and permits epistemic actions that do not directly modify the best solution, such as checking for data leakage, feature distributions, or evaluator behavior. Revision organizes results from multiple experiments into claims with defined scopes and statuses, allowing them to be updated or retracted when contradicted, rather than merely feeding lengthy logs back into the context.
The researchers powered Iris with Claude Opus 4.6 and gave it 12 hours for each of the 75 Kaggle competitions in MLE-Bench. Across three runs, it achieved an average any-medal rate of 64.9% and a gold-medal rate of 39.1%. Every task produced a valid submission, and 76.0% of submissions surpassed the human median. The AIBuildAI baseline using the same backbone achieved 63.1% and 25.8%, respectively, with a 24-hour budget. Iris also outperformed controlled Codex and Claude Code baselines on four cross-domain tests. For example, it reached 67.7% accuracy on BrowseComp; when tasked with designing a post-training recipe starting from Qwen3-1.7B, it achieved 47.0% on HumanEval and 68.4% on GSM8K.
The key engineering contribution is not the addition of another planning role, but the transformation of “what is known, where the evidence applies, and which conclusions are no longer valid” into persistent, actionable state. In an ablation study on 15 small-data tasks, removing information management reduced the medal rate from 66.7% to 53.3%, while removing the adaptive action topology lowered it to 40.0%. However, each full MLE-Bench task used 23 vCPUs, 234 GB of RAM, and one 144 GB H20-3e GPU, making full reproduction very costly. The paper also does not provide the Iris source code. MLE-Bench has currently paused new leaderboard submissions and documented multiple data-preparation issues, so these self-reported results still require confirmation through public execution traces, detailed cost disclosures, and independent reruns.