RECLAIM tests research agents on 100 papers; full-materials group reaches a 41% reproduction rate

The RECLAIM research team submitted a new preprint on September 23, setting the goal of its research-agent benchmark as reproducing specified experimental results from papers. The benchmark covers 100 NeurIPS 2025 papers and requires agents to handle the entire process themselves, from installing packages and resolving dependency conflicts to running experiments, while providing evidence that can be audited. [Paper](https://arxiv.org/abs/2609.28850), [project overview](https://github.com/mithils3/reclaim)

Tasks are divided into three groups based on the materials made public by the original authors: Run provides code, data, and weights; Retrain lacks weights, so the agent must train the model again; Reimplement lacks code, so the agent must reconstruct the method from the paper. Each task has a fixed target metric, scope, and evaluation procedure, with a GPU budget of up to 96 H100-equivalent hours. Agents can see the target value, but do not receive the experimental configuration that produced it, so they still have to read the paper and design an experiment. [Task specifications](https://github.com/mithils3/reclaim#match-target-and-tolerance)

The scoring process separates agents’ self-reported claims from evaluation. A fixed Claude Sonnet 5 review model examines execution evidence and turn-by-turn traces, assigning a score from zero to ten; a score of eight or higher counts as a reproduction. Serious evidence-integrity issues result in a score of zero. Public traces preserve tool commands, standard output, and error messages, allowing readers to trace how the results were produced and review the rationale for the evaluation. [Trace viewer](https://reclaim-traces.vercel.app/)

The paper reports that DeepSeek-V4-Flash succeeded on 14 of 34 Run tasks, about 41%, and 9 of 33 Retrain tasks, about 27%. The best performer in the Reimplement group was Muse Spark 1.2, which succeeded on 5 of 33 tasks, about 15%. Because the three groups contain different papers, these gaps cannot be interpreted directly as the causal effect of removing the code or weights from the same paper. [Results and methods](https://arxiv.org/html/2609.28850v1)

For developers of research tools, the engineering lesson is to tie evaluation criteria to concrete artifacts: target values must be linked to measurement evidence, and experiments must fit the specified scope. Data fields record the comparison direction, tolerance, and file containing the evidence, so another reviewer can recheck the evaluation; partial reproduction, unverifiable results, and disqualification are also labeled separately. The repository provides a frozen data split, prompts, scoring rules, and execution tools, offering a starting point for testing improvements to agent workflows. [Public implementation](https://github.com/mithils3/reclaim)

The results have clear limitations: each model-and-paper combination was tested at most once, and Claude, GPT, and Gemini were not included as agents under evaluation. The study also did not complete a comprehensive human reproduction effort on the same cluster, so not all discrepancies can be attributed to the agents. Future work should add repeated trials and a human baseline to establish whether improvements can reliably translate into verifiable research results. [Study limitations](https://arxiv.org/html/2609.28850v1#S6)