Reconstruction Hides Papers and Provides Only Older References; Multi-Model Review Raises Research-Idea Match Rate from 14.6% to 35.6%

Reconstruction attempts to turn the question of whether a model can propose research ideas into a more verifiable blind test. The system hides the title and abstract of a seed paper and provides only references published before the paper’s publication date. The model must generate five hypotheses, which are then evaluated against the actual idea by language-model judges that did not participate in generation. The dataset covers ICML 2026 oral papers and five Nature-family research domains. After date parsing, reference deduplication, and completeness filtering, 643 papers remained for the final multi-agent comparison.

The leakage-prevention process includes strict temporal cutoffs, anonymized reference IDs, a frozen bibliography for each paper, and a ban on web searches during generation. Match rates for seven individual models were generally only 3% to 15%, showing that even when relevant literature is present in the context, reconstructing a specific problem and contribution from the research landscape remains difficult.

In the multi-agent version, each of the four best-performing models first generates five candidates, which are then grouped into five sets by position. The other models cross-review the candidates using the same anonymized literature. Candidates compete through three rounds of Swiss-system pairings; a model that proposed a candidate must recuse itself from judging that candidate, and the presentation order is reversed within each pairing to reduce self-evaluation and position bias. Only the five winning hypotheses are ultimately submitted to an independent Match judge. The overall match rate is 35.6%, roughly 2.4 times the 14.6% achieved by the best single model after judge alignment. Multi-agent results across domains range from 22.9% to 41.6%.

This does not mean multi-agent collaboration itself accounts for the entire improvement. The single-model setup retains five candidates, whereas the multi-agent setup selects five from a pool of 20 and consumes more tokens; the judge pools are also not identical. Models may have encountered the seed papers during pretraining, so temporal cutoffs can prevent prompt leakage but cannot rule out parametric memorization. The authors have also not yet provided inter-rater agreement among human judges, a compute-matched best-of-20 baseline, or a public implementation. Developers of research agents should view this as a useful benchmark design proposal, not as evidence that automated scientific discovery has been validated.