AI 評測與科學研究
AI Scientists Turn to Race Cars and Decks for Prospective Evaluation; Only 10 of GPT-5.2’s 166 Ideas Match Real-World Innovations
The new evaluation seals model-generated ideas before using season-specific technologies and professional tournament decks as answers revealed only later, reducing concerns about data contamination in retrospective benchmarks. The results suggest that models do not lack plausible-sounding ideas; the real bottleneck is filtering, combining, and identifying genuinely useful novelty.
Evaluating whether “AI scientists” can produce new discoveries typically faces several challenges: the lack of ground-truth answers, excessively long validation cycles, and the possibility that models have already encountered historical results. A new study uploaded on August 4 instead adopts a prospective, delayed-reveal real-world evaluation. Researchers asked models to propose race-car design ideas based on the 2026 Formula 1 regulations, then compared them with 40 innovations that appeared on actual cars during preseason testing. In another experiment, the models generated decks after the Magic: The Gathering card pool was updated, and those decks were compared with 19 decklists from a Pro Tour event. Public results independently produced by competitors thus became observable answers after the fact.
The F1 experiment covered six models. The best-performing model, GPT-5.2, proposed 166 ideas across multiple runs and matched 10 of the 40 real-world innovations. After normalizing for output volume, it matched six innovations per 100 ideas. Across all models, 19.4% of ideas judged compliant with the regulations corresponded to actual innovations, while only 0.9% were clearly noncompliant. This suggests that rule checking can narrow the search space but is still insufficient to identify the most valuable solutions. Of the 519 potential matches initially flagged by an LLM, 89% were downgraded to non-matches after human review, also revealing how semantic similarity can mistake superficial overlap for an identical mechanism.
The deck-building experiment generated 108 decks in total. Gemini 3 Flash’s best individual deck recovered five of the seven new cards in one third-place deck. Notably, adding tools substantially improved main-deck legality—for example, GPT-5.2 rose from 11% to 100%—but reduced the new-card discovery rate for every model. This indicates that tool-based frameworks may strengthen constraint compliance while narrowing exploration. Ultimately, both domains are only proxy tasks for scientific discovery, and matching a human solution does not prove that a model derived it independently. However, the code, prompts, outputs, and data have been made public, giving researchers an opportunity to further examine cutoff dates, evaluator bias, and whether the same method can be transferred to scientific problems that support validation through simulators or experiments.