AI 評測與科研代理
TruthInsightBench Tests Scientific Research Agents by Hiding Paper Conclusions; All Four Systems Plateau at Around 60 Points
The new benchmark provides only research objectives and frozen datasets, without revealing the original papers’ conclusions or predetermined analysis paths. Although all four coding agents can execute analyses, they routinely omit control experiments, robustness checks, falsifiability criteria, and cross-dataset validation.
TruthInsightBench attempts to separate “reproducing known results” from “forming credible discoveries from data.” Researchers selected peer-reviewed studies from ten scientific disciplines to create 40 blind evaluation tasks. Agents receive only neutral research objectives, frozen datasets, and an executable environment; they cannot see the source papers’ conclusions, expected numerical results, or analysis procedures. Each agent must independently decide which phenomena to examine and which claims to make, then submit code, analysis artifacts, and a report.
Rather than comparing submissions against a single reference answer, the evaluator assesses each finding proposed by the agent. Six dimensions—evidence auditability, robustness, control testing, cross-dataset generalization, novelty, and falsifiability—are divided into 29 criteria that must be supported by executable artifacts. Tests mentioned only in the report but not actually run receive no credit. The judge is a fixed quantized model with reasoning disabled. Each criterion is first scored as 0, 0.5, or 1, then aggregated using deterministic rules to reduce evaluation cost and run-to-run variability.
When all systems used the same frozen foundation model, Claude Code, OpenScience, Codex CLI, and DeepSeek Harness achieved average scores ranging from just 58.40 to 60.27. Pairwise tests across the 40 tasks found no statistically significant leader. Their evidence-auditability rates approached 80%, but they rarely tested alternative methods, perturbations, negative controls, or validation with external data. This suggests that current agents are better at completing an analysis and writing it up than at determining whether an apparent signal represents a novel phenomenon, a numerical artifact, or dataset bias.
From an engineering perspective, the benchmark can serve as a regression test for scientific research agents, but a score of 60 should not be interpreted as a measure of general scientific research capability. The experiment used only a single run and one shared foundation model. Agent prompts, decoding parameters, tool restrictions, and complete raw execution traces were also not released with the public package. Future evaluations should cover multiple models and random seeds, while also auditing the consistency of the fixed judge model itself.