EarthVerse Tests 405 Disaster Investigations: Agents Average Over 80% Correct, but Near-Complete Task Success Remains Just 34.81%

EarthVerse advances scientific-agent evaluation from single-question answering to replayable investigation workflows. The benchmark contains 405 tasks derived from 199 documented events across 19 types of natural disasters. Each event package may include reports, time series, remote-sensing products, and derived data. Agents must independently identify compatible evidence, choose tools, handle units and time windows, compare conflicting sources, and preserve the relationships between calculations and sources in their answers. Altogether, the data packages contain 6,709 files and 6,397 provenance records.

The scoring system does not require agents to reproduce a single prescribed sequence of tools. Instead, it decomposes answers into fine-grained units and separately evaluates evidence selection, calculations, source reconciliation, and physical interpretation. Among the 25 systems tested, Claude Fable 5 with Claude Code achieved the highest answer-unit accuracy at 84.65%. GPT-5.6 Sol with Codex posted the highest Strict@95 score, but that figure was still only 34.81%. This gap indicates that agents can often complete most local steps yet omit a time window, unit conversion, alternative mechanism, or citation, invalidating the full chain of conclusions. More turns do not guarantee better results: OpenResearcher used an average of roughly 539,000 tokens per question in the paper, yet could still carry outdated values into its final answer.

These figures should not be treated as a bare model leaderboard. Some models used the shared EarthVerse controller, while Codex, Claude Code, and research frameworks retained their own execution loops, meaning the results actually compare model–harness combinations. Raw imagery was also used directly in only a small number of tasks. Scoring additionally relies on a fixed GPT-4.1 judge, while the offline event packages improve reproducibility at the cost of real-time web exploration. For developers of scientific agents, the most direct engineering takeaway is to maintain a “claim–evidence” graph in which every value carries its source, unit, time window, and transformation history, and to implement the final review as an item-by-item coverage check rather than merely polishing the report’s prose.