評測／AI 代理
DiG-bench Tests Proactive Discovery Through Hidden Game Rules: Opus 5 Solves 50 of 70, While General-Purpose Agent Frameworks Bring No Gains
DiG-bench uses 70 entirely new text-based games that require models to experiment independently and infer unknown rules and victory conditions. Opus 5 leads by solving 50, but existing agent frameworks do not consistently outperform the base model loop on the most difficult problems.
DiG-bench turns “discovery” from a knowledge question-answering task into a controlled interactive experiment. Each game is a partially observable environment represented by short Unicode strings, with its rules, objectives, and state transitions kept hidden. The agent may choose only one legal action per turn. Some games also provide a creative mode whose actions do not count toward the official step limit, allowing the system to design experiments, eliminate hypotheses, and then return to the main task. The 70 hand-designed games are divided into seven difficulty levels: 21 are public for development, while 49 are held back for contamination-resistant evaluation.
In a full evaluation using the base harness, Opus 5 solved 50/70 games, Gemini 3.1 Pro solved 18, and Qwen3.6-27B solved only one. Even selecting the best-performing model for each game yielded just 57 solved games overall and 9/20 across the two highest difficulty levels. A controlled experiment pinpoints the bottleneck more clearly: when given concise versions of the rules directly, Gemini improved from 18/70 to 69/70, indicating that the main failure point is indeed rule discovery rather than planning once the rules are known. General-purpose agent wrappers—including Claude Code, Codex, Kimi Code, PRO-LONG, and Prime Agent—also showed no consistent gains on the hardest games. Having more tools does not necessarily mean an agent will propose a more informative next experiment.
The implication for scientific agents and automated R&D systems is straightforward: a harness should record competing hypotheses, expected observations, and counterexamples, then select actions according to information gain—not merely extend the context window or add more shell tools. However, most model-game combinations were run only once, and the models did not have identical cost limits, context lengths, or serving configurations. Human samples were used only to confirm that the games were solvable. Engineering teams should next focus on rerun results from public APIs, variation across seeds, and whether memory and planning systems built specifically for experimental design can genuinely break through the advanced private tasks.