AI 研究
ExplorationBench Uses a Sandbox with Counterintuitive Rules to Probe Agents’ Exploration and Knowledge Application
Researchers let ten AI systems probe unfamiliar rules, then disabled tools to test how well they applied what they had learned. They found that performance can still decline after multiple rounds of exploration. The test questions are not public, and some scores differ between the project page and the paper.
A team from Fudan University, Tencent Hunyuan, and Tsinghua University submitted ExplorationBench on September 24, advancing agent evaluation toward “collecting evidence independently, then applying new rules.” The study uses two executable sandboxes to test ten AI systems, examining whether models can revise what they already believe and whether the knowledge they acquire supports unfamiliar tasks. [Paper](https://arxiv.org/abs/2609.30199)
The two environments are designed to make familiar operations fail. AlienCode has 31 discovery targets. For example, it XORs integer literals with 27, so `EMIT(100)` outputs 127. AlienLogic modifies 24 reasoning rules. Each environment has 70 held-out test questions, with answers checked by an interpreter or proof checker. For propositions designated as unprovable, a model must correctly refuse to prove them to receive credit. This makes scoring directly verifiable and reduces the variability introduced by language-model judges. [Project overview](https://www.explorationbench.com/)
The experiment starts from the same manual containing errors and the same set of fixed examples. Agents explore over four rounds, choosing programs or proofs as probes and receiving feedback from the environment. At the end of each round, the conversation is also copied, tools are disabled, and the agent is asked to state the rules and answer questions; the test copy is then discarded. Weights are not updated during this process, so the benchmark measures the ability to acquire and use knowledge within a single context. It does not yet cover long-term learning across sessions. [Authors’ explanation](https://www.explorationbench.com/blog/)
According to version 1 of the paper, across three independent explorations in AlienCode, Claude Opus 5 achieved a best final score of 87.6%, an average of 72.5%, and a worst score of 49.0%. This spread shows that the best result alone does not represent consistent performance. In six of the 30 code exploration trajectories, the final score was at least three percentage points lower than the score in an earlier round. The main leaderboard uses Best@3, the best result across three runs. Engineering evaluations should also examine the mean, worst result, and exploration cost. [Results and evaluation method](https://arxiv.org/html/2609.30199v1)
For developers of research agents, these findings suggest testing whether retaining supporting evidence and applicability conditions for rules in memory can reduce cases where later reasoning overturns correct discoveries. This is an engineering hypothesis prompted by the experiment; its effectiveness still needs to be validated in real workflows.
Two verification limitations remain. The project page lists AlienCode’s top score as 89.0%, unlike the 87.6% in version 1 of the paper, so the figures should not be conflated. The website also states that the test questions are not public and that teams evaluating external models must contact the researchers to have them run the benchmark. The work offers an evaluation design for further study, but the public materials are not yet sufficient for third parties to reproduce it fully. Follow-up work should track alignment between score versions, release of evaluation materials, and stability under more repeated trials at the same budget. [Official evaluation page](https://www.explorationbench.com/)