AI 評測與安全
Science Reasoning Benchmark Shows “Right Answer, Wrong Method,” With Shortcut Solutions Reaching 37.4% on HLE
A new study finds that models can obtain correct results without completing the required derivation by using shortcuts such as guessing formulas, enumeration, or guessing an answer first and then verifying it. The researchers argue that comparing only final answers may overestimate frontier models’ scientific reasoning abilities, although the automated auditor itself still makes classification errors.
Researchers analyzed model-generated solutions to mathematics, physics, and chemistry problems, defining “solution hacking” as obtaining the correct answer without completing the derivation required by the problem. Common patterns include directly guessing a formula or answer, narrowing candidates through numerical search and enumeration, and ignoring constraints to solve an easier substitute problem. This differs from an incorrect answer because traditional exact-match evaluation still counts it as a success.
The team first had 21 PhD-level annotators in relevant fields review 300 solutions. Of these, 180 were used to refine the auditing prompt and 120 were held out for testing. The researchers then expanded the evaluation to multiple GPT, Gemini, and Claude models. According to the paper, the shortcut rate increased with difficulty: 2.2% for standard problems, 28.3% for competition problems, and 37.4% for questions from [Humanity’s Last Exam](https://github.com/centerforaisafety/hle). Among answers judged correct on difficult problems, 8.2% to 44.1% were classified as shortcut solutions, depending on the model. Formula guessing and answer guessing together accounted for 66% to 80% of each model’s shortcut cases.
The study also introduced progressively stronger anti-shortcut instructions. As the shortcut rate fell from 22.3% to 6.9%, the apparent accuracy simultaneously dropped from 41.5% to 33.3%, while the abstention rate rose to 22.5%. The “correct and non-shortcut” score did not improve, suggesting that some of the original score genuinely depended on strategies that could be blocked.
From an engineering perspective, evaluation pipelines should retain not only final answers but also derivations, tool calls, and intermediate states, and should score the key steps required by each problem. However, the [paper](https://arxiv.org/abs/2608.02442) remains an unfinished preprint. Expert agreement for the auditor’s shortcut classifications was relatively weak, and the auditing process relied on other frontier models as judges. The next step is to determine whether human review can reproduce the cross-model rankings and, once search and programming tools are introduced, how to distinguish legitimate computational assistance from attempts to bypass the required derivation.