代理評測與推理效率
R³-Bench Gives Six Problems a Shared Reasoning Budget; Models Fall Below Their Own Offline Upper Bound in 71 of 72 Test Settings
The new evaluation no longer provides a separate allowance for each problem. Instead, models must allocate tokens or tool calls across six problems. Even though all six models can solve the problems individually, under a shared budget they generally fail to direct compute toward the problems with the highest expected payoff.
R³-Bench organizes mathematics, competitive programming, and abstract reasoning problems into 50 six-problem contests per domain. Each contest contains exactly three low-resource problems, two medium-resource problems, and one high-resource problem. In tool-free mode, the shared budget is measured in output tokens; in agent mode, it is measured by shell, testing, and other tool actions. Models must therefore not only solve problems, but also decide when to continue reasoning, verify an answer, switch problems, or stop.
The researchers first had each model answer problems individually at six budget levels, rerunning every condition five times to build an empirical response curve showing how many resources had been invested when the model succeeded. The authors then formulated an offline oracle as a multiple-choice knapsack problem: within the total budget, it retrospectively selects among configurations in which the model had already demonstrated success. Across the six models’ 72 primary test cells, the oracle was never worse than the actual contest run and was strictly better in 71. Under moderate pressure, even a simpler equal-allocation strategy outperformed some models’ self-directed planning.
Trajectory analysis shows that the issue is not merely that models fail to cover every problem. Five models performed substantive work on more than half of the problems, but only one adjusted its strategy based on new evidence and the remaining budget in more than half of cases. Under high pressure, a common failure mode was exhausting the budget on other problems first; under lower pressure, models often stopped too early despite having made progress. Adding fixed-schedule instructions improved six of the nine model–domain combinations, but no single strategy consistently led across domains.
The implication for coding agents handling multiple tasks concurrently is direct: single-problem benchmark scores cannot predict actual throughput under concurrent workloads. Schedulers may need to explicitly estimate remaining cost, probability of success, and opportunity cost. One limitation is that the oracle relies on post hoc success records rather than representing a deployable strategy. Problem difficulty is also stratified using the output length of a reference model. In addition, the open-source evaluator omits hidden tests for programming problems and some verification assets, so full reproduction still requires researchers to supply the missing components themselves.