模型評測與推論
Rankings Reverse for the Same Models as Token Limits Change, Four-Model Test Records 56,476 Inferences
A new study increased the generation limit for four open-weight models from 64 to 4,096 tokens and found that the leading model changed on all three reasoning benchmarks. Even after truncated outputs were excluded, 3% to 19% of questions changed from correct to incorrect as the budget increased, showing that a single token setting is insufficient for deployment decisions.
Model leaderboards often compress performance under a single, fixed inference configuration into one score, but the accuracy of different models does not necessarily improve in step as generation budgets increase. Researchers at the Federal University of Santa Catarina in Brazil compared Llama 3 8B, Qwen3 32B, Llama 3.3 70B, and [gpt-oss-20b](https://huggingface.co/openai/gpt-oss-20b) on GSM8K, MATH-500, and GPQA-Diamond, using token limits of 64, 128, 256, 512, 1,024, 2,048, and 4,096. In total, they ran 56,476 deterministic inferences at a temperature of zero.
The [paper](https://arxiv.org/abs/2608.12150) found ranking reversals on all three benchmarks. On GSM8K, for example, Llama 3.3 70B led at the 256-token limit with 62.4% accuracy, while gpt-oss-20b achieved only 48.6%. At 4,096 tokens, however, gpt-oss-20b rose to 94.9%, surpassing Qwen3 32B at 91.7%. Llama 3.3 70B led MATH-500 at intermediate budgets, while the apparent ranking at 4,096 tokens placed gpt-oss-20b first, although the difference was not statistically significant. GPQA-Diamond contains only 198 questions, and its top ranking likewise shifted from the 8B model to the 70B and 20B models, so small-sample differences should not be overinterpreted.
Some of these changes resulted from truncated outputs. In the 4,096-token GPQA evaluation, 59.7% of Qwen3 32B responses still stopped because they reached the length limit, compared with only 0.5% for Llama 3.3 70B. The researchers therefore also compared the shared subset of questions for which every model completed its answer. On that subset alone, Qwen3 32B would replace gpt-oss-20b as the top-ranked model on MATH-500 at 4,096 tokens. In other words, “finishing within a limited budget” and “answering correctly once finished” are two distinct deployment metrics.
Truncation cannot explain all the results. After truncated trajectories were removed, approximately 3% to 19% of questions for each model and dataset still changed from correct to incorrect as the budget increased. Moreover, 86% to 94% of such questions occurred in only one model. The study’s XGBoost router used both textual features from the question and the token budget as inputs, but in cross-domain tests it recovered only 14.1% of the gap to an ideal oracle, indicating that budget effects do not generalize easily. In practice, evaluation reports should publish complete curves for accuracy, truncation rate, cost, and latency. Model gateways should also determine limits through workload-specific testing instead of assuming that more inference tokens are necessarily better.