模型訓練與推論效率
Three-Tier Self-Routing Cuts Output Tokens by 41% for a 1.5B Reasoning Model on MATH-500
A new study turns NoThink, Short, and Long into the model-generated first token, allowing it to allocate a reasoning budget for each problem without a separate router. On MATH-500, the method maintains accuracy close to the base model while cutting tokens by 76% on the easier GSM8K benchmark.
A research team added three-tier self-routing to [DeepSeek-R1-Distill-Qwen-1.5B](https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B). After receiving a problem, the model first generates `NoThink`, `Short`, or `Long`, then uses a maximum output budget of 1,024 tokens, 3,000 tokens, or no fixed limit, respectively. This choice is updated jointly with the main model using GRPO, with no separate classifier. To prevent the policy from rapidly collapsing to a single mode, training also incorporates a forced-exploration warm-up, a mode-proportion balancing term, and reward curves for different length ranges.
The [paper](https://arxiv.org/abs/2608.20256) trains on MATH-lightEval, using four H100 GPUs for 90 update steps and repeating the experiment with three random seeds. On MATH-500, the self-routing model achieves 0.782 accuracy, slightly below the untrained baseline’s 0.796, while reducing average output from 4,743 to 2,810 tokens—a 41% decrease. In zero-shot transfer to [GSM8K](https://github.com/openai/grade-school-math), average length falls from 1,930 to 459 tokens, while accuracy declines from 0.831 to 0.781. On the more difficult AIME 2024/2025 benchmarks, the model mostly selects Long, reducing tokens by only about 13%. This suggests that the gains primarily come from identifying problems that do not warrant extended reasoning, rather than raising the model’s problem-solving ceiling.
The key engineering insight is that the reasoning budget can become a discrete action within the policy itself, eliminating the added latency of running a difficulty predictor first. However, the current evidence covers only a single 1.5B model and mathematical tasks, and the mode limits were tuned to the length distribution of that data. The paper also acknowledges that its validation method, which does not enforce the limits, may overestimate short-mode accuracy. The next step is to determine whether the same routing stability holds for tasks with more irregular length requirements, such as code and tool-using agents, and whether real-world GPU latency and throughput improve in proportion to the reduction in tokens.