AI 研究與訓練
VIGOR Dynamically Allocates RLVR Rollouts Based on Reward Variance, Cutting Sampling for Mathematical Reasoning Training by Up to 2.3×
A new study replaces the fixed number of reasoning trajectories generated for every problem with a strategy that concentrates additional compute on samples exhibiting greater reward disagreement. The authors report higher rollout efficiency on mathematics and coding tasks, but have not released their code, while actual GPU time and scalability to larger models remain to be verified.
Reinforcement learning with verifiable rewards (RLVR) has become an important approach to training reasoning models. The commonly used GRPO method, however, generates a fixed number of responses for every problem in a batch and estimates gradients using relative rewards within each group. This setup can spend large numbers of tokens on samples for which every response is either correct or incorrect. Such groups exhibit almost no reward variance and provide little learning signal.
Researchers at the University of California, Los Angeles proposed VIGOR (Variance-Guided Online Rollout Allocation). The method first generates a small number of rollouts for every problem in a batch and then calculates each problem’s group reward variance. In successive rounds, it allocates the remaining budget to problems with the highest variance—those for which the model remains near the decision boundary—instead of increasing the sample count uniformly. This process occurs within the current batch, requiring neither historical difficulty estimates maintained across steps nor a large candidate pool whose unused results must be discarded.
The authors derive the relationship between reward variance and gradient magnitude from the RLVR gradient, and analyze the theoretical speedup over fixed allocation under the assumption that variance follows a Pareto distribution. In experiments, they trained Qwen2.5-3B on mathematical reasoning and evaluated Qwen3-8B on LiveCodeBench v6. VIGOR required up to 2.3× fewer rollouts to reach a specified mathematics accuracy. On the coding task, it used 1.49× fewer rollouts to reach GRPO’s final full-pass rate while improving the average test pass rate by 3.4 percentage points. The results were obtained using three random seeds, but fewer rollouts do not necessarily translate into proportional end-to-end savings because iterative allocation, synchronization, and variable-length generation also introduce overhead.
Engineering teams should next examine wall-clock time, total generated tokens, GPU utilization, and whether the method remains stable with sparse or noisy rewards. The paper also currently provides no reproducible codebase. It has yet to be demonstrated whether variance remains a reliable basis for sampling with larger models, non-binary scoring, or proxy tasks that cannot be verified exactly.