RISE Synthesizes a Per-Token Teacher by Extrapolating the RLVR Update Direction, Eliminating External LLM Distillation

The RISE study, submitted on September 4, seeks to address the credit-assignment problem in RLVR, where rewards are provided only for complete answers, making it difficult to determine which tokens were actually useful. The method first performs a verifiable-reward update using a policy-gradient algorithm such as GRPO, then compares the updated checkpoint with a previous or EMA anchor. The researchers multiply this displacement by a coefficient greater than one and extrapolate in weight space or logit space to construct a “future policy,” whose full token distribution is then distilled back into the current model. Because the teacher is regenerated every round, the approach requires neither an external stronger model nor privileged context containing the correct answer.

The two implementations have different costs. The weight-space variant must materialize the extrapolated parameters and run an additional teacher forward pass during distillation. The logit-space variant instead geometrically mixes the log-probabilities from the anchor and updated models, caching the top 100 tokens along with a probability bucket for the tail. Both reuse rollouts already generated by RLVR, so they incur no additional sampling cost, but total wall-clock time is still 1.3× to 1.6× that of vanilla RLVR. The extrapolation coefficient is gradually decayed toward one to reduce the risk of overshooting the optimum as training approaches convergence.

The authors evaluated Qwen and OLMo models ranging from 1.7B to 8B parameters on mathematics, STEM, coding, and agent tasks. OLMo3-7B’s accuracy on AIME 2024 rose from 30.2% with GRPO to 46.9% with the logit-space variant. For Qwen2.5-3B, the weight-space variant increased the ALFWorld success rate from 75.0% to 84.4% and WebShop accuracy from 63.3% to 74.2%. The gains also held across mathematics experiments using three random seeds.

However, the “future teacher” cannot independently determine whether the update direction is correct; if the reward function contains exploitable flaws, extrapolation may amplify the errors. The paper observes that three directions explain about 87% of the variance in the training trajectory, but this is insufficient to guarantee that other models, longer training runs, or tasks without verifiable rewards will remain linear. The current results come from author-controlled experiments on small and medium-sized models and established benchmarks. Independent replication, stress testing under reward misspecification, and comparisons against stronger RLVR recipes using the same total compute remain necessary.