模型訓練
u-OPSD Trains Solely Through Internal Model Voting, Boosting Qwen3-8B’s Five-Benchmark Math Average by 10.7 Points
u-OPSD uses eight reasoning trajectories from the same model to produce a majority-vote pseudo-answer, then distills token distributions conditioned on that consensus into trajectories that disagree. Qwen3-8B’s average across five benchmarks in non-thinking mode rose from 43.57 to 54.31, though incorrect consensus and the cost of repeated sampling still limit its applicability.
A research team has introduced unsupervised on-policy self-distillation (u-OPSD), an approach that attempts to eliminate the need for ground-truth answers, external rewards, and stronger teachers during reasoning-model post-training. For each unlabeled problem, the method samples eight trajectories and parses their final answers. If at least half support the same answer, it selects the shortest agreeing trajectory as the pseudo-answer. Given this pseudo-answer, an identical but stop-gradient model produces a full vocabulary distribution over prefixes of trajectories whose answers differ. These per-token signals are then distilled back into the student using forward KL divergence. Training therefore focuses on problems for which the model already has a stable tendency but still produces mutually contradictory answers.
The authors used 30,000 problems from OpenThoughts, reading only the problem statements, and trained Qwen3 with LoRA. In non-thinking mode, Qwen3-4B’s average score across AIME 2024/2025, HMMT 2025, MATH500, and AMC 2023 increased from 40.96 to 49.49, while Qwen3-8B improved from 43.57 to 54.31. The two models outperformed OPSD trained with ground-truth solutions by 3.2 and 2.3 points, respectively. The advantage narrowed substantially in thinking mode: the 8B model’s score of 77.99 was virtually tied with OPSD’s 77.97. Experiments also showed that matching only the tokens actually sampled by the student discards a large amount of signal. Using the top-100 or full vocabulary distribution was more effective, but incurred higher inference and training costs.
The results suggest that engineering teams may be able to expand post-training with large collections of unlabeled problems that have fixed answer formats, without first building a verifier. However, this is not yet a general recipe for making models improve on their own: the experiments covered only Qwen3 and competition mathematics, and relied on short answers that could be extracted and normalized. Manual inspection still found errors in 13.3% of the pseudo-labels, and the authors have not yet reported multi-seed error estimates for every configuration. Future work should examine whether reliable soft consensus can be established for open-ended tasks and whether repeated self-distillation amplifies a model’s existing biases.