模型訓練與推理
TTPO Routes Positive and Negative Trajectories by Majority Vote, Enabling Models to Update at Test Time Without Ground-Truth Answers
Rather than treating the majority-vote answer as ground truth, TTPO distills trajectories that agree with the consensus and uses grouped reinforcement learning to penalize high-confidence disagreements. The authors report that Qwen3-1.7B’s Avg@12 across three mathematics competitions rises from 38.0 to 45.2, though this remains a transductive setting in which parameters are updated directly on the test problems.
A research team from Zhejiang University and Alibaba has introduced Test-Time Policy Optimization (TTPO), an approach that updates a model’s policy at test time using multiple reasoning trajectories generated by the model itself, without access to ground-truth answers. Conventional self-training treats the majority-vote result as a pseudo-label. On difficult problems, however, if the majority answer is itself incorrect, distillation can propagate that error throughout the entire reasoning chain. TTPO’s central observation is that the signal is asymmetric: even when the majority vote is wrong, trajectories that disagree with it are usually not correct either, so they can still provide a relatively reliable negative signal.
For each problem, TTPO first samples K trajectories and clusters them by final answer. Trajectories that agree with the largest cluster enter an on-policy self-distillation branch, which uses forward KL to propagate token-level signals while reducing the weight of positions that have already converged. Disagreeing trajectories enter a grouped RL branch that penalizes only incorrect tokens in which the model has high confidence, avoiding the wholesale removal of locally reasonable steps. Majority voting is used only for routing and is not assumed to produce the ground-truth answer.
On AIME25/26, HMMT25/26, and BRUMO25, the authors report that label-free TTPO achieves Avg@12 scores of 40.1, 58.6, and 62.6 with Qwen3-1.7B, 4B, and 8B, respectively, matching or slightly outperforming OPSD with answer supervision. Under pure test-time training, the 1.7B model improves from 38.0 to 45.2 across the three benchmarks; the 4B model’s score of 61.1 also slightly exceeds the unupdated 8B model’s 60.7. The released code includes LoRA and vLLM support, along with scripts for four-GPU setups.
One limitation is that the evaluation uses Avg@12, so the cost of additional rollouts and parameter updates cannot be compared directly with single-pass inference. Moreover, training on the test problems themselves constitutes transductive adaptation and does not imply equivalent gains on entirely new distributions. Cross-competition transfer provides preliminary evidence, but the current tasks are concentrated on mathematics problems with extractable short answers. Engineering teams should further investigate whether erroneous consensus accumulates in open-ended tasks, as well as rollback, isolation, and cost-control mechanisms for online updates.