模型訓練與蒸餾
TIDE Corrects Teacher–Student Mismatch, Raising Qwen3 Reasoning Avg@8 from 6.9% to 20.3%
TIDE no longer treats high token-level agreement as direct evidence of successful distillation. Instead, it separately addresses student overgeneration and the omission of teacher-preferred tokens. Under the severe-mismatch setting, the method also reduces the average response length by 3.6×, although reproducing the full training run still requires eight GPUs and the authors have not released a checkpoint.
Released on August 10, TIDE identifies a “degenerate agreement” loophole in existing on-policy distillation (OPD): after the student model enters a repetition loop, the teacher may predict the same continuation from the student-generated prefix, causing the local KL divergence to approach zero even though the overall answer has become meaningless. In one case presented in the paper, a repetitive prefix reduced the teacher–student KL divergence by 63×; after training, the share of rollouts containing loops increased from 16.8% to 48.4%.
The researchers therefore divide mismatch into two directions. For “student-excess” tokens—outputs to which the student assigns high probability while the teacher nearly rejects them—TIDE replaces a potentially unbounded log-ratio penalty with bounded Hellinger shaping, preventing a small number of extreme tokens from dominating the gradient. For “student-deficit” tokens, the method directly inspects the teacher’s top-K distribution and restores the missing probability mass, without waiting for the student to sample them with extremely low probability. A batch quantile gate retains only the positions with the most severe mismatch, excluding already aligned tokens from the loss.
Across Qwen3 teacher–student pairings evaluated on AIME 2024, AIME 2025, and AMC 2023, Avg@8 under the severe-mismatch setting increased from 6.87% with standard OPD to approximately 20.2%, while the average output length fell from 22,395 to 7,294 tokens. The result shows that distillation monitoring based only on average KL divergence or token agreement may mistake model collapse for convergence.
The code has been integrated into the authors’ maintained fork of verl and includes CPU unit tests, vLLM rollouts, and a rule-based scoring pipeline. However, the default recipe requires eight GPUs, four rollouts per problem, and responses of up to 7,168 tokens. The repository does not include the training Parquet files, final model weights, or complete execution logs, and the current evidence is concentrated on small mathematical reasoning models. Future work should examine whether TIDE can maintain the same stability on code generation, tool calling, and teacher–student pairings that more closely resemble real-world production systems.