模型後訓練
Open-MOPD Reallocates the Token Budget for Multi-Teacher Distillation, Raising Capability Integration from 35.6% to 83.4%
Open-MOPD finds that the main bottleneck in multi-teacher online policy distillation is not conflicting teacher gradients, but a severe imbalance in the effective number of token updates allocated across domains. By combining token-share balancing, gap-aware allocation, and student reward refresh, it recovers 83.4% of the available capability gains in a single 3B model.
Multi-teacher online policy distillation (M-OPD) has a student model generate its own responses, after which teachers specializing in mathematics, coding, and instruction following provide dense supervision for the student tokens in their respective domains. The goal is to combine the capabilities of multiple RL experts into a single deployable model. Open-MOPD establishes a fully open experimental setup using SmolLM3-3B-Base: it first performs mixed-domain SFT, then trains three separate RL teachers, and finally uses known domain labels for perfect routing, thereby eliminating router errors.
Even with perfectly accurate routing, naive M-OPD achieves an aggregate score of only 28.05 across six evaluations, below the 31.55 achieved by RouteOPD with domain-specific deployment. Relative to the gain between mixed SFT and RouteRL, it recovers only 35.6%. The problem is concentrated in instruction-following tasks with short outputs: these data account for 20.3% of prompts but generate just 0.99% of gradient tokens. Because the loss is aggregated by token, long mathematics and coding responses naturally receive a larger update budget. Different convergence rates across domains also cause reward magnitudes to drift over time, while repeatedly reusing the same rollouts makes rewards that depend on student probabilities stale.
Open-MOPD introduces three corresponding mechanisms. Token-share balancing directly controls each domain’s share of gradient tokens; gap-aware allocation redirects subsequent budget toward domains where the student still has a larger gap from the teacher; and reward refresh recomputes the student log probability before every gradient step while caching teacher states to reduce the additional cost. The complete method raises the aggregate score to 31.24 and the capability-gain recovery rate to 83.4%, approaching RouteOPD, which requires maintaining three student models.
The result offers an important lesson for post-training engineers: balancing the number of prompts is not the same as balancing optimization signals, especially when output lengths differ by tens of times across tasks. The study is limited to a single 3B base model, three domains, and oracle routing, while its evaluations also use different numbers of samples for each domain. Real-world deployment will still need to determine whether this budget-control approach remains stable with automatic routing, larger MoE models, and a broader set of overlapping capabilities.