Transformers Reports Qwen3-MoE Loss Scaling Issue That May Amplify Router Auxiliary Weight in Multi-GPU Training

On October 19, the Transformers community reopened a distributed training issue for Qwen3-MoE and inspected that day’s main-branch code. The report says Trainer may still scale the router auxiliary loss along with the main loss. This is a follow-up to an issue that was automatically closed; the new material consists of source-code inspection and a response from a project member, not a GPU retest on the latest version. Issue and discussion

The issue concerns a training path that enables average_tokens_across_devices=True and passes in the global token count. According to the report’s analysis, the model first combines the token loss with the weighted auxiliary loss, then Trainer multiplies the combined loss by the number of data-parallel processes. This means the correction intended only for token normalization also changes the auxiliary term’s weight. Project member Rocketknight1 has responded about the scaling logic and asked Trainer maintainers to join the discussion. Technical analysis

This coefficient affects the balance of expert usage. The official documentation explains that router_aux_loss_coef penalizes uneven expert routing, while output_router_logits controls whether routing information and the auxiliary loss are output. Therefore, if the effective weight changes with the number of processes, moving the same logical batch to more GPUs could also change the training objective. This is a risk inferred from the reported mechanism; a complete model quality evaluation is not yet available. Qwen3MoE documentation

The report includes a single-GPU versus dual-GPU comparison using a randomly initialized small model, along with a reference fix that scales the two losses separately. The original results used Transformers 5.15.0, FP32, and fixed synthetic data, so they do not directly represent the current version or all MoE training. The proposed fix is still under review. Users should track its merge status and any official release, check whether configurations such as gradient accumulation are included in validation, and compare routing gradients and individual loss components before scaling up training. Proposed fix