Transformers Reports Zero MoE Expert Gradients; BF16 with Certain Distributed Configurations May Halt Updates

A bug in training mixture-of-experts (MoE) models with Transformers may be masked by the loss curve. On October 17, a project member reported that all expert gradients could become zero when the token dispatch path uses BF16 and the expert-parallel size falls within a particular range. The reproduction environment included Transformers 5.20.1, PyTorch 2.13.0, and four NVIDIA B300 GPUs; the report also included a small Qwen3 MoE reproduction script. Issue report

The official documentation explains that token dispatch sends tokens from each process to the corresponding experts, then exchanges the results; expert weights can also be further sharded with FSDP2. The report says that when tp_size < ep_size < fsdp_size * tp_size, the divisor for expert gradients differs from the size of their FSDP group, triggering NCCL's PREMUL_SUM reduction path. According to the reporter's analysis, incorrect type handling of a related BF16 scaling factor causes a particular coefficient to be read as zero. Parallelism documentation, root cause and reproduction

The engineering risk is that other parameters can still learn, allowing the overall loss to keep falling and making the loss curve alone insufficient to determine whether experts are being updated. In a full-parameter fine-tuning run of Qwen3-30B-A3B, the reporter observed that all 96 expert gradient tensors were zero after a single backward pass. This is an observation from a specific environment and does not establish that all MoE training is affected; exceptions listed in the report include FP32, frozen experts, and some parallelism boundary configurations. Test results and scope

The proposed fix sums expert gradients first and then divides locally, avoiding the path described above; the proposal's author said that the BF16 gradient norm in the reproduction recovered to nearly the FP32 level. A commit added on October 18 further compares the loss and each gradient. However, the PR was still open at the time of review. Although the original issue was marked complete, this does not confirm that a released version includes the fix. Engineers should check the merge and release status, then spot-check expert gradients and weight updates under the same configuration rather than relying only on a falling loss to validate training correctness. Proposed fix