Back Home

GitHub Repo

Transformers Proposes Tensor-Parallel Gradient Accumulation Fix for Duplicate Qwen3 Gradient Summation

A November 17 Transformers report found that combining tensor parallelism with gradient accumulation can repeatedly amplify some Qwen3 gradients while leaving the forward loss nearly unchanged. A fix proposed the same day synchronizes each newly added gradient and is still awaiting review and merge.

zh-Hant

On November 17, the Transformers community reported a training correctness issue: when Qwen3 uses tensor parallelism with gradient accumulation, gradients for the query and key normalization weights may diverge from a non-parallel reference model starting with the second backward pass. The report used the 5.24.0.dev0 development version, two CPU processes, FP32, and the Gloo communication backend. It did not use Trainer, DDP, or FSDP; GPU/NCCL has not been tested. Issue report

The key is when synchronization occurs. The PyTorch documentation says that register_post_accumulate_grad_hook runs after gradient accumulation is complete and the parameter’s .grad has been updated; the hook operates on the parameter itself. If the entire gradient buffer is summed across processes at that point, contributions that were synchronized earlier are included again. PyTorch documentation

According to the reporter’s analysis, with two-way tensor parallelism and three accumulated microbatches, the expected gradient is the sum of three contributions. Instead, the first contribution is multiplied by four, the second by two, and the third is added once. In the test, the maximum relative L2 error for the affected gradients was about 2.02 after the third backward pass, while the loss difference was only on the order of one three-millionth. This means that checking only the forward loss could miss an error in the accumulation path; the data so far comes from the submitter. Reproduction and analysis

The fix proposed that day replaces the hook with register_hook: it first copies and synchronizes the incoming gradient for that step, then lets autograd accumulate it. A new test compares the gradients of each layer’s Q/K normalization weights after each of three microbatches, instead of checking only the final result. Proposed patch

As of this check, the pull request remains open, with no review conclusion. For teams using this training path, the practical check is to compare gradients at each accumulation step using identical initial weights and inputs. Teams should then track upstream merge status, which official versions include the fix, and validation in GPU communication environments. The report does not establish that all Qwen3 training is affected. Pull request status

Sources

  1. Qwen3: tensor-parallel gradient accumulation silently amplifies Q/K norm gradients
  2. Fix tensor-parallel replicated gradient accumulation
  3. Tensor-parallel gradient accumulation patch and regression test
  4. torch.Tensor.register_post_accumulate_grad_hook