GitHub Repo
Transformers Community Reports ZeRO-3 Initialization Precision Degradation; RoPE Frequencies May Be Cast to BF16
A Transformers community report on October 29 says DeepSpeed ZeRO-3 initialization may reduce the precision of rotary position embedding frequencies, affecting long-sequence training. The author submitted a fix and version comparisons, but the PR had not been merged as of the time of review; the impact on actual model quality remains unverified.
On October 29, the Hugging Face Transformers community reported a precision issue in long-sequence training: when loading a model with DeepSpeed ZeRO-3 and BF16, the rotary position embedding (RoPE) inverse-frequency buffer, inv_freq, may be created in BF16, differing from the previous behavior of retaining FP32. The author provided reproductions on version 5.17.0 and the development branch; the issue can be triggered on a single H200. Issue report
The issue lies at the intersection of model initialization and mixed precision. The author’s analysis says deepspeed.zero.Init intercepts tensor construction and casts floating-point results to the training precision. A previous RoPE refactor switched to floating-point torch.arange, so even when torch.float is specified, the result may be downcast first. Copying the new values into an existing BF16 buffer afterward cannot restore the original precision. Fix details
Using a head dimension of 256 and a RoPE theta of 10 million, the report compares results against an FP64 reference. For Llama at position 65,536, the maximum unreduced angle error was 117.7 radians with ZeRO-3, compared with 0.003 radians on the control path. This is a numerical error under a specific configuration and cannot be directly translated into an accuracy loss. The author also notes that the YaRN path in the tests was unaffected. Reproduction and measurements
The official DeepSpeed documentation explains that ZeRO-3 saves memory by partitioning model parameters and that zero.Init can be used to create large models. This makes precision handling during initialization worth checking. An engineering implication is that verifying weight precision alone is insufficient to rule out numerical differences in position-embedding buffers. DeepSpeed documentation
The proposed fix switches back to creating an int64 sequence first and then converting it to floating point, across 203 files. The author reports that the fix restores FP32 and has provided local test results; as of the time of review, the PR remained open without an approving review. Teams fine-tuning long-context models should monitor maintainer reproduction, inclusion in an official release, and the precision of inv_freq after loading their own models, along with training comparisons, rather than generalizing these measurements to all RoPE models. Fix proposal