Transformers Proposes RoPE Precision Fix for Reduced-Precision Issue During ZeRO-3 Loading

Transformers member Cyrilvallez proposed a fix on November 2 to prevent DeepSpeed from reducing the precision of rotary position embedding (RoPE) frequency buffers during model initialization. This follows a report filed on October 29; as of the time of review, PR #50275 remained open and had not been merged. Fix proposal

The issue occurs in the ZeRO-3 initialization path. The reporter said that deepspeed.zero.Init wraps tensor constructors such as torch.arange, converting floating-point results to the data type used for training. After Transformers previously refactored the RoPE module, some frequency initialization began using floating-point sequences, causing inv_freq, which should have remained float32, to become bf16. Even when the values were later recalculated and copied into the existing buffer, the lower-precision result remained. Reproduction code and version comparison

This kind of error warrants particular attention with long sequences: a small frequency deviation can produce a larger difference in rotation angle when multiplied by the position index. In a test with a specified Llama configuration at position 65,536, the author reported a maximum angle error of about 117.7 radians on the ZeRO-3 path, compared with about 0.003 radians on the reference path. This is a numerical test under a specific configuration and cannot be directly translated into a decline in model quality. Test conditions

The new proposal intervenes in DeepSpeed's wrapping of tensor constructors. The reporter tested six models, including Llama, Qwen3, and Gemma3, on CPU with DeepSpeed 0.19.9, and said that both loading entry points preserved float32 frequencies and matched results without DeepSpeed. The reporter also noted that older versions use a different calling interface; the author subsequently added a compatibility commit, but the page does not yet show a complete retest against the latest version. Testing and compatibility discussion

The official documentation explains that ZeRO-3 partitions model parameters during initialization, making the loading process another area engineering teams need to check. For long-context fine-tuning, the next steps are to track whether the fix is merged, whether older DeepSpeed versions pass validation, and what the actual training results show before and after the fix. Current evidence is still mainly numerical reproduction and loading tests. DeepSpeed integration documentation