Transformers Community Reports RoPE Precision Differences; Converting to bf16 After Loading May Amplify Long-Context Angle Errors

On October 4, the Transformers community reported that two common bf16 model initialization workflows may produce different positional encoding values. The case was reproduced in version 5.18.0 and the main branch at the time: when using from_pretrained(dtype=torch.bfloat16) directly, RoPE’s inv_freq remains in fp32; loading first and then calling model.to(torch.bfloat16) also converts the frequency buffer to bf16. The report also noted that Trainer with bf16_full_eval=True follows the latter path. Reproduction case

This difference relates to numerical stability in long contexts. The official documentation explains that RoPE rotates attention query and key vectors according to position, giving the model relative position information; frequency values therefore directly contribute to calculating the rotation angles. Based on this mechanism, rounding frequencies to lower precision before multiplying them by larger position indices may amplify angle errors. Official RoPE documentation

The reporter compared a tiny Llama across 32,768 positions against an fp32 reference. The mean angle error was about 0.0006 radians when loading directly in bf16, compared with about 0.6733 radians when converting after loading. These are measurements of rotation angles and cannot yet be translated into answer accuracy or the extent of any decline in long-document retrieval. The test also did not cover GPUs, distributed-wrapped models, or downstream task metrics using real weights. Test methodology and limitations

The community then proposed a patch to preserve the relevant fp32 buffers in PreTrainedModel._apply, converting weights as requested while keeping positional frequencies at their original precision. However, on October 5, a maintainer said they would not accept this approach and closed the PR. The page does not give a detailed reason, so this should not be considered an upstream fix. Patch proposal and maintainer response

For engineering teams, this report is a reminder to document precision-conversion workflows when evaluating long contexts. Using the same weights and inputs, compare loading with a dtype specified directly, converting after loading, and the full evaluation workflow. First inspect the frequency buffers, then measure logits and task performance. Worth tracking next is whether maintainers propose an alternative and whether the difference has an observable quality impact on real models and deployment hardware.