Transformers Community Reports RoPE Initialization Gap: Some Long-Context Configurations Fail on Null Values

On October 21, Transformers received a report about compatibility with long-context configurations: when a model configuration contains head_dim=None, some RoPE calculations and validation procedures fail to infer the attention head dimension correctly and stop with a type error. The reported environment used the main-branch development version 5.21.0.dev0 and CPU-only PyTorch; this does not establish that all stable versions are affected. Issue report

The reporter created a reproduction case using a small, single-layer Mixtral model and compared five rotary positional embedding configurations. The standard and linear-scaling paths completed the forward pass; Dynamic and YaRN failed during model construction, while LongRoPE failed earlier, during configuration validation. The case does not require downloading large weights, making it easier for developers to distinguish configuration handling issues from deployment factors such as display memory and weight loading. Reproduction script and output

The issue stems from the semantics of Python attribute defaults. The relevant code uses getattr to read the head dimension and provides hidden size divided by the number of attention heads as a fallback. But when the attribute exists and its value is None, the fallback does not take effect. Multiplying that null value by the rotary scaling factor later triggers an error. The report notes that other RoPE paths already handle null values and proposes a consistent way to derive the dimension. Root-cause explanation

This gap has direct implications for long-context deployment. The official documentation lists Dynamic, YaRN, and LongRoPE as supported scaling variants, selected through rope_parameters and configured with their respective parameters. The engineering takeaway is that a model starting with its default configuration does not prove it will run after changing the positional embedding settings; configuration validation, model initialization, and an actual forward pass all need coverage. Official RoPE documentation

As of the time of review, this remains a community report with reproduction materials. The reporter also listed Ministral, MiniMax, and HunYuan as models that may use the same configuration pattern, but the public case primarily demonstrates Mixtral, so this should not be treated as a verified list of affected models. Next, watch for maintainer confirmation, a merged fix, and its inclusion in a release; even if the initialization issue is fixed, long-context quality still needs separate evaluation. Report status