GitHub Repo
Transformers Community Proposes RoPE Compatibility Fix After Legacy Scaling Settings May Drop Model Frequency Base
A Transformers development-branch report on October 12 says that overriding the legacy `rope_scaling` field may delete existing RoPE parameters, causing initialization to fail or reloading to restore default values. A fix proposal includes regression tests but has not been merged; maintainers have also raised migration questions about the newer configuration field.
On October 12, the Transformers community reported a RoPE configuration compatibility issue: setting the legacy rope_scaling field on an already initialized model configuration may replace the entire rope_parameters object, also dropping the existing frequency base. The report uses the 5.19.0.dev0 development branch and provides a CPU reproduction that requires neither model weights nor a GPU; it does not establish that all stable releases are affected. Issue report
The reproduction first creates a Llama configuration with rope_theta=500000, then specifies linear scaling with a factor of two. The legacy type key is not converted to rope_type, so rotary embedding initialization raises a missing-key error. Even when using the new key name, the existing base is still lost. The reporter says that saving and reloading also restores the constructor’s default base of 10000, turning an obvious exception into a silent change in configuration values. Reproduction details
This has technical implications for deployments that extend context length. RoPE provides positional information by rotating attention query and key vectors according to position; the frequency base, scaling method, and factor together determine its behavior. The official documentation currently uses rope_parameters to configure these values and explains that omitting rope_theta uses the model’s default. Engineers should therefore check the full set of parameters before and after overrides and serialization; checking only the scaling factor is not enough to ensure the configuration remains consistent. Official RoPE documentation
Fix proposal #49515 preserves the existing base and rotary ratio unless the user explicitly overrides them, and handles legacy key names, layered configurations, and resetting scaling. The author reports adding ten regression tests, with all 45 tests in the RoPE test file passing after the fix. However, maintainers questioned during review why downstream users were still overriding the legacy field after adopting v5, indicating that whether the compatibility layer should support this behavior remains under discussion. Fix proposal and review
As of the time of review, the proposal remains open and unmerged. Teams using long-context overrides should first verify their deployment version, the full RoPE configuration, and the results of saving and reloading, and track whether upstream retains compatibility with the legacy interface. Passing tests also does not replace validation of actual model quality.