Transformers Proposes GLM-5.3-Flash Precision Fix, Keeping Key mHC Parameters in FP32

On November 5, a Transformers maintainer proposed PR #50413 to fix mixed-precision loading behavior in GLM-5.3-Flash: the checkpoint’s mHC base and scale parameters, stored in FP32, were still being downcast to BF16 during BF16 loading. At the time of review, the proposal remained open, so it should not be treated as a fix in an official release.

The issue concerns Manifold-Constrained Hyper-Connections in the model architecture. The official model card says GLM-5.3-Flash uses mHC alongside sparse and linear attention. The model files also contain tensors in multiple precisions. As a result, deployment checks should verify not only the overall dtype, but also whether specific parameters retain the precision specified by the original checkpoint.

The bug report provides a reduced reproduction that does not require a GPU: save the relevant parameters as FP32, then reload them; base and scale become BF16. The reported environments include a development build from the main branch, as well as versions 5.17.0 and 5.22.1. This demonstrates a dtype mismatch before and after loading, but does not quantify any change in full-model answer quality or benchmark scores.

The proposed fix adds the attention and feed-forward connection base and scale parameters individually to _keep_in_fp32_modules_strict, rather than forcing the entire mHC module to remain in FP32. This allows fn, which the checkpoint originally stores in BF16, to retain its stored precision. The author says shared precision tests cover this distinction. The review discussion also recommends checking outputs through inference with a small model to supplement validation based only on parameter dtypes. Implementation and review discussion

For teams deploying or fine-tuning models, this highlights a practical point for mixed-precision validation: inspect parameter dtypes after loading, and compare outputs while holding prompts, seeds, and generation settings constant. Track whether the fix is merged and included in a package release, then use the full model to confirm any numerical differences. Current evidence is insufficient to claim the fix improves accuracy, or to infer that other inference frameworks have the same issue.