GitHub Repo
Transformers Proposes Restructuring Quantized Weight Loading, Moving Decompression to the Loading Stage
A Transformers patch proposed on November 2 integrates compressed-tensors weight decompression into the loading conversion process to address compatibility issues with weight-name mappings. The proposal has not been merged, and public CI reports one newly failing test; its effect in actual deployments remains to be verified.
Hugging Face Transformers contributor ArthurZucker proposed PR #50238 on November 2, changing how compressed-tensors checkpoints are loaded: decompression is performed through the weight-conversion mapping, so dense weights are available immediately after loading by default, instead of being unpacked at the first forward pass. This is still a proposal under review and should not be treated as behavior in an official release. Proposal
The change involves the distinction between model file formats and execution precision. According to the official documentation, compressed-tensors can store compressed weights such as FP8 and INT4. The current default flow keeps the data compressed and decompresses it at the first forward pass. Users can also set dequantize=True to restore weights to the model’s data type, such as BF16, during loading. To retain FP8 and use accelerated kernels, they must explicitly enable use_optimized_inference=True. Loading modes documentation
The proposal specifically addresses cases where checkpoint weight names need to be remapped during loading. The author used MXFP8 Dory as an example, reporting that the new flow produced no unexpected keys and that its generation results matched a BF16 reference. The patch also avoids forcing dequantization for uncompressed checkpoints, preserves scale parameters needed by models that quantize only the KV cache, and stops quantizing activations when dequantization is specified for W8A8. These results are currently based on the author’s report. Patch details
For deployment engineers, the key question is how work is divided between loading and the first inference. Based on the design described above, moving decompression earlier could change startup time, first-request latency, and how post-load memory measurements should be interpreted. However, the proposal provides no performance figures, so it does not support claims of faster inference.
As of this review, the PR remains open and awaiting review. A CI comment reports one newly failing dequantization test, and no subsequent result showing all checks passing was available. Before adopting the change, track the test status and validate weight-name mappings, scale parameters, and output consistency with your own checkpoints. Review and CI status