GitHub Repo
Transformers Proposes Per-Layer Quantization, Exploring NVFP4 and FP8 in One Model
A Transformers proposal submitted on November 2 lets different model layers load through their own quantizers, and tests a small model mixing FP8 and NVFP4. The proposal has not been merged; accuracy, memory use, and inference performance on real models remain to be validated.
Hugging Face Transformers contributor ArthurZucker submitted a proposal for per-layer quantization settings on November 2. It aims to let different modules in the same model checkpoint load through their corresponding quantizers. The change targets support for loading mixed quantization formats. As of the latest check, the PR remains open, and its page lists no review comments. Proposal and status
The proposal expresses settings as quantization_config={glob: config}, reusing the name-matching approach from dtype plans to route different layers to their respective quantizers. Already-quantized NVFP4 weights are loaded through a deserialize operation. The author says the change was tested with a small, randomly initialized Mistral4 model, with attention and expert modules using FP8 and shared experts using NVFP4. This provides a functional test case for mixed formats, but is not evidence of quality or speed on a production model. Author’s description
The current official documentation says that pre-quantized checkpoints store quantization settings in config.json, and from_pretrained selects the corresponding quantizer; different methods may also require separate library dependencies. Quantization involves trade-offs among memory use, speed, and accuracy, and reducing bit width does not guarantee the same gains on all hardware. Official quantization documentation
From an engineering perspective, if per-layer settings mature, they could make it easier for developers to specify the precision requirements of different modules and reduce the need to write separate loading workflows for mixed formats. Successful loading is only a first step, however: module-name matching, weight formats, and execution kernels still need to be checked for compatibility, and performance must be measured on the target workload.
The PR is currently targeting another development branch, and the author notes that FP8 experts depend on additional kernel support. Next, it will be important to see whether those dependencies are available and whether testing on complete models reports accuracy, peak memory use, and throughput. The available public information does not yet support a conclusion that deployment performance will improve. Dependencies and development branch