Transformers Community Reports FP8 Dequantization Boundary Error That May Cause Some Weights to Fail to Load or Have Incorrect Values

On October 22, the Transformers community reported a FineGrainedFP8 dequantization issue: when a weight matrix dimension is not an integer multiple of the quantization block size, the loading process may raise an exception; some shapes pass validation but are restored with incorrect values. The reporter ran a CPU reproduction test on 5.17.0 and noted that 5.20.2 and the main branch as of that day still contained the same logic. This is a community report with code and output; it does not confirm that all related model deployments are affected. Issue report

The official documentation says Fine-grained FP8 quantizes weights in 128×128 blocks by default and quantizes activations by groups per token. It supports DeepSeek-V3 and R1. Each block has its own scaling value, so dequantization must map every element back to its original block; in practice, the same boundaries must be preserved when the final block is incomplete. Official technical documentation

The report says the current code infers block size by dividing the number of weight rows by the number of rows in the scale grid. A projection matrix in DeepSeek-V3/R1 has 576 rows, which requires five groups of scaling values when divided into 128-row blocks; the last block has only 64 rows. Since 576 is not evenly divisible by five, this triggers a loading error. In another test case with 192 rows and two groups of scaling values, the division succeeds, but the code incorrectly infers a block size of 96 rows, shifting where the scaling values are applied. Reproduction and root cause analysis

For engineering teams, this suggests validating tensors after dequantization when deploying across hardware; checking only whether a model loads successfully is not enough. Native FP8 execution and conversion back to higher precision may follow different code paths, so passing tests on one path does not cover the other. The reporter recommends reading the block dimensions from the configuration first, then expanding the scaling values and cropping to the weight shape, but mixed block formats still need clarification. Follow-up should track maintainer confirmation, the merge of a fix, and an official release, and compare boundary blocks against reference dequantization results. There is currently no evidence quantifying the issue’s impact on real-world generation quality. Proposed fix and limitations