GitHub Repo
PyTorch Merges ROCm FP8 Matrix Multiplication Fix for Scaling Indexing and Out-of-Bounds Reads
On November 6, PyTorch merged a fix to Inductor’s CK backend addressing incorrect indexing of per-row FP8 scaling factors. The author reported regression test results on MI350X, while inclusion in official packages and validation on other hardware remain unconfirmed.
On November 6, PyTorch merged PR #202816 into the main branch, fixing the scaling configuration used by Inductor’s ROCm Composable Kernel (CK) backend for per-row FP8 matrix multiplication. The change has been recorded, but that does not establish that the fix is included in currently installed stable packages.
The issue involved the scaling tensors for _scaled_mm: activations use [M, 1], while weights use [1, N]. The old template inferred the CK configuration from memory strides. However, a contiguous [M, 1] tensor can have strides (1, 1), causing activation scaling to be treated as column-wise. This produced incorrect values and, when the number of output columns N exceeded the number of rows M, could also read beyond the scaling tensor’s allocated bounds. The fix assigns ("Col", "Row") according to the tensors’ roles, avoiding the ambiguity of a single dimension. Fix commit
The added regression tests cover just-in-time compilation and AOTI, with and without bias, and multiple row counts using the same compiled model. The author reported that all four new tests passed after the fix in an MI350X, ROCm 7.0, and ck4inductor environment. Without the fix, two AOTI tests showed 98.7% element mismatches. This provides evidence of correctness in the specified environment, but does not establish improvements in overall model quality or performance. Test details
As technical background, FP8 can use a scale shared across an entire tensor, or assign a separate scaling factor to each row. The latter offers finer control over numerical representation and requires the kernel to map each scaling value correctly. A previous PyTorch technical article on AMD FP8 explained these quantization granularities and CK’s role in the relevant operations. Official technical background
For operations teams, a sensible next step is to check the deployed commit, confirm which kernel is actually selected, and compare outputs before and after compilation across different matrix shapes. The official documentation notes that torch.compile autotuning benchmarks and selects matrix multiplication configurations, so enabling FP8 alone does not establish that a workload takes the affected path. Compiler documentation