GitHub Repo
PyTorch Community Proposes Fusion Fix as Large Intermediate Buffers May Slow Single-Token Decoding in Quantized LLMs
An October 10 report from the PyTorch community says Inductor may write a full matrix product to memory for certain operator combinations, raising the cost of single-token decoding in quantized LLMs. The issue affects both a stable release and a development build; the proposed fix has limited tests and measurements but remains an unmerged draft.
On October 10, the PyTorch community reported an Inductor performance issue: with settings related to max-autotune, a single-row matrix multiplication may produce a large FP32 intermediate buffer that could otherwise be avoided, depending on how preceding operators are considered for fusion. The reporter observed slowdowns with specific operator combinations on an RTX 5090 in both version 2.14.1 and the 2.16 development build, and provided a minimal reproduction and generated code. Issue report
The report’s analysis says that when the matrix multiplication’s M dimension is fixed at 1, this path decomposes the operation into a broadcasted multiplication followed by a sum. If the number of operations in each input expression is below a threshold individually, but exceeds the GPU’s default threshold of 30 when combined, the compiler may materialize the full (1, K, N) product and then pass it to a reduction kernel. With K=9216 and N=2560, this writes and reads back about 94 MB of data each. Adding or removing a small amount of computation may avoid this window. Reproduction and analysis
The official documentation says max-autotune uses Triton or template-based matrix multiplication and enables CUDA Graphs by default on GPUs; max-autotune-no-cudagraphs disables graph replay. This lets engineering teams compare kernel generation and replay effects separately. The case suggests that when evaluating quantized decoding, teams should measure the activation function, dequantization, and matrix multiplication together: testing the multiplication alone may miss memory traffic caused by fusion. Official documentation
Draft PR #200540 proposes materializing the smaller broadcast input first, avoiding the need to write the full product to memory. In the author’s RTX 4090 tests, an int4 weight-quantized case with a SwiGLU pre-operation fell from 179.5 microseconds to 27.1 microseconds. The PR also adds buffer allocation and numerical comparison tests. These are measurements of a specific kernel and cannot yet be translated into a token-throughput gain for a full model. Fix proposal
As of the time of checking, the proposal remained a draft with no review, and some CI jobs were awaiting approval. Engineers can first check whether the generated code allocates the product buffer described above, then compare compile modes, latency, and numerical results. They should also track upstream review and whether the fix is included in an official release. Proposal status