GitHub Repo
Transformers Merges GQA Decoding Optimization to Reduce K/V Copies During Masked Generation
On November 4, Transformers merged a single-token decoding optimization that reorders query heads to reduce K/V copying in some SDPA paths. The author’s tests on a specific GPU show faster decoding, but inclusion in a released package and gains on real models remain to be confirmed.
Transformers merged PR #49743 into its main branch on November 4, improving single-token decoding for grouped-query attention (GQA). The change targets SDPA paths that cannot use native GQA directly, especially batched generation with left-padding masks. The merge confirms that the change is in the main branch; it does not establish that existing pip installations benefit.
GQA lets multiple query heads share a smaller number of key/value heads, but under some conditions the old path still called repeat_kv, copying K/V at every layer and decoding step. The new method reorders the query heads that share a key/value head into multiple query rows for that head, preserving the original mapping, then restores the output. This avoids repeatedly moving large key/value tensors. Prefill, multi-token queries, and the existing enable_gqa path retain their prior flow. Technical details
The author tested on an RTX PRO 6000 Blackwell with PyTorch 2.9.1, using a randomly initialized model with dimensions matching Llama-3.1-8B. At batch size 32 and a 4k prompt length, decoding time per step fell from 152.1 ms to 54.2 ms. This result applies to the decoding stage in a specific configuration; it is not a guarantee of higher end-to-end request throughput or faster performance across all models. The author also noted that only CUDA and CPU had been tested. Test setup and results
Performance still depends on backend selection. The PyTorch documentation says SDPA selects an implementation based on its inputs, fused kernels have their own constraints, and floating-point results may differ across kernels. When evaluating an upgrade, engineering teams should hold the model, mask, batch size, and context length constant, then compare per-step latency, memory use, and output consistency.
Before deployment, teams should also confirm that they are actually using sdpa. Transformers lets users select an attention backend through attn_implementation; this change should not be assumed to apply to other inference engines. Key follow-up points include whether the change ships in a release and whether results reproduce with real weights, different hardware, and compilation modes. Backend configuration documentation