Transformers Community Proposes T5 Attention Fix as Tensor Layout May Send SDPA to Slower Kernel

On October 12, the Hugging Face Transformers community reported a performance regression in the T5 family: after switching to scaled dot-product attention (SDPA), the memory layout of positional bias may cause the runtime to fall back to a MATH kernel. The reporter held the PyTorch environment constant and traced the issue through Transformers commits, finding that the change introducing T5 SDPA support increased short-sentence encoding time by about 7.4% to 7.7% for two T5 embedding models on a Radeon 8060S. Issue report

The analysis points to compute_bias: it uses dimension permutations to create a non-contiguous tensor, then passes the positional bias to SDPA as attn_mask. The report’s key diagnostic shows that the mask’s last-dimension stride is 12, which does not meet the requirements of the tested fused kernel. Adding .contiguous() changes the selected kernel from MATH to EFFICIENT_ATTENTION. This places the performance issue at the boundary between tensor layout and kernel dispatch, rather than in the model weights themselves. Diagnostics and reproduction code

A fix proposed the same day modifies eight model files so positional bias is returned in a contiguous layout. The author’s tests show that, in a sentence-t5-base encoding case using half precision, a batch size of eight, and an input length of 2,048, the first-run average time dropped from 272.67 to 132.01 milliseconds on a Radeon 8060S, and from 34.19 to 16.94 milliseconds on an RTX 5090. However, gains were smaller for generation tasks, and some attention paths still use MATH. Results from long-input encoding should not be assumed to apply to all T5 services. Proposed fix and tests

PyTorch’s official documentation explains that SDPA automatically selects an implementation based on its inputs, and that each fused kernel has input limitations. Switching kernels may also produce floating-point differences. Engineers evaluating an upgrade should therefore check the actual kernel, latency, and outputs, rather than only confirming that SDPA is enabled in the configuration. SDPA documentation

As of the time of review, the proposal had not been merged and had not received maintainer review. The author’s generated-token and embedding-similarity checks support local compatibility, but the tests used Windows and a development build of PyTorch. Follow-up should track a fix in a stable release, reproduction on Linux, and performance across different sequence lengths and padding ratios. Proposal status