Back Home

GitHub Repo

PyTorch Community Reports Abnormal Attention Gradients on B300; Certain Short Sequences May Produce NaNs in cuDNN

A PyTorch community report on October 9 says that a combination of certain short sequences and Boolean masks on B300 may cause NaNs during cuDNN attention backpropagation, even though forward outputs remain normal. Maintainers said the issue did not reproduce with newer cuDNN and recommended testing a nightly build or 2.15 release candidate; the reporter’s environment has yet to be upgraded and verified.

Pytorch Deepdream (https://github.com/gordicaleksa/pytorch-deepdream) by gordicaleksa (https://github.com/gordicaleksa/pytorch-deepdream/commits?author=gordicaleksa) · MIT · Image source
zh-Hant

On October 9, the PyTorch community submitted a training-correctness report with a minimal reproduction: with NVIDIA B300, PyTorch 2.14.1+cu130, and cuDNN 9.24, scaled dot-product attention (SDPA) using bfloat16, a per-head dimension of 256, and an explicit Boolean causal mask can produce NaNs in the query tensor gradients for some short sequences. The forward results and key and value gradients remain finite, so workflows that monitor only the loss may detect the issue late. Community report and reproduction

The report shows that at sequence length 102, the cuDNN backend produces abnormal results, while the memory-efficient and math backends behave normally. The issue also did not reproduce with a head dimension of 128, or when the mask tensor was removed and is_causal=True was used instead. These comparisons narrow down the triggering conditions, but are not enough to conclude that all Blackwell GPUs or all short sequences are affected. Test results

The key technical factor is backend selection. The official documentation explains that SDPA automatically selects an implementation based on its inputs, and that engineers can control the backend through sdpa_kernel(). When handling half-precision inputs, the math backend retains float intermediate results. When investigating training anomalies, compare gradient finiteness across backends and record the mask, data type, and tensor dimensions. SDPA documentation

A temporary workaround is to call torch.backends.cuda.enable_cudnn_sdp(False) to disable the cuDNN SDPA path. This setting is documented officially but is still marked beta; measure its effects on speed, memory requirements, and model compatibility with the actual workload. Backend settings documentation

For a fix, an upstream change upgrading to cuDNN 9.26 and adding a guard for a failure condition in some older versions was merged in September. In response to this report, maintainers said the issue did not reproduce with newer versions and asked the reporter to test a nightly build or 2.15 release candidate. This gives a direction for verification, but does not confirm that the reporter’s environment has been fixed. Training teams should run the same reproduction before and after upgrading and check gradients before optimizer updates, rather than relying only on a normal forward loss to judge training health. Related fix, maintainer response

Sources

  1. Issue #197811: torch.compile silently reuses a graph when closures capture different Tensor method descriptors
  2. PR #197845: Guard on captured method/wrapper descriptors
  3. torch.compile — PyTorch 2.14 documentation
  4. B300 上 cuDNN SDPA 反向傳播 NaN 回報與維護者回覆
  5. PyTorch SDPA API 文件
  6. PyTorch 後端設定文件
  7. 升級 cuDNN 9.26 並加入舊版本防護的修補