GitHub Repo
PyTorch Community Reports Out-of-Bounds Write in Large bmm; GB200 Reproduction Highlights Stable-Branch Patch Gap
A PyTorch community report on October 27 says certain matrix shapes may cause 32-bit offset overflow in a native Triton kernel, leading to cascading failures in subsequent CUDA operations. The reporter says fixes are in main and the 2.15 branch, while backports to stable branches remain unconfirmed.
On October 27, the PyTorch community filed a bug report with a minimal reproduction: on NVIDIA GB200 with PyTorch 2.13.0+cu130, torch.bmm with a contraction dimension of K=1 may trigger an out-of-bounds write. At the time of review, the issue remained open, with no maintainer confirmation of a backport plan. The reporter disclosed that AI helped prepare the article, while they conducted the experiments and branch checks themselves. Bug report
bmm multiplies two batches of three-dimensional tensors, with input shapes [b,n,m] and [b,m,p] and output shape [b,n,p]. When the intermediate dimension is 1, each batch operation is equivalent to an outer product; as a result, the output can grow rapidly even when the inputs are small. This is why checking indices for large tensors requires looking beyond input sizes. PyTorch API documentation
The reported test used inputs shaped [512,B,1] and [512,1,512]. When B increased from 8208 to 8209, the starting element offset for the last batch exceeded INT32_MAX, and the native kernel began to fail. The author identified the root cause as offset multiplication using int32, which overflows and writes to the wrong location. In the comparison tests, setting TORCH_DISABLE_NATIVE_JIT=1 to use cuBLAS, or converting the relevant indices to int64, passed the cases shown. Reproduction and boundary tests
This kind of error can interfere with subsequent diagnosis. NVIDIA's documentation states that cudaErrorIllegalAddress means a kernel accessed an invalid address, leaving the process in an inconsistent state; subsequent CUDA work continues to report errors, and the process must be terminated and restarted. Engineering teams should therefore run boundary cases separately to avoid mistaking cascading failures for multiple independent issues. CUDA error definitions
The version scope should remain qualified: the hardware reproduction was conducted in a GB200 environment with version 2.13.0, while the risk in 2.14 is based on the author's comparison of branch code. The claim that main and 2.15 have been fixed also comes from the report. Next, track whether maintainers confirm stable-version impact, schedule backports, and assess the performance impact of alternative execution paths on real workloads. The report does not establish that all GPUs or all matrix multiplications are affected. Version checks and backport request