GitHub Repo
PyTorch Community Reports Overlapping BF16 Dropout Random Sequences, Says Consecutive Calls May Produce Correlated Masks
A PyTorch community report dated October 28 says that some masks fully overlap when Dropout is run consecutively on specific CUDA BF16 inputs. The report includes reproductions on two versions and an analysis of the kernel code, but as of verification, no maintainer has confirmed the issue or issued an official fix.
On October 28, PyTorch received a bug report involving training randomness: on an NVIDIA L40, consecutive calls to aten::native_dropout in eager mode produced regular overlap between two masks for specific BF16 tensors. The reporter observed the same behavior on 2.7.0+cu126 and 2.14.1+cu126. These are results from tests on two versions and do not establish that every intervening version is affected. Issue report
The reproduction uses a contiguous BF16 input with shape (4096, 128) and a memory address aligned to 16 bytes, with a dropout probability of 0.5. When the masks are grouped into sets of eight elements, the last four elements from the first call exactly match the first four elements from the second call; independently sampled Bernoulli masks would be expected to match about half the time. The test sets the seed only once, before the two calls. The specific overlap disappears with FP32 or when the BF16 starting address is offset by eight bytes. Reproduction and comparisons
The reporter suspects a mismatch between the number of random values consumed by the vectorized kernel and the Philox counter reservation. The public 2.14.1 source shows that the number of random-number generations varies with vector width, while the counter offset is calculated using UNROLL. If each operation processes eight elements but reserves space for only four random values, the next call could reuse the previous sequence. This explanation is consistent with the reported results, but the root-cause analysis remains unconfirmed. CUDA kernel source
The technical issue centers on mask independence. The official documentation says that during training, Dropout uses a Bernoulli distribution to select elements to zero, sampling independently on each forward call to help prevent co-adaptation among neurons. Fixed correlations across calls could therefore alter regularization behavior. No convergence or accuracy data from a full model is available, so the impact on training cannot yet be quantified. Dropout documentation
As of verification, the issue remains open with only an automated classification label; no maintainer confirmation or linked fix was found. Engineering teams can use the minimal reproduction to check their own hardware, tensor shapes, and execution paths. Follow-up should track validation across GPUs, a fix to the random-number offset, and backports to stable branches. The result should not be assumed to apply to compiled mode or other fused kernels. Issue status