GitHub Repo
PyTorch Community Reports BF16 Gradient Error in Compiled Models; Convolution Followed by log_softmax May Affect Training
A report submitted to the PyTorch community on October 17 says that, under BF16 mixed precision, compiling certain convolution and log_softmax computation graphs with Inductor may produce incorrect gradients. The report includes a reproducer and temporary workarounds, but maintainers have not confirmed the full scope or announced a fix release.
On October 17, the PyTorch community raised a training correctness issue: after enabling CUDA BF16 autocast, a computation graph that applies log_softmax after a convolution may produce gradient deviations when compiled with torch.compile, particularly when additional operations follow. The report used PyTorch 2.10.0 and CUDA 12.8. At the time of review, the issue remained open, with no related fix available. Issue report
The reproducer compares two functions that should have identical gradients, one of which adds an extra term processed with detach(). Their gradients match without compilation. After compilation, when a common offset of 1000 is added to the logits, gradient cosine similarity drops to 0.6943 and the norm ratio reaches 2.295. This gives developers a concrete case to investigate, rather than relying only on training curves to infer an anomaly. Reproduction results
According to the reporter’s analysis of the generated kernel, the forward pass computes the maximum and log-sum-exp using FP32 logits, but stores the logits in a BF16 buffer. During backward recomputation, rounded logits are combined with statistics from the unrounded data, causing an inconsistency between the forward and backward passes. This is the reporter’s root-cause assessment and has yet to be verified by maintainers. Kernel analysis
The official numerical accuracy documentation explains that floating-point operation order and backend differences can change results, and that mathematically equivalent computations are not guaranteed to be bitwise identical. Engineering teams should therefore assess the magnitude of the deviation and its effect on convergence, while retaining the uncompiled path as a comparison. Numerical accuracy documentation
The reporter says switching to aot_eager or setting torch._inductor.config.emulate_precision_casts = True resolves the issue in their case. Teams using similar operation chains can first compare gradient direction and norm before and after compilation; the performance cost of the internal setting should be measured independently. Follow-up should track maintainer reproduction, affected versions, and whether the fix is included in regression tests. Temporary workarounds and tracking