PyTorch Community Reports CTC Loss Anomaly; Certain CUDA Inputs May Produce Incorrect Losses and Gradients

PyTorch has a new report concerning correctness in sequence-recognition training: when F.ctc_loss uses int32 labels and length arguments and input lengths vary within a batch, the reporter observed incorrect—and sometimes call-to-call inconsistent—losses with non-contiguous log_probs. Under some settings, non-zero gradients also appeared outside the valid input lengths. Issue report

The reproduction environment was Windows, an RTX 4060 Laptop GPU, and PyTorch 2.14.1+cu130. The test applies log_softmax and permute to the model output to put it in the layout required by CTC. Making the same values contiguous brought the forward loss into agreement with CPU, and disabling cuDNN also produced consistent results. However, adding .contiguous() alone did not address the reported gradient issue, so it should not be treated as a complete fix. Reproduction code and output

The official documentation explains that CTC sums over possible alignments between input and target sequences, using each sample's length to mask padded portions. It also specifies that the cuDNN path requires all input_lengths to equal the time dimension T and requires integer arguments to be int32; the general implementation uses torch.long. The report says that variable-length inputs still entered the cuDNN path, raising questions about whether backend selection fully checks these constraints. The root cause remains unconfirmed. CTCLoss documentation

For sequence-recognition engineers, the report is a reminder to check both the forward loss and gradients in padded regions when debugging training. The documentation notes that cuDNN may use non-deterministic algorithms, but that alone does not establish that this discrepancy is expected behavior. Based on the available evidence, affected teams can compare results on their own batches across CPU, with cuDNN disabled, and with int64 arguments. They should watch for whether maintainers can reproduce the issue, whether other platforms are affected, and whether a fix adds regression tests for both loss and gradients. Documented constraints Report status