Transformers Proposes Whisper and BART Compilation Fix for LayerDrop Disabled

On Nov. 8, the Transformers community proposed PR #50475, adjusting LayerDrop checks in the Whisper and BART encoders and decoders. The issue is that even when the layer-drop probability is set to zero, the training path still draws a random tensor and branches on its value, preventing torch.compile(fullgraph=True) from completing the trace. At the time of review, the proposal was still open, and CI was awaiting maintainer approval to run.

A bug report filed the previous day listed Transformers 5.22.2, PyTorch 2.15.0, and a macOS ARM64 environment. It reproduced the error with a small Whisper model, random inputs, and backend="eager". This setup mainly isolates a Dynamo frontend tracing issue; it cannot establish how GPU kernels perform or how fast the compiled model runs. The reporter also reproduced the same error with BART and said that adding a conditional check locally allowed the reproduction script to pass. Issue and reproduction script

The key problem is that the Python branch condition depends on a tensor value. The original code calls torch.rand([]) and then checks whether the result is below the layer-drop probability. Even with a probability of zero, the tracer encounters data-dependent control flow. The proposal first checks self.training and self.layerdrop > 0.0, drawing a sample only when LayerDrop is actually enabled, and adds full-graph training compilation regression tests for both models. Change description

For fine-tuning workloads, this change could remove a compilation obstacle under the default settings. PyTorch's official documentation explains that the default compile mode allows graph breaks, while full-graph mode requires capturing a single FX graph and raises an error if a break occurs. It can also be used to check a library's compatibility with compilation. Passing a full-graph trace therefore has engineering value, but does not by itself imply higher throughput. PyTorch full-graph mode documentation

Two limitations remain. When the layer-drop probability is greater than zero, the dynamic branches caused by random layer dropping still occur. Also, skipping an unnecessary random draw changes the CPU random number generator's state, so subsequent operations with a fixed seed may differ from the previous version. Engineering teams should track the merge and release status, then validate gradients, reproducibility, and performance with their actual training configurations. Limitations and random-number impact