PyTorch Community Reports Data Loading Performance Regression That May Slow CUDA Training

A PyTorch user filed a performance issue on October 24, reporting that CUDA training loops may be significantly disrupted from version 2.10.0 onward when a background thread in the same process continuously receives large DataLoader batches. The community report includes a reproduction script; as of the time of review, the issue remained open, with no maintainer confirmation of the root cause or official fix. Issue report

The reproduction script has the main thread train a small convolutional network while a background thread receives image batches from eight worker processes, each batch about 79 MB, and immediately discards them. On the reporter’s RTX PRO 6000 Blackwell, each step took about 8.83 ms on version 2.9.1 and about 428 ms on 2.10.0. With background loading disabled, both versions took about 8.4 ms per step. This comparison suggests that the concurrent data loading and computation path warrants investigation, but it is not enough to determine the underlying cause. Test script and measurements

The slowdown in this extreme test should not be directly extrapolated to real-world training. In the report’s YOLO workload, step time rose from about 73–75 ms to 80–85 ms, a smaller difference. The reporter also stated that Claude Code wrote the detailed report based on measurements from their machine, so the figures still need to be independently reproduced in other environments. Measurement notes

The official documentation explains that num_workers controls the number of subprocesses used for data loading; when set to zero, data is loaded in the main process. pin_memory=True places tensors in page-locked memory to speed transfers to CUDA GPUs. These settings control different parts of the pipeline. Engineering teams investigating the issue should measure worker count, batch size, and memory pinning separately, rather than diagnosing bottlenecks based only on GPU utilization. DataLoader documentation

For teams that have seen throughput fall after a recent upgrade, this report offers a test they can use for comparison. Keep the model, driver, and data conditions fixed, then compare per-step latency for compute alone and for the full input pipeline. Watch for whether maintainers identify the change that introduced the issue, and whether a fix improves both the microbenchmark and real-world training.