GitHub Repo
Transformers Reports Out-of-Memory During CUDA Graph Capture for Continuous Batching; Graph Count Under Investigation
A November 16 issue report says continuous batching may run out of GPU memory while capturing CUDA graphs on some machines, with the problem resurfacing after adding an MLA model. The report uses a development build; there is not yet a complete reproduction, confirmed root cause, or official fix.
The Transformers project opened an issue on November 16 reporting that continuous batching may trigger GPU out-of-memory errors while capturing CUDA graphs on some machines. The reported environment uses Transformers 5.24.0.dev0, PyTorch 2.13.0+cu130, and an NVIDIA B200. The available evidence is insufficient to extend the impact to all stable releases or GPUs. Issue tracker
Continuous batching reschedules requests at each generation step, allowing new requests to take the place of completed ones. CUDA graphs record GPU execution workflows in advance and replay batches with matching tensor shapes to reduce CPU dispatch overhead. The official documentation explains that the system pads query and KV lengths and caches the corresponding graphs; different shapes may require separate captures, so the number of graphs also contributes to memory costs. Continuous batching documentation
The reporter said that graph counts were limited for some models with fast execution paths, so the problem was less common in the past; it resurfaced after adding an MLA model. The project previously had a graph-count cap, but removed it after measurements suggested that each graph had a low cost. Areas under investigation include changes in CUDA, PyTorch, or model implementations; cache updates; and whether removing the cap was itself a mistake. No conclusion has been reached. Issue report
For deployments, the takeaway is to include graph capture and request-shape variation in capacity assessments. Based on the documented mechanism, measuring only steady-state workloads with fixed lengths and batch sizes may fail to reflect memory peaks across multiple shapes. This remains an engineering inference and has not been validated by measurements in this report. Smaller padding intervals can reduce redundant computation, but the documentation explicitly cautions that they require capturing more unique shapes. Configuration trade-offs
Troubleshooting should also account for version differences: the official API documentation marks max_cached_graphs as deprecated since 5.13, so applying the old limit parameter directly may not work. Useful follow-up details would include the specific affected model, a complete reproduction script, and measurements relating graph count to memory use. As of this review, the issue remains open, with no fix branch or PR listed, so there is not yet a confirmed available solution. Configuration reference, issue status