GitHub Repo
Transformers proposes Gemma 3n fix to remove host tensor copies blocking CUDA Graphs
On November 7, the Transformers community proposed a fix for repeated CPU tensor copies in Gemma 3n’s AltUp operation that caused CUDA Graph capture to fail. The author reports that graph capture and vLLM serving tests passed, but the proposal has not been merged, and throughput improvements have yet to be measured.
On November 7, the Transformers community submitted PR #50457, proposing a change to the AltUp loop in the Gemma 3n text model to remove an obstacle to CUDA Graph capture. As of a review on November 8, the proposal was still open, so the fix is not yet part of an official release.
The reported issue used Transformers 5.23.0.dev0, PyTorch 2.11.0, and an NVIDIA L4. A small model’s forward pass ran in eager mode, but failed when placed inside torch.cuda.graph because of copies between CPU and CUDA tensors. The author also reported that vLLM’s Transformers model backend encountered a Gemma-3n-E2B-it graph-capture error when compilation was enabled, and could run only with enforce_eager=True. Reproduction case
The patch changes two magnitude-floor operations. Previously, torch.maximum compared the computed result with an epsilon tensor on the CPU, which was then moved to the target device inside the loop. The proposal replaces this with new_magnitude.clamp(min=1e-5). The author reports that this eliminated six synchronization copies per forward pass in the test and produced identical outputs across three floating-point precisions on CPU and CUDA. Implementation and validation details
The change matters because it restores an inference execution path. PyTorch documentation explains that CUDA Graphs can record GPU work in advance and replay it with a single launch, reducing dispatch overhead across Python, C++, and the driver. However, operations that synchronize the CPU and GPU are prohibited during capture. As a result, even a small constant used for numerical stability can block capture of the entire graph if handled on the wrong device. How CUDA Graphs work and their limitations
For deployment, first confirm whether the Transformers model backend is in use. vLLM provides the --model-impl transformers option; you can also check the loaded model’s type. Follow-up points include whether the proposal is merged and included in a package release, as well as latency tests at actual batch sizes and sequence lengths. The author’s passing tests support the proposed fix, but no throughput benchmark has been provided, so the service speedup cannot yet be estimated. vLLM backend documentation