GitHub Repo
PyTorch Development Build Reports CUDA Graph Copy Bug: Deferred-Copy Tensors May Read Stale Data During Replay
Reports filed on October 7 for PyTorch and TorchTitan indicate that copy-on-write tensors materialized during CUDA Graph capture may omit the copy operation, causing incorrect results on replay. The reports include a minimal reproduction and an MoE inference comparison, but the impact on released versions and an upstream fix remain unconfirmed.
On October 7, a correctness issue was reported in the PyTorch repository: if a copy-on-write tensor created with torch._lazy_clone() triggers an actual copy during CUDA Graph capture, replay may continue reading stale data from the time of capture, without any error during capture. The reported environment was a PyTorch 2.16.0a0 development build, CUDA 13.0, and GB300; the issue can also be reproduced with torch.compile(mode="reduce-overhead"). PyTorch report
The minimal example initializes the input with NaNs, then creates a deferred-copy tensor inside the capture region and calls data_ptr() to trigger materialization. Even after new random values are filled in before each replay, the output still contains NaNs. The report’s analysis points to the allocator using a synchronous cudaMemcpy on an unspecified stream: the copy runs only once during capture and is not recorded as a node in the graph. Reproduction and root-cause analysis
This is directly related to CUDA Graph’s data update mechanism. The official documentation explains that replay reuses the same memory addresses, so users must update input buffers before replay; the performance benefit mainly comes from eliminating CPU-side kernel launch overhead. If an intermediate copy is not included in the graph, updating the original input cannot refresh the copy read by later kernels. Official CUDA Graph documentation
A related TorchTitan case encountered the issue during tensor-parallel MoE inference with Qwen3.5-35B-A3B: the sharding process produces deferred-copy tensors, and Dist-MoE then passes their raw pointers to Triton or CuTe kernels. The reporter also provided a single-GPU H100 test showing that a non-default stream may complete the copy before the data is produced; a comparison using a regular clone() returned the correct result. TorchTitan case
The next engineering step is to build correctness comparisons around deferred copies, raw pointers, and stream boundaries, and to verify whether the copy can be ordered on the current stream and safely captured. As of the review, both issues remain open, and neither page links to a fix PR. The available evidence is concentrated on development builds, so it does not establish that all stable releases or all vLLM deployments are affected. The temporary workaround of using regular copies also lacks a full performance-cost measurement. TorchTitan report and recommendations