GitHub Repo
PyTorch Community Proposes NCCL2 Memory Pool Fix After Later Allocations May Miss Symmetric Communication Kernels
An October 13 PyTorch report says the NCCL2 backend may fail to register new segments added to a symmetric memory pool, causing collective communication to fall back to ring kernels. The proposed fix includes tests on two GB200 GPUs and kernel-path comparisons, but has not been merged and includes no end-to-end performance data.
On October 13, PyTorch received a report of a distributed communication regression: with the NCCL2 backend, registering an empty symmetric memory pool and then allocating tensors from it may fail to enable the expected symmetric communication path. The reported setup used two GB200 GPUs, CUDA 13.0, and NCCL 2.30.7, and covered both development and nightly builds. The author links the issue to the default backend switch in 2.15, but the impact on stable releases still needs to be confirmed. Issue report
The gap concerns the registration lifecycle of memory segments. register_mem_pool(pool, symm=True) creates symmetric windows only for the segments present when it is called; segments added later receive ordinary registration without retaining the symmetric attribute. In the report, the profiler shows that AllReduce therefore uses a ring kernel; an AllGather configured with zero-CTA also fails to use the Copy Engine. Reproduction and analysis
This affects how engineers assess communication and computation overlap. NVIDIA documentation explains that symmetric windows require every rank to use the same buffer offset and are also necessary for zero-CTA optimizations. Subject to hardware, driver, and collective-operation constraints, the Copy Engine can perform communication without using CTAs. It follows that missed registration may undermine the planned resource allocation, but the available evidence cannot be translated into a fixed training-speed loss. NCCL technical documentation
The proposed fix stores a symm flag for each pool and registers a window when a new segment is allocated. The author’s targeted validation reports that four added kernel-selection tests pass; with the fix disabled, NCCL2 tests fail while the legacy backend still passes. Reviewers also requested an explicit error if symmetric registration fails, to avoid silently falling back. However, window registration is a collective operation, so ranks must still allocate segments in the same order. Fix and tests
As of this review, the proposal remains open and cannot yet be treated as a released fix. Teams using symmetric memory pools can first compare kernel paths before and after registration, then evaluate the legacy-backend workaround listed in the report in a test environment. Follow-up should track whether the change is merged and included in a release, as well as its latency, bandwidth, and overlap effects on real workloads. Current evidence supports a communication-path regression; it does not indicate incorrect tensor computation results. Proposal status