AI 基礎設施
PyTorch 2.14 Brings Failure Reconfiguration into c10d as Inductor Expands Low-Precision and Multi-Hardware Kernels
The new release strengthens large-scale distributed training with `nccl2`, in-place process group reconfiguration, and a cross-backend Flight Recorder. Inductor also adds NVGEMM, declarative dynamic shapes, and complex tensor compilation, though many interfaces remain marked unstable.
PyTorch 2.14 is focused not on a single model-performance figure, but on moving the compiler, distributed communication, and hardware backends closer to production readiness. The release integrates the former torchcomms work as the nccl2 c10d backend, implementing non-blocking communicators, memory pause/resume, and one-sided RMA windows. Backend and ProcessGroup also gain in-place reconfiguration interfaces, allowing process groups to be rebuilt after node failures without always tearing down the entire job. The Flight Recorder, previously tied to NCCL, now collects data through process group hooks, enabling Gloo and custom backends to provide similar collective traces.
On the compilation side, NVGEMM now allows candidate kernels generated by CuTeDSL/CUTLASS to participate alongside Triton and ATen in autotuning for mm, addmm, and scaled_mm. It can also fuse bias, elementwise operations, and certain reductions. The NVFP4 path requires Blackwell. ShapesSpec and @dynamic_spec allow the same set of dynamic-dimension constraints to be reused across torch.compile, torch.export, and make_fx, reducing discrepancies caused by describing shapes separately for each entry point. torch.switch adds traceable multi-branch control flow, while complex tensors can now experimentally enter torch.compile by splitting their real and imaginary components.
Hardware support also expands to ROCm 7.14, Intel XPU graph capture, and Rubin sm_107. Apple Silicon gains native SVD, eigh, QR, and Cholesky support, while more operators have moved from MPSGraph to Metal. Several caveats remain before upgrading: nccl2 currently supports eager mode only, and many new APIs are marked unstable. Although a Python 3.15 wheel is available, it cannot use torch.compile. The TVM Relay backend, some profiler interfaces, and ROCm 7.1 support have been removed. The project also provides no single end-to-end speedup figure that generalizes across all hardware and workloads, so engineering teams should benchmark again using their own models, topologies, and precision settings.