AI 基礎設施
PyTorch 2.14 Brings Failure Reconfiguration into c10d as Inductor Expands Low-Precision and Multi-Hardware Kernels
The new release strengthens large-scale distributed training with `nccl2`, in-place process group reconfiguration, and a cross-backend Flight Recorder. Inductor also adds NVGEMM, declarative dynamic shapes, and complex tensor compilation, though many interfaces remain marked unstable.
PyTorch 2.14 is focused not on a single model-performance figure, but on moving the compiler, distributed communication, and hardware backends closer to production readiness. The release integrates the former torchcomms work as the `nccl2` c10d backend, implementing non-blocking communicators, memory pause/resume, and one-sided RMA windows. `Backend` and `ProcessGroup` also gain in-place reconfiguration interfaces, allowing process groups to be rebuilt after node failures without always tearing down the entire job. The Flight Recorder, previously tied to NCCL, now collects data through process group hooks, enabling Gloo and custom backends to provide similar collective traces.
On the compilation side, NVGEMM now allows candidate kernels generated by CuTeDSL/CUTLASS to participate alongside Triton and ATen in autotuning for `mm`, `addmm`, and `scaled_mm`. It can also fuse bias, elementwise operations, and certain reductions. The NVFP4 path requires Blackwell. `ShapesSpec` and `@dynamic_spec` allow the same set of dynamic-dimension constraints to be reused across `torch.compile`, `torch.export`, and `make_fx`, reducing discrepancies caused by describing shapes separately for each entry point. `torch.switch` adds traceable multi-branch control flow, while complex tensors can now experimentally enter `torch.compile` by splitting their real and imaginary components.
Hardware support also expands to ROCm 7.14, Intel XPU graph capture, and Rubin `sm_107`. Apple Silicon gains native SVD, `eigh`, QR, and Cholesky support, while more operators have moved from MPSGraph to Metal. Several caveats remain before upgrading: `nccl2` currently supports eager mode only, and many new APIs are marked unstable. Although a Python 3.15 wheel is available, it cannot use `torch.compile`. The TVM Relay backend, some profiler interfaces, and ROCm 7.1 support have been removed. The project also provides no single end-to-end speedup figure that generalizes across all hardware and workloads, so engineering teams should benchmark again using their own models, topologies, and precision settings.