AI 基礎設施
NVLink 6 Unifies Link Retry, Fault Isolation, and Hot Standby for Inference in a Single Recovery Stack
NVIDIA has disclosed NVLink 6’s multilayer fault-tolerance design, extending from physical-layer bit-error correction to the NMX Controller and Dynamo inference processes. Internal B200 testing reduced the time required to restore service on two workers after a failure from 283 seconds to 7.3 seconds, though checkpointing tools and NCCL edge cases are still evolving.
NVIDIA’s newly disclosed NVLink 6 resilience architecture is designed not merely to improve link availability, but to connect hardware error handling, the network control plane, and model-serving recovery into a single path. At the physical layer, lightweight Forward Error Correction first corrects recoverable bit errors. When errors exceed FEC capabilities, Physical Layer Retry retransmits packets before software intervention is required. If the link fails further, UPHY recalibrates the physical parameters while packets are temporarily held in a replay buffer. The link layer also uses credit-based flow control to prevent packet loss caused by receiver buffer overflow.
At the system layer, an NMX Controller provides redundant paths. When a component anomaly is detected, the domain can enter a “contain and drain” state, stopping new traffic from entering the affected area while maintaining forwarding across the rest of the data plane. If the management CPU restarts, control functions can also migrate to a standby node. This is particularly important for tensor parallel and expert parallel workloads, because instability on a local link can stall every rank in synchronization rather than merely taking a single GPU offline.
On the inference side, Dynamo Shadow Engine Recovery maintains initialized standby processes, pre-establishes their own NCCL and NIXL communicators, and uses GPU Memory Service to share existing model weights, avoiding the need to store another complete copy in HBM. In a failure-injection test using two B200 workers and GLM-5.2, NVIDIA reduced the time needed to restore service on the second worker from 283 seconds with a cold start to 7.3 seconds. After the failure, p50 TTFT was 1,311 milliseconds, compared with 23,815 milliseconds for the cold-start group.
These figures remain vendor measurements from a single model and topology. The more consequential deployment question is which state can be transferred safely. The CUDA checkpoint project still does not support UVM or some exportable IPC memory, while a recent NCCL issue documented a potential permanent deadlock during checkpoint/restore on the shared-memory path. Platform teams should test link degradation, controller restarts, disappearing ranks, and checkpoint restoration as separate failure scenarios, then validate them using actual MTTR, TTFT, and incomplete-request counts. “Lossless” should not be interpreted as meaning there is no application-level interruption.