AI 推論基礎設施
NVIDIA Dynamo Prewarms a Standby Engine with Shared HBM Weights, Cutting Failover from 283 Seconds to 7.3 Seconds
Dynamo’s experimental Shadow Engine Failover lets two inference processes map the same GPU weights, avoiding a model reload after a process crashes. NVIDIA’s GLM-5.2 test showed a nearly 39× reduction in recovery time, but the feature currently primarily supports vLLM and cannot handle GPU, node, or cross-node failures.
NVIDIA has added an experimental Shadow Engine Failover feature to the open-source Dynamo inference framework, reducing service degradation time when an inference process dies but the GPU and node remain healthy. In a conventional cold start, HBM-resident weights are released along with the CUDA context, forcing the replacement process to reload the model, allocate the KV cache, perform autotuning, and capture CUDA Graphs. For large models, this can take several minutes.
The new design runs a GPU Memory Service (GMS) alongside each GPU, with an independent sidecar retaining the physical memory pages. Through the CUDA Virtual Memory Management API, the primary engine and a preinitialized shadow engine map the same read-only weights into their respective virtual address spaces, so the standby process does not need to duplicate the entire set of weights. The shadow engine creates its CUDA context, NCCL/NIXL communicators, and CUDA Graphs in advance, while releasing its physical KV cache when idle. After the primary process exits, the kernel releases its POSIX `flock`; the standby acquires the lock, remaps the weights, creates a KV cache, and registers with the router.
NVIDIA deployed an NVFP4-quantized GLM-5.2 model across two B200 nodes, using eight-way tensor parallelism per node, a 200K maximum context length, and an FP8 KV cache, then terminated one worker with `SIGKILL`. Recovery by cold-starting a second worker took 283 seconds, compared with 7.3 seconds through the shadow path. After the failure, p50 time to first token was 23,815 milliseconds and 1,311 milliseconds, respectively. However, this was a single synthetic workload designed by NVIDIA and compared only same-node process failures.
For now, engineering teams can treat the feature only as a non-production evaluation capability. The documentation requires Kubernetes 1.34, DRA v1, and the NVIDIA GPU DRA driver, while vLLM is the primary backend. Hardware, node, and multi-node failures must still rely on conventional rescheduling. The KV cache is also empty after failover, so existing prefixes cannot be reused. The next major milestone will be determining whether the KV cache can be transferred safely and whether the gains can be reproduced with SGLang, TensorRT-LLM, and different model topologies.