AI 基礎設施
NVIDIA ModelExpress Transfers Weights GPU-to-GPU, Cutting Scale-Out Time for an 806 GiB Model to Under Two Minutes
NVIDIA has detailed ModelExpress’s complete transfer path, allowing new inference replicas to copy weights and compilation caches from existing GPUs, with P2P RDMA as the preferred method. In an eight-B200 test, the company cut DeepSeek-V4-Pro’s time to API readiness from eight minutes to 1 minute 44 seconds, though the result depends heavily on identical GPU models, high-speed networking, and compatible tensor configurations.
On July 24, NVIDIA published a detailed explanation of ModelExpress (MX), a Dynamo component designed to address one of the toughest sources of latency when scaling large-model serving. New workers must not only load hundreds of GiB of weights, but also repeat tensor transformations, JIT kernel compilation, autotuning, and CUDA Graph capture. MX’s control plane locates compatible sources through Redis, Kubernetes CRDs, or service metadata. Its data plane then attempts, in order, P2P RDMA, ModelStreamer, GPUDirect Storage, and finally conventional host-staged I/O.
If an existing replica holds the same model, quantization format, parallelism configuration, and tensor layout, MX uses NIXL to read directly from the source GPU into the destination GPU. Once the new replica is ready, it also joins the source pool, creating a weight fan-out that expands with the number of replicas instead of having every machine repeatedly download from object storage. In shared-disk scenarios, a cache service uses an atomic claim to select a single downloader. NVIDIA estimates that having ten replicas simultaneously retrieve an 806 GiB model would generate roughly 8 TiB of duplicate ingress traffic with the conventional approach.
MX can also transfer compiled artifacts from Triton, DeepGEMM, TileLang, CuTe DSL, and FlashInfer. In a single-node test using eight B200 GPUs, ConnectX-7, vLLM 0.23.0, and TP=8, NVIDIA claims that DeepSeek-V4-Pro’s weights and kernel caches were transferred in under ten seconds, reducing total startup time from eight minutes to 1 minute 44 seconds. A VMM arena can also consolidate per-tensor RDMA memory registrations into a single region.
From an engineering perspective, MX reframes weight distribution from a storage problem into a data-plane problem between serving replicas. The approach can also extend to RL rollout workers proactively pulling updated weights. However, the first worker still requires a cold load, and the source process must remain online. Post-transformation receiving in TensorRT-LLM is currently supported mainly for the Llama family. NVIDIA’s tests also did not cover cross-region deployments, mixed GPU fleets, network congestion, or large-scale concurrent scale-outs. Before deployment, teams should therefore measure fallback behavior and source saturation under their actual topology.