推論基礎設施
NVIDIA Dynamo Provides Multi-Node Deployment Recipes for Kimi K3; Million-Token Inference Requires at Least 16 GB200/GB300 GPUs
An experimental Dynamo release adds multimodal processing, reasoning, and tool-call parsing for Kimi K3, along with Kubernetes recipes for aggregated inference and disaggregated prefill/decode. The recipes quantify the deployment threshold for the 2.8T-parameter MoE model in terms of rack topology, but they have not undergone formal QA, and the GB200 configuration still has a known startup error.
Following the release of the Kimi K3 weights, the deployment challenge has shifted from “can they be downloaded?” to “how can the model actually be served across multiple nodes?” On July 27, NVIDIA released the Dynamo `v1.4.0-kimi-k3-dev.1` preview, integrating Kimi K3 with the vLLM backend and adding text and image preprocessing, a reasoning parser, a tool-call parser, and Kubernetes templates for GB200 and GB300. This is a subsequent deployment milestone following the earlier model release, not a new model version.
Four recipes cover aggregated inference and disaggregated prefill/decode. The aggregated GB200 configuration uses 16 GPUs across four nodes with TP16, while the disaggregated configuration requires 16 prefill GPUs plus 16 decode GPUs. The aggregated GB300 configuration uses 16 GPUs to create two TP8 replicas, while the disaggregated configuration uses 8 prefill GPUs and 16 decode GPUs. Both use Kubernetes ComputeDomain to preserve the NVLink topology. In disaggregated deployments, NIXL transfers the KV cache: GB300 uses NVLink, while GB200 uses RDMA over InfiniBand.
The routed experts in the weights use MXFP4, while the dense components remain in BF16 and the KV cache uses FP8. The recipes also enable prefix caching, KV-aware routing, and data-parallel sharding of the multimodal encoder, and are specified to support a 1,048,576-token context window. These choices reflect the system bottlenecks of extremely large MoE models: memory formats, KV transfers, network topology, and request routing often have a greater impact on cost than the token throughput of any single kernel.
However, this remains a branch-specific snapshot rather than a QA-gated stable release, and no end-to-end throughput, time-to-first-token latency, or cost curves across different context lengths have been published. NVIDIA also lists a known issue in which `MODEL_PATH` is not set for the GB200 disaggregated frontend, potentially causing the container to start with the literal string instead of the intended path. Evaluation teams should first apply the fix, validate the storage class and model cache, and then use their own agent traffic to measure the KV-cache hit rate and prefill/decode ratio. The existing hardware tables prove only that the recipes exist, not that they are economically viable.