推論系統
NVIDIA Dynamo Splits Multimodal Inference into Three Stages, Cutting Time to First Token by Up to 42.2% Under Mixed Traffic
Dynamo can now assign visual encoding, LLM prefill, and decoding to separate workers and different GPU tiers, preventing image requests from blocking text-only traffic. Although NVIDIA’s tests show improvements of up to 5× in time to first token and 7× in end-to-end latency, long outputs or large dense models may offset the benefits of disaggregation.
NVIDIA has detailed Dynamo’s implementation of multimodal Encode–Prefill–Decode (EPD) disaggregation and its deployment criteria. In conventional aggregated serving, the same worker sequentially handles media decoding, the vision Transformer, LLM prefill, and token-by-token decoding. As image or video workloads increase, visual encoding not only delays their own time to first token (TTFT), but also causes head-of-line blocking for text-only requests in the same batch. Dynamo instead has the encoder generate visual embeddings and transfer them via NIXL to prefill and decode workers, allowing the three stages to be scheduled, batched, and scaled independently.
Deployments can retain an aggregated configuration, colocate the encoder and PD worker on each GPU, or move the encoder to a less expensive GPU tier. For the latter configuration, NVIDIA’s tests used two RTX 6000D GPUs for visual encoding and four GB200 GPUs to run the LLM; embeddings were transferred over a UCX RC/TCP network with peak bandwidth of 20 Gbps. The main benchmark used Qwen3.5 122B-A10B NVFP4, with 10 images per request, up to 256 visual tokens per image, and an output length of 1,024 tokens. The colocated configuration reduced TTFT by 58%, while the heterogeneous configuration reduced it by 50%. Under an inter-token latency limit of less than 100 milliseconds, the heterogeneous setup handled 70% more traffic. In a 50:50 text/image traffic test, text TTFT fell from 92.3 to 53.3 milliseconds, while image TTFT dropped from 289.9 to 200.6 milliseconds.
This is not a universally applicable, cost-free speedup. When output length increased from 128 to 2,048 tokens, the heterogeneous EPD configuration’s end-to-end advantage narrowed from 20.3% to 5.2%, while the colocated configuration actually regressed by 2.5%. At the same SLO, a 4B model achieved 2.62× the goodput of the aggregated configuration, but a 27B dense model reached only 0.65×. Engineering teams should first measure the proportion of time spent on the ViT, prefill, and decode stages, as well as the cost of transferring embeddings, before deciding whether to disaggregate. Dynamo’s documentation provides launch paths for vLLM, SGLang, and TensorRT-LLM, but support for images, video, and KV transfers remains inconsistent across backends.