NVIDIA publishes multi-GPU video inference benchmarks, cutting generation latency to about 34 seconds on eight GPUs

On September 21, NVIDIA published video generation benchmarks using Dynamo-Triton with TensorRT across multiple GPUs: Cosmos 3 Nano took an average of 156.6 seconds on one GPU and 34.2 seconds on eight, an overall speedup of approximately 4.58×. The new information concerns serving integration and benchmark results; multi-device backend support was already listed in the 26.07 release notes. [Technical announcement](https://developer.nvidia.com/blog/simplifying-model-serving-across-multiple-gpus-with-nvidia-tensorrt-multi-device-integration-in-nvidia-dynamo-triton/), [release notes](https://docs.nvidia.com/deeplearning/triton-inference-server/release-notes/rel-26-07.html)

The architecture wraps the denoising Transformer in a single gRPC model endpoint, with the backend managing execution contexts, CUDA streams, and NCCL communication across GPUs. The application still uses Diffusers to orchestrate the generation pipeline. According to the Cosmos 3 Nano model card, its architecture includes both an autoregressive Transformer for text and a diffusion Transformer for continuous modalities. These benchmarks therefore cannot be treated as a universal speedup factor for every capability of the full multimodal model. The model card currently reports validation only at BF16 precision; quality and memory requirements at other precisions still need to be checked separately. [Model card](https://huggingface.co/nvidia/Cosmos3-Nano)

Developers must first compile a distributed computation graph, then enable `KIND_MODEL` and the multi-device configuration. Simply expanding the GPU list will not automatically partition a single-GPU engine. TensorRT documentation requires each rank to use its own execution context and buffers, with work submitted concurrently. Collective communication requires all participants to join; a missing rank can cause execution to wait. NVIDIA also provides sample code for checking communication results individually: after setting up a specified collective operation, developers can verify that each GPU's output matches expectations. This can help rule out communication or execution-context issues, but it cannot replace validation of the complete generation pipeline. [Execution documentation](https://docs.nvidia.com/deeplearning/tensorrt/latest/inference-library/multi-device-inference.html), [communication sample](https://github.com/NVIDIA/TensorRT/tree/main/samples/sampleDistCollective)

The benchmarks used fixed settings of 720p, 189 frames, and 35 denoising steps, with one warm-up run followed by five measured runs for each configuration. Measurements included the generation pipeline but excluded model loading and MP4 encoding. The Transformer RPC achieved a 6.09× speedup, exceeding the overall gain and indicating that other stages still limit total latency. Quality checks sampled only five frames and applied error thresholds; the outputs were not pixel-identical. [Benchmark conditions](https://developer.nvidia.com/blog/simplifying-model-serving-across-multiple-gpus-with-nvidia-tensorrt-multi-device-integration-in-nvidia-dynamo-triton/)

Deployment decisions still lack data on concurrent throughput and cost per video. Engineering teams should first confirm the GPU model, interconnect, precision, and engine version, then compare scheduling performance when the same hardware runs single-GPU replicas or distributes requests across GPUs. They should also record queueing time and tail latency under heavy load. These figures support shorter waits for individual requests, but are insufficient to conclude that the same GPU budget can produce more videos. Generation quality with Traditional Chinese prompts and across different scenes also requires separate testing. Revalidation should preserve prompts, seeds, and all deployment settings for comparison.