vLLM publishes Qwen3.8-2.4T disaggregated inference benchmarks, reaching about 5,000 total tokens per second per GPU

On September 21, the vLLM team published benchmarks of disaggregated prefill and decode serving for Qwen3.8-2.4T on a GB300 NVL72 cluster. Peak combined input and output throughput reached about 5,000 tokens per second per GPU, while a low-latency configuration generated roughly 180 tokens per second per user. These results occupy different points on the performance tradeoff curve across configurations and cannot be combined into a guarantee for a single service. [Benchmark report](https://vllm.ai/blog/2026-09-21-qwen38-pd-serving)

This deployment approach assigns prompt processing and incremental generation to separate instances, then transfers cached data through a connector. Its engineering value lies in tuning time to first token and inter-token latency independently, reducing tail latency caused by new prompts interrupting decoding. The official documentation still labels the feature experimental and explicitly states that disaggregation itself does not improve throughput. These results should therefore be viewed as the outcome of the complete hardware and configuration setup. [Disaggregated prefill documentation](https://docs.vllm.ai/en/latest/features/disagg_prefill/)

Estimating cache requirements is central to tuning. The model’s 92 layers comprise 69 Gated DeltaNet layers and 23 full-attention layers. The former store recurrent state for each request, while the latter maintain a key-value cache that grows with the token count. This hybrid architecture means memory capacity requirements cannot be estimated from context length alone. The model card also lists support for multi-token prediction, and additional memory must be reserved when speculative decoding is enabled. [Model card](https://huggingface.co/Inferact/Qwen3.8-2.4T-A95B-NVFP4), [cache analysis](https://vllm.ai/blog/2026-09-21-qwen38-pd-serving)

The experiments fixed input length at 8,192 tokens and output length at 1,024 tokens, sweeping concurrency from 1 to 2,560 requests using NVFP4 weights and a specified vLLM development container image. The team first measured the two stages separately, then combined configurations. All selected configurations achieved 95% accuracy on GSM8K, but a single mathematics benchmark is insufficient to validate the quantized model’s quality on Chinese-language tasks, coding, and long-running tasks. [Experimental setup](https://vllm.ai/blog/2026-09-21-qwen38-pd-serving)

The reproduction materials use NVIDIA’s srt-slurm recipes repository, which organizes sweep configurations by model, GPU, and framework. The repository stores reusable configurations, while the srt-slurm project handles the actual execution logic. Versions of both are therefore part of the experimental conditions; pinning model weights alone is insufficient. Deployers should pin code and model versions, verify cache-block compatibility between the two sides, and include prefill hardware in cost calculations. A more useful next step is to repeat the benchmarks with real request lengths, arrival rates, and latency thresholds, measuring cost per successful response and checking failure and retry rates. The published peaks cannot yet be translated directly into production serving capacity. [Configuration repository](https://github.com/NVIDIA/srt-slurm-recipes)