推論系統
vLLM Streams Hidden States Across Nodes with Mooncake to Train a DSpark Draft Model for the 2.8T-Parameter Kimi K3
vLLM and Red Hat AI have released a 5-billion-parameter DSpark draft model for Kimi K3, along with a complete recipe for extracting and transferring hidden states across nodes. In official tests, single-user generation speed increased from about 110 to 435 tokens/s, though the results depend on a GB300 cluster and a specific inference workload.
The vLLM team has trained and released a DSpark speculative model for the 2.8-trillion-parameter Kimi K3, addressing the difficulty of fitting such a large model on a single machine by separating inference and training into independently scalable planes. Even with the Kimi K3 weights quantized to 4-bit, extracting hidden states still requires at least two GB300 nodes. The team therefore ran the target model on two four-GPU nodes, trained the draft model on another four-GPU node, and used Mooncake to stream intermediate states and target logits over RDMA or TCP.
The released draft model has five layers and approximately 5 billion parameters, proposing eight tokens per iteration. Unlike EAGLE-3, which generates drafts token by token, DSpark first predicts an entire block in parallel, then uses a low-rank Markov head to restore conditional dependencies between adjacent tokens. A separate confidence head estimates the probability that the target model will accept each position, allowing the scheduler to shorten or extend the verification prefix based on load. Across nine evaluation categories, the average accepted length was 4.11 tokens per iteration, reaching 6.42 for mathematical reasoning and 4.96 on HumanEval.
In the team's published mathematics workload, single-stream performance increased from about 110 to 435 tokens/s. As concurrency rose from 1 to 16, aggregate throughput increased from 177 to 683 tokens/s, while median time to first token rose only from 379 to 479 milliseconds. A 378K-token LongBench-v2 case achieved 5.31 output tokens per iteration, indicating that long context does not necessarily cause draft acceptance rates to collapse.
The engineering value extends beyond a single checkpoint: the Apache 2.0-licensed Speculators project now includes DSpark, a cross-node hidden-state connector, training commands, and a Hugging Face format that can be loaded directly into vLLM. However, the performance figures come from the same team, using GB300 NVL72 systems and a carefully tuned software stack. Different prompt distributions, sampling settings, older GPUs, or network interconnects could offset the draft model's additional computation. The next question is whether third parties can reproduce the cost-benefit results—and whether Mooncake transfers and the five-layer draft model remain economical in multi-tenant serving environments.