推論系統
vLLM 0.28.0 Shards Kimi K3’s Shared Expert, Reducing Weight Memory by About 17 GiB per GPU
The release also adds Decode Context Parallel, fused FlashKDA kernels, and adaptive speculative token budgets, while extending DeepSeek V4’s sparse MLA across standard and speculative decoding. Some performance figures cover only individual kernels or specific DSpark configurations, and upgrades must account for breaking changes to defaults and plugin interfaces.
vLLM 0.28.0 focuses on communication, memory usage, and decoding paths for large mixture-of-experts models. The [release notes](https://github.com/vllm-project/vllm/releases/tag/v0.28.0) list 584 commits. Kimi K3 can now use Decode Context Parallel, which partitions a sequence’s decoding context across multiple ranks. FlashKDA prefill and decode now use fused kernels, reducing intermediate tensors and kernel-launch overhead. The sequence-parallel path also combines some all-gather operations. The project reports that the relevant kernels are 1.5× to 3× faster, but this does not represent end-to-end server throughput.
Optional shared-expert sharding has a more direct effect on deployment capacity. Kimi K3’s shared MLP previously replicated the full weights on every rank. With `VLLM_KIMI_K3_SHARD_SP_SHARED_EXPERT` enabled, each rank retains only part of the intermediate dimension, with computation completed through all-gather and reduce-scatter operations. The [model documentation](https://docs.vllm.ai/en/v0.28.0/api/vllm/models/kimi_k3/nvidia/model/) describes the additional collective communication, while the release notes estimate savings of about 17 GiB per GPU. Engineering teams should benchmark whether this memory reduction is sufficient to offset the communication cost on their own topology.
For speculative decoding, the adaptive token budget improved time to first token by about 60% in the project’s DSpark tests. DeepSeek V4 also gains end-to-end sparse MLA support for standard decode, MTP, and DSpark. Model Runner V2 adds E/P/D disaggregation, weight offloading, and a multi-level MTP KV cache. Secondary KV cache tiers can now be offloaded to disk or managed by external modules.
This is not a risk-free update: the default `max_num_batched_tokens` increases from 8,192 to 16,384, potentially changing peak memory usage and scheduling latency. bitsandbytes has moved to an out-of-tree plugin, Transformers has been upgraded to 5.15.0, and several legacy parameters and output fields have been removed. Before deploying the upgrade to production, teams should rerun service-level tests for TTFT, per-user output throughput, GPU memory usage, and client compatibility instead of relying solely on speedup figures from individual kernels.