Inference systems
vLLM 0.28 Adds Disk-Tier KV Cache as Model Runner V2 Supports Disaggregated Encoding, Prefill, and Decode
vLLM 0.28.0 can hierarchically offload the KV cache to disk and exposes an external secondary-storage manager, allowing long-context serving to move beyond the capacity limits of GPU and system memory alone. The release also advances Kimi-K3, DeepSeek V4, speculative decoding, and E/P/D disaggregation, but changes several defaults and dependency interfaces.
The central change in vLLM 0.28.0 extends hierarchical KV cache offloading from CPU memory to disk. SimpleCPUOffloadConnector now supports disk offload, while external modules can implement a secondary tier manager through `module_path`. The system can also report partial-load results, emit tiering metrics, and use a standard CPU layout independent of the parallel topology. This allows servers to trade slower storage for a larger pool of reusable prefixes and greater long-context capacity. However, the practical benefits still depend on SSD latency, throughput, cache hit rate, and request scheduling; disk should not be treated as equivalent to GPU memory.
Model Runner V2 adds encoder/prefill/decode (E/P/D) disaggregation, weight offloading, a multi-level MTP KV cache, encoder CUDA Graph support, and support for attention-free models and pooling workloads. The Rust frontend and gRPC paths gain a standalone renderer, multimodal image inference, data-parallel rank routing, and reinforcement-learning lifecycle controls. The protobuf schema has also been published to Buf, making it easier for large deployments to evolve preprocessing, model execution, and the control plane independently.
Model optimizations focus on Kimi-K3 and DeepSeek V4. The official release notes list a fused FlashKDA kernel, Decode Context Parallel, sequence-parallel GEMM-RS, and shared-expert sharding that can save approximately 17 GiB per GPU. DeepSeek V4 gains sparse MLA, MTP/DSpark speculative decoding, and ROCm support. These figures largely come from internal measurements of specific kernels or configurations and cannot be directly extrapolated to end-to-end throughput.
Before upgrading, users must review the breaking changes: bitsandbytes has moved to a plugin, the minimum Transformers dependency has been raised to 5.15, and two legacy KV/attention options have been removed. The default `max_num_batched_tokens` has increased from 8192 to 16384, and the CUDA Graph capture limit for Blackwell has also been raised. Production environments should remeasure time to first token, tail latency, disk write amplification, and failure recovery before deciding whether to enable hierarchical caching and the new runner.