推論基礎設施
vLLM 0.29 Makes Model Runner V2 the Default for All Models, With Upgrade Implications for Caching, Communication, and Startup Interfaces
vLLM 0.29.0 completes the transition to Model Runner V2 as the default while expanding speculative decoding, reinforcement learning weight synchronization, and support for multiple hardware backends. This is not a minor update that can be adopted by simply replacing the image: existing deployments must revalidate memory configurations, output consistency, and removed interfaces.
vLLM 0.29.0 incorporates 594 commits, with the central change being that Model Runner V2 (MRV2) is now the default execution path for all models. The new version estimates KV cache capacity based on the actual memory reserved by CUDA Graph, supports sharded sampling and prompt embeddings, and fixes memory lifecycle issues involving sleep mode, shutdown, and multimodal caching. As a result, upgrading may change the number of sequences that can be accommodated and the graph capture configuration. Even if the API remains unchanged, existing GPU memory tuning results should not be carried over without validation.
For large-scale inference, version 0.29 adds admission control based on the number of queued requests and tokens, allowing services to limit queue growth before becoming overloaded. The distributed KV path also gains asynchronous loading, request-level P2P offload, and additional integrations with Mooncake, NIXL, and context parallelism. With `sharded_rdt` weight synchronization for reinforcement learning workflows, each worker can receive only its own tensor-parallel or expert-parallel shard, avoiding repeated transfers of the full weights between nodes. Speculative decoding can now also report per-request acceptance rates, making it easier to determine whether the draft model is actually reducing computation.
Hardware updates span NVIDIA, AMD, Intel, and CPUs. Some of the gains reported by the project include a 9% to 25% improvement in time to first token for Mamba prefix caching and a 33.6% reduction in end-to-end latency from specific Blackwell autotuning. These results come from project tests on particular models and platforms and should not be generalized to typical services.
The migration risks are equally clear: ten legacy model architectures and the PyAV video backend have been removed, the legacy Python API server startup method has entered deprecation, and the default behavior of FlashInfer all-reduce and some prefix-caching features has changed. Engineering teams should run pre- and post-upgrade regression tests using fixed models, batch shapes, and hardware, paying particular attention to CUDA Graph GPU memory usage, prefix-cache logits, pipeline parallel correctness, and ROCm fallback behavior. Although MRV1 remains available for unsupported features, it is no longer the primary execution path.