推論與執行期系統
LMDeploy 0.17 Introduces DeepEPv2 and Mooncake KV Storage, Expanding Cross-Node MoE Inference Data Paths
LMDeploy 0.17.0 adds the DeepEPv2 expert-parallel backend to TurboMind and connects its KV connector to Mooncake Store. The release also supports Kimi K2.6, non-power-of-two page sizes, and structured responses, but provides no end-to-end performance data across hardware configurations.
InternLM released LMDeploy 0.17.0 on September 1, focusing on communication, memory, and serving paths for large MoE models. TurboMind can now use DeepEPv2 as its MoE all-to-all backend. During inference, each GPU dispatches tokens according to routing results to the devices hosting the target experts, then combines the outputs. The integration includes token dispatch/combine, separate communication groups for expert parallelism and tensor parallelism, multi-node processing, and the `--moe-a2a-backend` configuration option. This means communication-intensive expert parallelism no longer has to rely solely on the existing NCCL path.
Another change adds Mooncake Store support to the KV connector. This enables inference engines to exchange KV cache data through external storage, providing a new integration point for prefill–decode disaggregation, cross-instance cache reuse, and tiered storage. However, the release notes do not publish cache hit rates, network latency, or failure-recovery test results, so it cannot be assumed that this will improve every long-context serving workload. Actual gains will still depend on prompt repetition, interconnect bandwidth, and cache lifetime.
At the model and core-engine levels, the PyTorch engine adds support for Kimi K2.6 and optimizes routing preparation for compact-blocked FP8 MoE, as well as speculative decoding pre- and post-processing. Paged attention and V4 prefill now use PDL, while memory page sizes are no longer restricted to powers of two. The serving layer adds server-side fan-out for `n>1` choices. Both TurboMind and the PyTorch engine can output the `structural_tag` format, and the Anthropic-compatible endpoint fixes handling of inline system messages.
Users upgrading should pay particular attention to two compatibility areas: GPUs older than SM90 revert to the FP8 weight-only fallback, while building DeepEPv2 introduces NCCL version and symbol compatibility requirements. The project did not publish standardized throughput, time-to-first-token, or multi-node scaling curves with this release. Before deployment, users should compare the NCCL and DeepEP backends using their own models, batch distributions, and topology, and test fallback behavior when Mooncake is unavailable.