推論基礎設施
Xinference 3.4 Adds Prefill/Decode Disaggregation, Cross-Worker Replica Scaling, and Resumable Model Caching
The new release brings disaggregated inference, vLLM multiprocess routing, and multi-worker replica management into a single open-source serving layer. It also allows models to be downloaded in advance, launched from an existing cache, and resumed after interrupted downloads, although the release notes provide no throughput or latency comparisons.
Xorbits released Xinference 3.4.0 on September 11. Rather than merely expanding the model catalog, this update focuses on completing the data paths and lifecycle controls needed for cluster deployments. The new version adds prefill/decode disaggregation, allowing input prompt processing and token-by-token decoding to be assigned to different worker processes. It also supports routing for vLLM’s native multiprocessing executor and replica scaling across multiple workers. For operators handling a mix of long prompts, short responses, and long-form generation traffic, these capabilities provide a way to allocate compute resources independently. However, the actual benefits still depend on model size, KV cache transfers, batch shapes, and network latency.
For model management, version 3.4.0 adds a download-only workflow, resumable cache management, and the ability to launch models directly from an existing cache. Models can also be retrieved by passing an `oci://` URI to llmman serve. This allows air-gapped environments, bandwidth-constrained nodes, and rolling deployments to separate weight transfers from service startup, reducing the cost of restarting large file downloads after a failure. New engine registration hooks also make backend capability registration easier to extend. Existing documentation indicates that users must still explicitly select vLLM, SGLang, llama.cpp, Transformers, or MLX when launching a model with Xinference, and that the model format and quantization method must be compatible with the chosen engine. In cluster mode, the supervisor aggregates the engine capabilities reported by workers.
Compatibility updates include Transformers batch inference for Gemma 4, MiniCPM5-2B, dots.ocr, MonkeyOCR, Fish Audio S1-mini/S2-Pro, and multimodal embedding and reranking capabilities. For users in Taiwan, the Web UI also adds the `zh-TW` locale.
However, the release page provides no topology examples, fault-tolerance semantics, or benchmark data for prefill/decode disaggregation. Nor does it explain how in-flight requests are drained during cross-worker scaling. Before upgrading production systems, operators should use their own prompt-length distributions to measure time to first token, decoding throughput, and cross-node traffic. They should also verify the hashes, provenance, and permissions of resumed cache contents; resumable downloads are not a substitute for supply-chain integrity checks.