DeepSeek Open-Sources DeepInfer Inference Engine, Supporting Two Accelerator Types but Still Missing the Serving Layer

On October 20, DeepSeek made DeepInfer public, releasing what it describes as an inference engine used for large-scale production serving. It currently supports DeepSeek-V4.1-Flash and is limited to NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs. The release lets developers inspect the model-specific execution and scheduling designs directly. Official repository

Key features include DSpark speculative decoding, which uses draft tokens and confidence estimates to schedule verification. Asynchronous scheduling overlaps CPU batch scheduling with device computation. The engine also provides constrained decoding, which can restrict output according to regular expressions, JSON Schema, or EBNF grammars—a practical feature for agent systems that need tool arguments to be parsed reliably. Features and configuration

The deployment boundaries matter too. The README states that this release provides offline inference and does not yet include the KV cache system, prefill-decode disaggregation, or load balancing across data-parallel replicas required for a complete serving setup. Engineering teams will therefore still need to assess request routing, cache lifecycle management, and peak traffic handling; they cannot estimate deployment costs based solely on core execution speed. Deployment scope

The accompanying deepseek-recipe addresses another factor that can affect reproducibility: it converts Messages, Chat Completions, and Responses requests into a unified conversation format, then encodes them as prompts or tokens and parses model outputs. The official model card says the model does not include a Jinja-format chat template. For production deployments, teams can use this Rust library and its Python bindings, but inference, tool execution, and HTTP transport remain the caller’s responsibility. Model card

From an engineering perspective, the value of this open-source release is that it makes both model execution and protocol conversion implementations available for inspection. The next thing to watch is whether the community can reproduce performance under fixed prompt formats, concurrency levels, and hardware conditions, and what latency and costs look like after the serving layer is added. The available speed figures are still official measurements and do not establish that typical clusters will see the same gains. The engine’s Apache 2.0 license and the model weights’ MIT license should also be checked separately. Engine overview, weights license