推論系統

推論 runtime、量化、 serving、KV cache、成本與效能。

DeepSeek Open-Sources DeepInfer Inference Engine, Supporting Two Accelerator Types but Still Missing the Serving Layer

On October 20, DeepSeek open-sourced DeepInfer, releasing an offline inference engine for V4.1-Flash with several decoding optimizations. The code is licensed under Apache 2.0, but a full deployment still requires teams to integrate KV caching, prefill-decode disaggregation, and load balancing themselves.

2026-10-21

GitHub Repo

AWS Publicly Tests TorchNeuron, Bringing Native PyTorch Execution to Trainium; Compilation Still Limited to Static Shapes

On October 20, AWS released the open beta of open-source TorchNeuron, supporting PyTorch eager execution and model compilation on Trainium2 and Trainium3. The source documentation also lists limitations for dynamic shapes, data types, and operator compatibility, so migrations still require item-by-item validation.

2026-10-21

GitHub Repo

Transformers Proposes Unified Quantization Architecture; MXFP4 Migration Could Change Loading Behavior on Older GPUs

On October 20, Transformers maintainers proposed a shared block quantization architecture covering formats such as FP8 and MXFP4, along with mixed-precision loading. The MXFP4 draft also raises the hardware requirements for quantization kernels, potentially changing execution on A100, T4, and CPU; it has not yet been merged.

2026-10-21

推論系統

llm-d Releases 1.0, Strengthening Distributed Inference Operations with Multi-Tenant Traffic Control and Stable APIs

On October 20, llm-d released version 1.0, adding multi-tenant quality of service, coordinated upgrades, and agent session routing to its inference stack. The release provides a stable v1 configuration API, but priority policies still require explicit configuration, and failover capabilities remain on the roadmap.

2026-10-20

GitHub Repo

Open WebUI Clarifies Context Compaction Limits: A Huge Single-Turn Reply Can Still Push a Conversation Beyond the Window

A technical discussion on October 20 noted that a small number of messages containing extensive tool output may exceed the threshold without triggering context compaction. Project members said this is expected behavior under the current message-splitting logic, and that compaction between tool turns in newer versions can mitigate the issue but cannot directly recover every existing conversation.

2026-10-21

GitHub Repo

Transformers Reports Continuous Batching Performance Issue; Unused KV Cache Capacity May Also Slow Generation

An October 20 Transformers report found that, with Flash Attention in continuous batching, increasing the cache allocation may increase runtime even when most of the cache is unused. Related TRL tests also showed slower training; no official fix release has been confirmed yet.

2026-10-21

GitHub Repo

Transformers Maintainers Confirm a Sampling Gap in Serve; Positive Temperature May Still Use Greedy Decoding

A Transformers report on October 19 found that some models may not enable random sampling when given a positive temperature through serve. Maintainers confirmed the issue and said they would rework the fix, but no confirmed release containing a fix was available.

2026-10-20

MiniMax Releases M3.1 Flash, Targeting Low-Cost Agent Tasks and Million-Token Context

MiniMax launched M3.1 Flash on October 19, tuning sparse inference to improve speed and lower the cost of agent tasks. The model is available through an API, but its weights have yet to be released, and its official performance figures await independent verification.

2026-10-19

Inception Releases Mercury 3, a Diffusion Language Model with One Million Tokens of Context

Inception launched Mercury 3 on October 19, expanding its context window to one million tokens and claiming output speeds of up to 1,068 tokens per second. Application developers have begun integrating it, but the company’s speed and quality comparisons still await independent evaluation.

2026-10-20

GitHub Repo

Transformers Community Reports Qwen4-Exp Cache Reset Gap That May Change Sparse Attention Results

A Transformers community report on October 19 says leftover position information may alter sparse attention selection when Qwen4-Exp reuses a reset cache. The report includes a reproduction and a local patch, but maintainers have not confirmed it, and evaluation with pretrained models remains incomplete.

2026-10-20

GitHub Repo

Ollama 0.40.7 Release Candidate Adds MLX Inference Support for PPLX Decider v1.2

Ollama released version 0.40.7-rc0 on October 18, adding MLX backend support for PPLX Decider v1.2. The update covers non-causal attention, decision readout, and image processing, but local performance and post-quantization probabilities still need validation.

2026-10-19

GitHub Repo

SGLang 0.5.22 Adds Deferred KV Cache Allocation Option to Improve Certain Mixed Short- and Long-Prompt Workloads

Released on October 19, SGLang 0.5.22 lets decode nodes in disaggregated deployments wait until prefill is complete before allocating KV space for a request. Official tests on specific workloads showed higher throughput but worse tail inter-token latency; the feature remains optional.

2026-10-19