Archive
推論 runtime、量化、 serving、KV cache、成本與效能。
推論系統
Colibrì, which has recently gained traction in the community, uses a pure C engine to load sparse experts on demand, enabling models with 744B to 2.8T parameters to run on non-data-center hardware. What it lowers is the resident-memory requirement; actual decoding speed remains heavily constrained by SSD bandwidth, expert hit rates, and quantization quality.
代理工具
OpenCodeReview, used internally at Alibaba, has seen a notable surge in open-source adoption this week. It delegates file selection, rule matching, and comment positioning to deterministic code, while LLM agents handle contextual retrieval. Official benchmarks show that it uses about 90% fewer tokens than general-purpose coding agents, at the cost of lower recall.
最新模型
OpenBMB has released MiniCPM5-2B, its training data, and multi-stage weights spanning pretraining through RL, along with GGUF, MLX, GPTQ, and speculative decoding variants. The model retains the standard Llama architecture and a 128K context window, but the SGLang parser required for tool calling may still need to be installed from the main branch.
推論系統
Starting September 13, Hugging Face is removing legacy compute kernels stored in regular model repositories. Users must upgrade `kernels` and switch to the new kernel repository type. The migration also requires explicit API versioning and restricts loading to trusted publishers, but signature verification during downloads is still not enabled automatically.
最新模型
Starting at 04:00 UTC on September 14, DeepSeek is temporarily routing all `deepseek-v4-pro` requests to the new V4.1 Flash. The new architecture compresses the KV cache to 890 bytes per token, but independent evaluations show that its longer reasoning outputs may make the cost per task higher than with the previous-generation Flash model.
代理工具
NVIDIA has updated its Nemotron IMO 2026 resources, releasing SFT/RL weights, datasets, prompts, and complete resumable inference code. The system scored 30/42, but the result depends heavily on three 561B-parameter checkpoints and massive test-time compute, so it should not be treated as the capability of a model answering in a single pass.
最新模型
Pipecat 1.10 adds cross-chunk speech continuity, server-side utterance segmentation, and configurable audio buffering, while supporting newer OpenAI, Anthropic, and MCP SDKs. The release also changes some HTTP, STT, and inference defaults, meaning compatibility of existing containers and voice pipelines cannot be determined through package-level testing alone.
最新模型
Together AI has added Expert LoRA, per-example weighting, pre-training data validation, and real-time metrics, allowing developers to tune MoE expert layers instead of only attention modules. In the company’s experiments, knowledge recall rose from 15% to as high as 89%, but the comparison was limited in scope, and expert and attention adapters currently cannot be used together.
最新模型
Bartowski has re-uploaded the Qwen3.8-27B GGUF files, assigning quantization formats according to each tensor’s sensitivity and publishing reproducible configuration files and KLD measurements. Some old and new files share the same names, but the meaning of `_L` and the actual bit allocations have changed, so deployers cannot compare filenames alone.
最新模型
Wringer first repairs a 2.655 bpw model using temporary LoRA adapters, then re-solves the corrections into integer codes and scales so no adapter is retained at inference time. The released model preserves about 94.65% of the BF16 score across three benchmarks, but the full model is actually 4.69 bpw and lacks a native low-bit kernel.
推論系統
The new release brings disaggregated inference, vLLM multiprocess routing, and multi-worker replica management into a single open-source serving layer. It also allows models to be downloaded in advance, launched from an existing cache, and resumed after interrupted downloads, although the release notes provide no throughput or latency comparisons.
AI 安全
A SANS honeypot captured a semi-automated workflow in which an agent searches for LLM resellers with weak authentication, obtains keys, then tests and aggregates them behind a unified API. This is not fully autonomous replication, but it shows that stolen inference resources can be used to support the next round of scanning and intrusion.