推論系統

推論 runtime、量化、 serving、KV cache、成本與效能。

Ultralytics 8.4.143/8.4.144 Integrates YOLO26 INT8 QAT Across Training, Checkpointing, and TensorRT Export

YOLO26 can now run quantization-aware training with `quantize=8`, save quantization ranges in checkpoints, and directly export ONNX models with Q/DQ nodes or TensorRT engines. The subsequent 8.4.144 release adds guidance on quantization gains and numerical stability fixes, while also highlighting the deployment path’s clear limitations.

2026-09-09

推論系統

vLLM 0.29 Makes Model Runner V2 the Default for All Models, With Upgrade Implications for Caching, Communication, and Startup Interfaces

vLLM 0.29.0 completes the transition to Model Runner V2 as the default while expanding speculative decoding, reinforcement learning weight synchronization, and support for multiple hardware backends. This is not a minor update that can be adopted by simply replacing the image: existing deployments must revalidate memory configurations, output consistency, and removed interfaces.

2026-09-09

NeoHorse-1 Feeds Agent Execution Traces Back into Model Training, but “Recursive Self-Improvement” Remains a Single-Round Prototype

NeoHorse-1 uses a router to collect traces from different models completing agentic tasks, then trains 4B and 9B models through curriculum-based SFT and on-policy distillation. Its official aggregate scores exceed those of same-size Qwen3.5 baselines, but it has yet to demonstrate that repeated rounds of self-updating can deliver sustained gains or prevent data drift.

2026-09-09

推論系統

Cohere Packs the Entire Decoding Step Into a Single CUDA Megakernel, Boosting End-to-End H100 Throughput by 25% to 41%

Cohere has open-sourced a dedicated inference engine for North Mini Code that replaces per-operator launches and full-GPU synchronization with a persistent megakernel. Official benchmarks show higher end-to-end decoding throughput than vLLM on a single H100, though support is currently limited to specific models, hardware, and small batch sizes.

2026-09-09

推論系統

Unsloth 0.1.807-beta Switches AMD APUs to Vulkan by Default, Boosting Strix Halo Prefill Performance by Up to 23%

Unsloth Desktop now uses Vulkan for Strix Halo, Strix Point, and AMD iGPUs without ROCm support, while also reducing the MLX KV cache footprint and installation package size. The performance figures come from official tests, but users have reported that the interface still displays ROCm after upgrading, so automatic backend selection still needs verification on real hardware.

2026-09-08

推論系統

Tenstorrent Plugs Into vLLM: A Separate Phased Scheduler and Data-Parallel Path for Mesh Accelerators

A new open-source plugin supports multiple language and vision models on Tenstorrent hardware through the same OpenAI-compatible API, without forking the vLLM core. It also exposes current limitations: prefill and decode cannot be mixed in the same step, while LoRA, speculative decoding, and multi-host serving are not yet supported.

2026-09-08

llama.cpp Adds Recurrent State Rollback for Kimi K3, Freeing Speculative Decoding from Host-Only Checkpoints

A new prerelease saves KDA state and convolution windows at each candidate position so rejected draft tokens can be rolled back correctly. Initial single-node tests show faster low-concurrency inference, but state memory grows substantially with rollback depth and the number of serving slots.

2026-09-08

代理工具

Ripwire 0.4 Uses a Code Graph to Narrow the Search Scope for Agents, but Saving Tokens May Not Reduce Task Costs

Ripwire uses tree-sitter, a symbol relationship graph, and Personalized PageRank to generate reproducible file rankings and change-impact scopes for coding agents. First-party tests show better localization rates than five comparison tools, but an independent review found that compressing context may still degrade answer quality and increase overall agent costs.

2026-09-08

EXAONE Finance Releases 202M-Parameter Attention-Free Time-Series Model, with Separate Licenses for Weights and Code

LG AI Research replaces self-attention with causal convolutions and a group-aware MLP, allowing computational costs to scale linearly with sequence length and the number of variables. The model outperformed 43 baselines on the team’s own FinVerse benchmark, but noncommercial model weights and independently unreproduced investment performance limit its immediate deployment value.

2026-09-08

推論系統

KVMem Pages Agent History to Memory and NVMe, Letting a 24GB Laptop GPU Address a Million-Token Workspace

Instead of compressing overflow agent history into summaries, KVMem preserves precomputed KV states and loads relevant blocks for each query. The authors demonstrate a logical million-token workspace using Qwen 27B on an RTX 5090 Laptop GPU, though model support and performance figures remain closely tied to their custom runtime.

2026-09-08

推論系統

ACE Skips Low-Contribution MoE Experts Per Token, Accelerating Prefill by Up to 2.25× on A100

ACE estimates expert contributions from SwiGLU weights and router directions without retraining the model or using calibration data. The authors achieved better quality at high skip rates across three MoE models, though the performance results remain limited to a single A100 and thresholds matched to the workload.

2026-09-08

AI 基礎設施

Megatron-LM Expands Prefix-Aware Routing and Dynamic Batching, Enabling Hybrid Model Inference to Share KV Cache and Mamba States

NVIDIA summarized Megatron-LM changes from the first week of September, advancing the data-parallel inference frontend, prefix caching, and variable-length sequence processing toward a more complete serving path. The new features improve state reuse for hybrid Transformer/SSM models, but the company has not yet published throughput, time-to-first-token, or cross-node scaling data.

2026-09-08