Archive
推論 runtime、量化、 serving、KV cache、成本與效能。
推論系統
ReCache rewrites attention and positional encoding so that individual tools or skills can reuse their KV caches across requests. Qwen3 experiments show up to a 3.655× improvement in time to first token, but the method requires model fine-tuning and is not yet a drop-in inference plugin.
推論系統
Teams from Tencent WeChat and the Chinese Academy of Sciences rewrote the Hopper sparse-attention kernel, adding FP8, paged KV cache, continuous batching, and mean compensation for pruned blocks. In H20 tests, kernel-level speedups reached 47.26×, but end-to-end gains were smaller, and the current integration depends on the SGLang fork bundled with the project.
AI 可解釋性研究
The new method treats the distribution of reasoning outcomes as a piecewise-smooth sequence, uses PELT to identify genuine decision points, and applies Dirichlet kernel pooling to suppress sampling noise. Experiments on two 8B models show that it can substantially reduce resampling, though the findings currently cover only tinyMMLU multiple-choice questions.
代理工具
NVIDIA adapted the AVO agent architecture, originally used for evolutionary search over GPU kernels, to interactive reasoning and achieved a 100.00 RHAE score with Claude Opus 5. The results show that agent memory and verification loops can substantially change the performance of the same model, but they cover only public environments, and the comparison with VISTA is not a controlled ablation.
推論系統
Google DeepMind formulates the decision of whether to run costly model evaluations as a “Pandora’s box” search problem, deciding sequentially whether an additional assessment is worthwhile based on its value of information. It achieved the lowest average routing regret plus evaluation cost across all three tests, although its effectiveness depends on calibrated signal distributions and cost settings.
代理工具
One week after releasing Mojo 1.0, Modular has made the KGEN compiler, tools, and standard library available under Apache 2.0 with the LLVM exception. The source code can now be inspected and built independently, though the compiler is not yet accepting external contributions, and MAX remains under a separate license.
硬體
Waymo has disclosed its Robotaxi heterogeneous computing architecture for the first time. Its custom ASICs process camera, lidar, and radar data in real time before passing it to the core inference engine. Because the stated 1,000 TOPS figure does not specify numerical precision or power consumption, it cannot be directly compared with NVIDIA’s or Tesla’s full-vehicle platforms.
最新模型
A new study turns NoThink, Short, and Long into the model-generated first token, allowing it to allocate a reasoning budget for each problem without a separate router. On MATH-500, the method maintains accuracy close to the base model while cutting tokens by 76% on the easier GSM8K benchmark.
推論系統
Axon expresses model architectures as functional specifications with symbolic tensor shapes, then compiles them to PyTorch, Triton, JAX, MLX, and native vLLM. Across 467 inference tests, it accelerated most generative models, though some BF16, encoder, and sequence-to-sequence models still showed accuracy or performance regressions.
代理工具
Four modules sharing Qwen2.5-0.5B and the same LoRA learned radically different composition strategies solely because of different attention masks. Limiting each module to its own evidence fragment made it easier to develop intermediate-value communication that transferred across examples, but the experiment still missed its preregistered overall success threshold by 0.0012.
最新模型
MidTool-Mix synthesizes 20.3 billion tokens from web pages, PDFs, code, and real-world tool interfaces, enabling models to learn tool boundaries, parameters, and workflows before SFT and RL. With the same SFT recipe, Qwen3-4B’s BFCLv3 overall score rises from 39.73% to 50.25%, although deep-search tasks remain entirely unsolved.
代理工具
AFANet converts agent conversations into graphs with temporal and same-agent edges, then uses a two-layer GCN to identify faulty agents and error types. It outperforms several LLM baselines on AEGIS-Bench, but its agent–error pair F1 remains low across datasets.