推論系統

推論 runtime、量化、 serving、KV cache、成本與效能。

vLLM Supports 16K Embeddings on Ironwood TPUs, StepPool Preserves Chunked Prefill State

Google and vLLM have added tensor alignment, compilation warmup, and pooling state that accumulates across chunks for Qwen3 embedding models. Official results report roughly 84,000 tokens/s on four Ironwood TPUs, but no cost-equivalent GPU comparison or real-world retrieval quality testing has been provided.

2026-08-28

Granite 4.2 Trains 8B/30B Models with Reinforcement Learning in Agentic Environments, Releases FP4, FP8, and GGUF Versions

IBM has released three Apache 2.0-licensed dense models in 3B, 8B, and 30B sizes, featuring switchable reasoning modes and native tool calling. The 8B and 30B models additionally underwent agentic reinforcement learning in codebase, terminal, and search sandboxes, although IBM’s official materials provide inconsistent accounts of their foundational training lineage.

2026-08-27

硬體

OpenAI Jalapeño Delivers 1.5–1.9× Higher Throughput per Watt on Three Open Models, but Long-Context Agent Workloads Remain Untested

OpenAI has published the first benchmark results for its in-house inference ASIC, which uses a single accelerator architecture for both prefill and decode. SemiAnalysis observed and verified the testing process on site, but the data was still supplied by OpenAI, while production silicon and AgentX results have yet to be validated.

2026-08-25

GitHub Repo

Transformers 5.16 Switches to the DTensor Parallel Backend and Adds NVFP4 and Per-Layer KV Cache Configuration

The latest Hugging Face release replaces the legacy tensor-parallel implementation with DTensor and adds initial pipeline-parallel inference and on-the-fly NVFP4 quantization. The upgrade also changes TP, Fuyu, and certain audio model behaviors, requiring regression testing of existing training and inference pipelines.

2026-08-27

代理工具

Planetary Prediction Engine Automatically Assembles Geospatial Data and Models, Achieving 83.3% Recall@10 for Ebola Hotspots

A Google team has introduced PPE, which turns natural-language questions into pipelines for data search, feature fusion, model selection, and geospatial prediction. The system outperformed human-built baselines across several evaluations by the authors, although its epidemic results currently cover only a single Ebola outbreak in the Democratic Republic of the Congo.

2026-08-27

硬體

LLMscope Reads Model Weights and Inference State on FPGAs Using Laser Voltage Imaging

A WPI team demonstrates how to recover quantized weights, activations, and other LLM assets from FPGA registers and block RAM. The attack requires physical access to the chip and replayable states, and the experiments did not fully extract a commercial large language model.

2026-08-27

Code World Model Stores World State as Executable Programs, Then Hands It Off to a Video Model for Rendering

The new framework lets a coding agent maintain rules and long-term consequences, then compiles the state into depth and semantic proxy videos that constrain MiniMax-H3’s visual generation. The prototype demonstrates five sets of controlled scenarios but provides no quantitative benchmarks, ablation studies, or real-time generation results.

2026-08-27

GLM-5.3-Flash Cuts Active Parameters to 18B with Hybrid Linear/Sparse Attention

Z.ai has released the weights for its first natively multimodal GLM-5 model. Of its 320B total parameters, only 18B are activated at a time, and it supports context lengths of up to 1 million tokens. Official measurements show 3× less attention computation and a 4.4× smaller KV cache than GLM-5.3, though the performance and cost figures remain largely vendor-reported.

2026-08-27

推論系統

vLLM 0.28.0 Shards Kimi K3’s Shared Expert, Reducing Weight Memory by About 17 GiB per GPU

The release also adds Decode Context Parallel, fused FlashKDA kernels, and adaptive speculative token budgets, while extending DeepSeek V4’s sparse MLA across standard and speculative decoding. Some performance figures cover only individual kernels or specific DSpark configurations, and upgrades must account for breaking changes to defaults and plugin interfaces.

2026-08-27

代理工具

MLPerf Measures the Full RAG Pipeline for the First Time, but Performance Mode Fixes the Retrieval Path

MLCommons has introduced an end-to-end RAG inference benchmark that separately measures vector database construction and multi-hop question-answering throughput. To control uncertainty in generation, performance tests replay prerecorded inputs for each stage, so they do not fully represent the dynamic behavior of online RAG systems.

2026-08-27

AI 安全

StepGuard Reviews Agent Actions Before Tool Execution, but Adds About 2.53 Seconds of Inference per Task

A team including the Shanghai AI Laboratory has released StepGuard, a 4B-parameter model that uses paired safe/dangerous trajectories and Balance-GRPO to calibrate overblocking and missed detections. It substantially reduces attack success rates in two agent environments, but its training corpus and data generator have not yet been released, and it still misses some policy violations.

2026-08-27

推論系統

AgentX 1.0 Replays Million-Token Inference Workloads from 393 Real-World Agent Traces

SemiAnalysis has released AgentX, featuring long-context, multi-turn, and concurrent subagent workloads so inference systems are no longer compared solely using fixed-length prompts. The dataset preserves request timing and KV-cache reuse patterns, but its sources are concentrated on Claude Code, and it measures serving-system efficiency rather than programming-task correctness.

2026-08-26