推論系統

推論 runtime、量化、 serving、KV cache、成本與效能。

TRACE Rewrites Agent Skills from Successful and Failed Trajectories, Raising the Consistent Pass Rate on Hidden Tests to 70%

Xiaomi’s team introduced TRACE, which iteratively updates a retrievable Skill Bank by contrasting successful and failed trajectories without modifying the underlying model weights. It increased the share of CAR-bench hidden-set tasks passed in all three runs, but also raised average token usage, cost, and latency.

2026-08-25

GitHub Repo

Headlong Implements a Persistent Agent With an Infinite Bash Loop, Thinking and Acting Even When No One Is Asking

Laude Institute and MIT have open-sourced Headlong, a Bash-based system with fewer than 10,000 lines of code that treats messages as asynchronous observations within a continuous stream of thought rather than as new sessions. The design enables research into long-term memory and autonomous action, but it also turns token costs, permission isolation, and cross-user data boundaries into persistent risks.

2026-08-25

NVIDIA Adds Two-Stage Generation for MiniMax H3, Cutting 10-Second Video Latency to 14.9 Seconds on a Single GB200

H3 Super Acceleration first creates a low-resolution draft with four denoising steps, then uses LTX and Sol-Attn to complete high-resolution refinement in three steps. NVIDIA reports that 768p, 10-second video generation is 27.7× faster than the SGLang baseline, but the acceleration changes the sampling path and is not a pure kernel optimization at equivalent quality.

2026-08-25

推論系統

Ray 2.58 Completes KV-Cache/Token-Aware Routing, but the New Router Remains Alpha

Ray Serve LLM now centralizes prompt tokenization, cache-hit estimation, and replica selection at the ingress, then passes tokens directly to the inference engine. The design can reduce repeated prefill and tokenization, but no updated performance data has been published, and the control plane introduces additional synchronization overhead.

2026-08-25

推論系統

AWS Uses a Small Model to Compress RAG Evidence First: Input Tokens Cut to One-Tenth, but Request Latency Rises 12%–19%

The new architecture adds a low-cost model call between retrieval and the primary model, retaining only verbatim passages relevant to the question and their source identifiers. AWS’s internal tests recorded a roughly 30% cost reduction, but the evaluation relies on a single enterprise corpus and an LLM judge.

2026-08-25

ReWorld Uses a Pose-Indexed Landmark Bank to Fix the KV Budget, Allowing Interactive World Models to Return to Earlier Scenes After Long Journeys

ReWorld assigns short-term action control and long-term scene memory to different attention ranges, then preserves history in a fixed-capacity landmark bank. The authors demonstrate real-time streaming at 704×1280 and 64-second return trajectories, but only the paper and demo page are currently available; neither model weights nor code have been released.

2026-08-25

GitHub Repo

DFlash 2 Drafts Tokens in Parallel for Qwen3.8-27B, but vLLM Nightly Currently Has Loading and VRAM Errors

DFlash 2 improves block-diffusion drafting with a candidate-path selector and dynamic convolution, delivering 2.7× to 3.4× the throughput of autoregressive decoding for Qwen3.8-27B in single-request tests. The latest reproducible issues show that vLLM nightly instantiates the wrong decoder layer and temporarily allocates nearly 4.74 GiB of extra memory before sharing vocabulary weights, potentially preventing startup on 24 GB GPUs.

2026-08-25

硬體

MTIA 300 Integrates 12 RDMA NICs and Collective Communication Engines On-Chip, Avoiding PCIe and Compute-Core Contention

Meta has disclosed the system design of MTIA 300, which accelerates recommendation-model training with built-in NIC chiplets, near-memory reduction units, and compiled HCCL. The company says communication time across 40 accelerators was reduced to roughly one-quarter that of a reference GPU cluster, but provided no comparisons of cost, power consumption, or general-purpose LLM training.

2026-08-25

AI 基礎設施

SageMaker HyperPod Adds Managed Ray: Training, Inference, and Failure Recovery Share the KubeRay Interface

AWS has integrated Ray cluster creation, remote job submission, monitoring, and workspace connectivity into HyperPod on EKS. The interface retains KubeRay and standard Ray APIs, but deployment still requires several add-ons, and AWS has not published cost or performance comparisons.

2026-08-25

AI 基礎設施

NVIDIA Groq 3 LPX Enters Mass Production: Static Scheduling Delivers 3,431 Output Tokens per Second at 100K Context

Groq 3 LPX integrates 256 LP30 chips, 128GB of SRAM, and compiler-prescheduled inter-chip transfers into a low-latency decoding system. Third-party testing shows it leading existing public endpoints in long-context speed, but cost, concurrent throughput, and production-service performance remain undisclosed.

2026-08-25

推論系統

AgentX Publishes Long-Context Agent Inference Results, Replacing Fixed-Length Stress Tests with Real-World Session Shapes

SemiAnalysis has published live AgentX results and a complete replay toolkit for comparing GPUs, inference engines, and cache designs using multi-turn coding-agent traces. The public corpus preserves request lengths, shared prefixes, pauses, and subagent topologies while replacing the original prompts and code with synthetic content.

2026-08-24

AI 安全

Google Agent Identity Auth Manager Is Now Generally Available, Moving Agent Credentials from Application Code to a Separate Control Plane

Google Cloud has made Agent Identity auth manager and two related APIs generally available, centralizing the OAuth credentials and API keys that agents need to call external tools. The legacy IAM Connectors API will not reach GA, so existing preview integrations must migrate.

2026-08-24