推論系統

推論 runtime、量化、 serving、KV cache、成本與效能。

F5 WAF Adds Per-Application Anomaly Baselines, Turns Scan Findings Directly into Virtual Patching Rules

F5 has added anomaly detection and agentic threat intelligence to Distributed Cloud WAF, while connecting vulnerability scan findings to runtime blocking workflows. The new capabilities could shorten the exposure window, but detection rates, false-positive rates, and latency figures are still based solely on vendor testing.

2026-09-01

AI 基礎設施

HUMAIN Brings MI355X and 800G Ethernet AI Cluster Online but Does Not Disclose Current GPU Count or Power Capacity

HUMAIN is now offering production-grade GPU services in Saudi Arabia built with AMD MI355X, EPYC, and Cisco Silicon One networking. This moves the partnership beyond the “planned capacity” stage, but 250 MW and 1 GW remain targets for deployment beginning in 2027 and by 2030, respectively.

2026-09-01

推論系統

Triton 2.72 Fixes Dynamic Batching Starvation and Response Ordering, Upgrades to CUDA 13.4 Inference Stack

NVIDIA Triton Inference Server 2.72.0 fixes an issue that could cause the scheduler to stop dispatching work while requests were still pending and restores the completion-order semantics of `preserve_ordering`. The release also updates its CUDA, TensorRT, and gRPC dependencies, but no TensorRT-LLM image is available, while security warnings remain for vLLM and Ray deployments.

2026-09-01

DeepSeek-V4-Flash-Vision-Exp Releases 305B Weights, but vLLM Still Cannot Load Vision Tensors

Ten days after launching the API, DeepSeek released the weights, prompt encoder, and minimal PyTorch inference implementation for its first V4 multimodal model under the MIT License. Although the model card shows generic deployment instructions, vLLM community testing found that the model still cannot be loaded directly because 316 vision tensors have no corresponding modules.

2026-09-01

代理工具

MCP Inspector 2.4 Unifies Web, CLI, and TUI in One Package; Deploying MCP Apps Requires Three Local Ports

The latest official MCP Inspector package provides browser, command-line, and terminal interfaces through a single executable, sharing a common protocol and OAuth state core. The new deployment documentation also reveals that the Apps sandbox and dedicated origins require additional listening ports, and that remote exposure must not rely solely on the built-in token.

2026-08-31

推論系統

fal Co-Designs Post-Training and GB200 Inference for H3 Max, Cutting Five-Second Video Generation to About Three Seconds

H3 Max adds new post-training data to MiniMax H3’s open weights and uses a custom inference stack to increase video-generation throughput. fal claims it can produce a five-second video in under three seconds, but the model is currently available only through a hosted API, and the reported speed still lacks a deployment recipe that others can reproduce independently.

2026-08-31

推論系統

gemma4.c Reimplements Gemma 4 E2B Inference in Roughly 700 Lines of Pure C, Outperforming General-Purpose Frameworks in Single-Machine CPU Tests

The open-source gemma4.c project packs the tokenizer, Transformer, KV cache, sampling, and SIMD kernels into a single C file, with no external inference library required. In the author’s Ryzen 7 7700 tests, it outperformed llama.cpp, but the results depend on a fixed model, a single CPU backend, and a specialized data layout, so they should not be treated as a general performance conclusion.

2026-08-31

AI 安全

Speculative Probing Reuses MTP Draft Heads for Real-Time LLM Safety Monitoring with a Few Soft Prompts

Cornell Tech researchers repurpose MTP/Eagle3 modules originally designed for speculative decoding as sequence classifiers, without rerunning the main model. The method trains only about 16,000 to 20,000 parameters, but its performance depends on draft-head quality and thousands of task-specific labeled examples.

2026-08-31

推論系統

vLLM 0.28 Adds Disk-Tier KV Cache as Model Runner V2 Supports Disaggregated Encoding, Prefill, and Decode

vLLM 0.28.0 can hierarchically offload the KV cache to disk and exposes an external secondary-storage manager, allowing long-context serving to move beyond the capacity limits of GPU and system memory alone. The release also advances Kimi-K3, DeepSeek V4, speculative decoding, and E/P/D disaggregation, but changes several defaults and dependency interfaces.

2026-08-31

代理工具

DeepSeek Harness 0.1.2 Alpha Enables Model Selection for Subagents, Completes ACP Permission and Cancellation Controls

The latest DeepSeek Harness release lets a primary agent select providers, models, reasoning effort, and output limits for subagents within its authorized scope. It also completes ACP interfaces for sessions, MCP, permissions, and cancellation. The release expands networking and external-model integrations, but the maintainers explicitly warn that its sandbox and approval mechanisms should not yet be considered secure isolation.

2026-08-31

代理工具

Hillock 0.6 Uses Bipolar Hypervectors to Retrieve Agent Memory, Skipping LLM Calls Below a Threshold

Hillock combines factual triples, Hebbian association weights, and 10,000-dimensional hypervectors to determine on local hardware whether enough data exists to answer a question. Version 0.6 adds per-token MaxSim and multi-hop path encoding, but its current gating accuracy of about 56% shows that it remains a research prototype.

2026-08-31

GSQ-RCO Allocates Bits by Tensor Sensitivity, Compressing Qwen3.8-27B into an 8.4GB Standard GGUF

ISTA DASLab has released three mixed-precision quantized versions of Qwen3.8-27B. GSQ learns the quantized values, while RCO selects the precision for each tensor under a fixed capacity budget. The smallest version is just 8.4GB and can be loaded directly by llama.cpp, Ollama, and LM Studio, although the near-lossless claim is still based on the team’s own evaluation.

2026-08-31