推論系統

推論 runtime、量化、 serving、KV cache、成本與效能。

ASCII Smuggling Shifts from Prompt Injection to Email Attacks as Invisible Unicode Disrupts Classifier Tokenization

Microsoft has uncovered phishing emails that insert invisible Unicode Tag characters into financial keywords, altering the tokens seen by rule-based and machine-learning classifiers. With attacks peaking at more than 2.3 million emails per day, the campaign shows that AI red-teaming techniques are beginning to migrate into conventional cybersecurity offense and defense.

2026-09-06

推論系統

SGLang Adds GPU-Free Simulator to Official Documentation for Replaying Inference Traffic with Latency Predictions

SGLang Simulator intercepts runtime components on the CPU to simulate scheduling, batching, and multi-tier KV caches without actually loading models or executing GPU kernels. The team claims errors below 5% on key metrics for two Qwen3 models running on H20 GPUs, but hardware and model coverage remain limited.

2026-09-06

GitHub Repo

Paddock Open-Sources Rust/CUDA Inference Engine Targeting Concurrent AI Agents on a Single NVIDIA GPU

Truespar has released Paddock’s scheduler, paged KV cache, memory management, and custom CUDA kernels, along with OpenAI- and Anthropic-compatible interfaces. The authors’ benchmarks show it outperforming vLLM and SGLang on some single-GPU workloads, but the results have not been independently reproduced, and Paddock does not support tensor parallelism, ROCm, or Metal.

2026-09-06

GPT-6 Astra API Adds Asynchronous Tool Calling and Mid-Turn Steering as Cybersecurity Capabilities Reach Critical for the First Time

OpenAI’s new flagship model can continue reasoning while tools run in the background and receive mid-execution corrections over WebSocket. Its stronger agentic capabilities come with new migration constraints—and the first Critical rating for cybersecurity capabilities.

2026-09-06

推論系統

llama.cpp 0.4.0 Reduces Large-Model Loading Spikes with Lazy Tensor Loading and Adds Support for the New Qwen3.8 Architecture

The new release adds `--lazy-mode`, a quantization working-memory limit, and row-slab streaming, eliminating the need to load extremely large tensors into RAM all at once. It also introduces preliminary support for Qwen3.8-Flash-Next and Nemotron-3-Puzzle, though performance optimizations and compatibility with some hardware remain incomplete.

2026-09-06

AI research

One-Shot OPD Captures Most Full-Data Distillation Gains With a Single Query, Shifting the Bottleneck to Teacher-Signal Absorption

Researchers found that repeatedly sampling from a single query while giving the student model token-level teacher supervision can, after 300 steps, cover 71.5% of the state reached by full-data training. Adding 16 semantically diverse queries raises coverage to 98.9%, though the findings remain limited to the specified small models, teachers, and benchmarks.

2026-09-06

推論系統

NVIDIA PAIR Dispatches Cross-Device Inference Through a Single Local Endpoint, but Does Not Pool GPU Memory

The open-source Personal AI Router can schedule independent Ollama or LM Studio requests across Windows, Linux, and macOS nodes. It is suited to increasing multi-agent concurrency, but it is not a distributed model execution engine: a single model must still fit entirely on one machine.

2026-09-05

GitHub Repo

K2 Horizon Launches in Six Sizes, With Its MoVA Model Activating Only a Subset of Value Projections per Token

IFM has released the K2 Horizon model family, ranging from 0.9B to 375B parameters. The 36B version extends MoE-style routing to the attention mechanism’s value projections. Weights and deployment recipes are now available, but the organization’s claim of being “fully open source” still runs ahead of what it has actually delivered.

2026-09-05

代理工具

Coder Agent Relay Moves Cursor Tool Execution Inside the Enterprise Network, While Inference Remains in the Cloud

Coder’s Agent Relay enables Cursor Cloud Agents to run commands, read and write files, and access internal services in customer-managed workspaces. This separation can reduce source code exposure, but it is not equivalent to full self-hosting: inference, planning, and session orchestration remain handled by Cursor’s cloud.

2026-09-05

代理工具

GitHub Copilot CLI 1.0.83 Adds CIMD Login Support for MCP and Expands Sandbox File Tool Read Access

The new release lets MCP OAuth clients publish identity metadata at a URL, reducing the need to register individually with authorization servers. It also allows built-in sandbox file tools to read development tool configurations, including `.npmrc` files that may contain tokens, so deployers should review their default permissions.

2026-09-05

推論系統

Random Attention Randomly Evicts KV Cache Entries After Preserving the Prompt, Outperforming Scoring-Based Methods on Long-Reasoning Throughput

Salesforce AI Research found that after protecting the original prompt, randomly retaining reasoning tokens within each KV head can match the accuracy of several content-scoring strategies. In H200 tests with 32K-token outputs, it processed 32% to 43% more tokens than TriAttention, though short outputs and long code prompts did not necessarily benefit.

2026-09-05

Compile by Training Compiles Natural-Language Specifications into Locally Executable Functions on a 0.6B Model

The new method uses a large teacher model to synthesize task data, then trains a function-specific LoRA for each task so subsequent requests do not need to call a remote model. It achieves 83.6% semantic accuracy on a difficult benchmark, but compilation still requires a teacher API, about 40 GB of accelerator memory, and output validation.

2026-09-05