推論系統

推論 runtime、量化、 serving、KV cache、成本與效能。

The Optimization Stack for Running Hybrid Qwen Models on Apple Silicon Remains Incomplete: MLX Conversion Does Not Fully Preserve MTP

Community testing indicates that Apple Silicon inference frameworks still cannot reliably provide built-in MTP speculative decoding, hybrid-state prefix caching, and batched serving at the same time. The problem affects more than benchmarks: it can also force long-running coding agents to repay the prefill cost on every conversational turn.

2026-08-16

最新模型

Gemini 3.7 Flash Targets Coding Agents; LiteLLM Warns That the `minimal` Reasoning Level Returns HTTP 400

Google is rolling out Gemini 3.7 Flash across its API, Antigravity, Android Studio, and enterprise agent platforms, offering a limited-time 50% discount to capture high-volume agent workloads. Early integrations show that multimodal input, tool calling, and cross-API formats are already supported, but reasoning parameters and cost data still pose migration pitfalls.

2026-08-16

生成式 AI 工具

ComfyUI 0.33.1 Adds Native MiniMax Music 3 and CUDA Graph Support, Fixes Non-Dynamic VRAM Mode

ComfyUI 0.33.1 brings MiniMax Music 3 model loading and inference nodes into the core and adds CUDA Graph support for backend use. The release also fixes Music 3 failing to run with fixed VRAM configurations and issues detecting QKV layouts in some MiniMax weights.

2026-08-16

推論系統

vToken Virtualizes the KV Cache at Token Granularity, Up to Doubling Concurrency Under Memory Constraints

vToken adds a logical token table and asynchronous compaction on top of the existing PagedAttention design, allowing per-token eviction to actually release physical blocks. Its vLLM prototype improves SLA-constrained throughput by up to 1.37×, but the code has not yet been released, and the authors have not shown that its data-movement costs are suitable for all hardware and workloads.

2026-08-16

最新模型

UGC Data-Dependently Allocates Masked Diffusion Steps, Offering Near-Oracle KL Guarantees in Theory

New research uses “unmasking growth complexity” to characterize the information difficulty of discrete diffusion at different reveal stages, concentrating computation in high-curvature regions. The schedule can be estimated from samples and comes with a high-probability KL certificate, but it has yet to be validated on large-scale language or image models.

2026-08-15

最新模型

DreamX-Phi Constrains Dual-Arm Video Prediction with SE(3) Trajectories, Temporarily Ranks First on WorldArena 2.0

DreamX-Phi injects the rigid-body transformations of two robotic arms directly into attention and constrains video generation using depth, object masks, and V-JEPA features. It scored 60.65 on a WorldArena 2.0 snapshot, but its weights and inference code will not be released until the challenge concludes, and generalization to real robots remains unverified.

2026-08-15

GitHub Repo

MARC Breaks Clinical Reasoning Into a Traceable Agent Pipeline, but Its First Release Lacks Accuracy Comparisons

The open-source MARC v1 uses YAML to define extraction, reasoning, and output agents, with support for the Gemini API or local deployment through Ollama. It improves error localization and data control, but presents only three types of use cases and does not demonstrate that a multi-agent approach is more accurate than a single prompt.

2026-08-15

最新模型

SAEs Independently Select Their Highest-Activation Tokens, Inflating Cross-Dictionary Causal Variance From Near Zero to 11.9%

A new study finds that sparse autoencoder ablation evaluations typically allow each dictionary to select its own highest-activation token, meaning ostensibly identical latents are tested at different positions. After fixing a shared position, the 7.6% and 11.9% of variance previously attributed to dictionary differences in controlled experiments nearly disappeared.

2026-08-15

最新模型

LittleLearner Limits Pretraining Knowledge to a Fifth-Grade Level, With Few-Shot Prompting on Advanced Math Still Stuck at Around 6%

Researchers filtered 88 billion tokens of K–5 material from FineWeb‑Edu and trained 0.6B, 1.3B, and 5B models from scratch alongside compute-matched control models. Experiments show that scaling the models, applying SFT＋GRPO, and providing in-context examples primarily elicit existing capabilities rather than clearly pushing beyond the pretraining knowledge boundary.

2026-08-15

AI 安全

SPP Instills Value Reflection From the First Token, Widening the 3B Model’s OOD Risk Gap to 19 Points

Synthetic Persona Pretraining embeds first-person reflections generated from a “values constitution” into roughly 10% of pretraining documents, then uses post-training to bind that persona to the assistant identity. Newly released full experiments, weights, and data show that early intervention mainly improves value generalization, while resisting conventional jailbreaks may not require training from scratch.

2026-08-15

推論系統

RMM Prunes the Matrix Multiplication Contraction Axis Based on Input, Accelerating Llama 3.1 8B Long-Context Inference by Up to 1.40×

Reduced Matrix Multiplication leaves weights unchanged and, based on the column norms of the current activation, computes only the highest-scoring shared dimensions in a matrix multiplication. A100 tests show that gains increase with sequence length, but for short contexts, selection and kernel-launch overhead almost entirely offset the savings.

2026-08-15

AI 基礎設施

OpScale Breaks LLM Scaling Down to the Operator Level, Cutting Average GPU Use for Qwen2 Serving by 35%–50%

Instead of replicating an entire model, OpScale scales only the operators currently creating bottlenecks—such as attention and MLP—based on traffic and sequence length. The authors replayed 929,000 requests on A100 and GB200 clusters, using up to 36.3% fewer GPUs while meeting the same latency targets, but the prototype code has not yet been released.

2026-08-15