推論系統

推論 runtime、量化、 serving、KV cache、成本與效能。

Fast Gemma Verified Build Breaks 510 Tokens per Second on a Single A10G by Combining Sliding-Window Attention and MTP

VIDRAFT has published the highest- performing verified configuration from the Fast Gemma Challenge, pushing Gemma 4 E4B to 510.58 tokens/s on a single NVIDIA A10G while holding perplexity at 2.3930. The result comes from stacking multiple inference optimizations, but the test uses a single stream, fixed hardware, and a private prompt set, so it should not be equated directly with production throughput.

2026-08-03

AI 基礎設施

Etched Unveils Rack-Scale Inference Architecture, Challenging GPUs with Low-Voltage Compute and Cluster-Shared Memory

Etched’s first N4P inference chips have entered rack-level validation. The system uses low-voltage compute arrays and cluster-scale memory to address throughput and latency bottlenecks, respectively. The company claims sparse MoE workloads can sustain more than 80% of peak FLOPs, but has yet to publish reproducible model, power, or end-to-end performance data.

2026-08-03

代理工具

OSReward Evaluates 27 Computer-Agent Judges, With Hard Trajectories Exposing a Widespread “False Success” Bias

Using 1,019 human-annotated trajectories spanning the web, Windows, Ubuntu, and mobile platforms, OSReward tests whether vision-language models can correctly determine whether computer agents have completed their tasks. The team also released 100,000 reasoning-based judgment examples and trained self-hostable 9B and 35B OS-Shepherd reward models.

2026-08-02

最新模型

Cosmos 3 Edge Shrinks the World Model to 4B, Handling Vision, Text, and Robot Actions on a Single GPU

NVIDIA has released the Cosmos 3 Edge weights and inference pipeline, combining autoregressive reasoning and diffusion-based multimodal generation in a 4B Mixture-of-Transformers model. The model accepts text, images, and action trajectories, and a robot policy variant fine-tuned on DROID data is also available, though physical consistency and edge-device latency still require independent verification.

2026-07-23

推論系統

WIDE Dynamically Prunes Attention and FFNs for Every Token, Accelerating End-to-End LLM Decoding by 1.55×

WIDE brings dynamic pruning down to token-level attention-head groups and FFN channel groups, supporting both prefill and decode. At 50% sparsity, the study reports up to a 4.95× decoding speedup for core layers, but the end-to-end gain falls to 1.55×.

2026-07-31

最新模型

POCKET Compresses a 35B Sparse MoE to 8.2 GB, Runs on Phones and CPU-Only Systems With Standard llama.cpp

VIDRAFT has released the weights for its POCKET series, quantizing a 35B model down to as little as 8.2 GB while retaining roughly 3B active parameters per token. The models run directly with upstream llama.cpp, although the compression method remains undisclosed and current performance and quality evidence comes primarily from the development team.

2026-08-02

最新模型

openPangu 2.0 Pro Releases Open 505B MoE Weights, Supports a 512K Context with 18B Active Parameters

Huawei’s openPangu team has released a 505B mixture-of-experts model trained on Ascend NPUs, activating approximately 18B parameters per token. The model introduces hierarchical sparse attention and three-head multi-token prediction, though its performance figures currently come primarily from the team’s own evaluations.

2026-08-02

最新模型

ReToken Filters the Visual KV Cache With a Single Learnable Token, Improving Long-Video QA by 8.0 Points

Rather than feeding all image tokens into the model at once, ReToken retrieves sparse, question-relevant blocks from a prefilled visual KV cache. Its frozen-backbone approach trains only the retrieval token and fits on a single H100, although the results still come mainly from two 8B models.

2026-08-02

AI 基礎設施

SageMaker HyperPod Inference v3.1 Opens Up Custom Pods, Certificates, and Per-Pod Request Limits

AWS has updated the HyperPod Inference Operator, allowing teams to directly customize Kubernetes Pod specifications, configure custom TLS certificates, and limit concurrent requests per Pod. These interfaces address isolation and backpressure control for large-model serving, but teams should verify before upgrading that custom settings will not interfere with Operator-managed resources.

2026-08-02

基準與基礎設施

MLPerf Endpoints v0.7 Launches, Adding Continuous Submissions and a Comparable Results Pipeline for Agentic Inference

MLCommons has released Endpoints v0.7, transforming inference testing from fixed-batch submissions into endpoint benchmarking infrastructure with automated review and continuous updates. The upcoming multi-turn agentic workloads will measure both per-user progress and aggregate throughput, rather than counting only tokens/s for isolated requests.

2026-08-02

最新模型

PCD Separates Multimodal Failures into Perception and Reasoning Signals, Raising the 32B-to-8B Distillation Average to 61.22

Perception-Correction Distillation strengthens distillation over the perception segment only when the answer is incorrect and the student’s perception output diverges from the teacher’s. Both Qwen3-VL model-scaling experiments outperform uniform on-policy distillation, but the method depends on the teacher perceiving the input more accurately than the student.

2026-08-02

AI 安全

DFOT Quantifies LLM Over-Trust in Derived Sensor Values; Cross-Modal Distillation Improves Correction Rates by Up to 6.69 Points

A new framework breaks down “model over-trust in derived values” into metrics covering conflict acceptance, contextual misdirection, error correction, and utility harm. In a PPG-based rhythm interpretation case study, a reliability model that uses ECG during training but requires only PPG at deployment improves four endpoints, although the data and weights were not released with the code.

2026-08-01