All Articles · SURL AI News

Transformers integrates ggml quantization kernels for direct inference with compressed GGUF weights on Apple Silicon

Hugging Face has announced GGUF inference integration on the main branch, bringing compressed weights into existing Python and PyTorch workflows. Initial support focuses on Apple Silicon and Qwen3.5 architectures, though the benchmarks use different measurement methods.

2026-09-22

AI 研究

Shared Inference Cost Study Publishes H100 Results, Showing How Metering Changes Tenants’ Cost Shares

unalloc consolidates cluster and model bills. The study found that different metering methods shifted the same tenant’s share of costs by roughly 12–14 percentage points. Hardware validation consisted of one short run per workload, so the results do not establish a definitive allocation.

2026-09-22

推論系統

NVIDIA publishes multi-GPU video inference benchmarks, cutting generation latency to about 34 seconds on eight GPUs

NVIDIA ran Cosmos 3 Nano using Dynamo-Triton and TensorRT, reducing generation latency from 156.6 seconds on one GPU to 34.2 seconds on eight. The tests did not cover concurrent throughput or per-video costs, and quality validation used only sampled frames.

2026-09-22

硬體

Googlebook opens for preorders, integrating agent development tools and an isolated Linux environment

Google is bringing Antigravity, a Linux terminal, and on-device AI hardware to its new laptop platform. The pKVM protection modes across processor variants and NPU access from Linux still require product documentation and testing on actual devices.

2026-09-22

GitHub Repo

vLLM Community Proposes RDNA Tiling Optimization, Measuring About 17%–19% Higher Embedding Throughput on W7900

Community tests improved embedding-serving performance on AMD GPUs by reducing the key-value block size in an attention kernel. The change remains a proposal, and kernel speedups, overall throughput, and retrieval quality must be evaluated separately.

2026-09-22

GitHub Repo

Ultralytics 8.4.157 adds CUDA Graph replay, cutting YOLO26n FP16 test latency by about 20%

The release uses real-image calibration and CUDA Graph replay to reduce inference overhead. Gains vary with precision and batch size, while the SiLU rewrite mentioned in the release notes was withdrawn during final review.

2026-09-22

GitHub Repo

tokenizers v1 overhauls CPU encoding, but release candidate still has Python feature gaps

Hugging Face reports Rust benchmarks showing encoding speeds 3 to 30 times those of v0.23 on a single thread on an M4 Max. Gains depend on the model and corpus, and existing Python workflows still need feature compatibility checks.

2026-09-22

Grok 4.7 Strengthens Long-Running Coding Tasks, API Automatically Returns Encrypted Reasoning State

The update adds training on long-running tasks and makes the Responses API automatically carry reasoning state across turns. Official evaluations use different reasoning effort levels, while the API and Cursor have separate billing thresholds for long contexts that need to be checked individually.

2026-09-22

GitHub Repo

Transformers cache offloading reportedly changes Qwen3.8 outputs on first reuse

A report with a minimal reproducer shows that a hybrid-attention model’s output can diverge on the next step even after prefill completes normally. Maintainers have yet to confirm the issue, underscoring the need to verify numerical consistency across multiple steps when optimizing GPU memory usage.

2026-09-20

AI 基礎設施

AgentCore Runtime V2 adds memory reclamation and snapshot-based startup, with cold starts of about two seconds in AWS tests

AWS has updated its agent runtime to reduce memory usage during long sessions and shorten cold starts. The performance figures come from no-op agent tests; cost savings and snapshot restore behavior still need to be validated against real workloads.

2026-09-20

推論系統

SemIf demonstrates in-browser option-score readout, but decision probabilities still need separate calibration

This local inference experiment compares reading option logits directly with generating JSON token by token, letting developers measure the cost of output formats. Results remain sensitive to quantization, browser behavior, and test order, and option probabilities cannot be treated directly as measures of reliability.

2026-09-20

推論系統

AMD details MLPerf inference optimizations, boosting eight-GPU GPT-OSS serving throughput by about 38%

AMD improved GPT-OSS-120B benchmark throughput on MI355X GPUs by splitting attention kernels and adjusting scheduling. Public weights and reproduction instructions provide a starting point, but aspects of the environment configuration and model card still need clarification.

2026-09-19