推論系統

推論 runtime、量化、 serving、KV cache、成本與效能。

VibeVoice-ASR-BitNet Shrinks Speech Recognition to 1.58 GB With Heterogeneous Quantization, Enabling Real-Time Transcription on Three CPU Threads

Microsoft Research has released VibeVoice-ASR-BitNet, applying INT8 and ternary-weight quantization to the speech encoder and language model, respectively, so multilingual ASR can run without a GPU. Official tests show the model is 1.6× to 2.3× faster than Whisper.cpp at a similar size, though it currently supports only offline batch transcription.

2026-07-26

AI 基礎設施

Hetzner Trials an OpenAI-Compatible Inference API to Test European Hosting Demand With a Single FP8 MoE Model

Hetzner has quietly opened an experimental LLM inference service that currently offers only Qwen3.6-35B-A3B-FP8. Existing OpenAI SDKs can connect by changing the base URL. Low-load community tests recorded a median time to first token of 153 milliseconds and roughly 224 tokens/s, but the service has no SLA, billing, or production guarantees.

2026-07-26

推論系統

A Million Tokens Does Not Equal Usable Memory: Tests Identify a Small Number of Retrieval Heads as the Long-Context Bottleneck

An inference-time experiment on an unreleased 27B hybrid model found that it retrieved only 21.4% of injected information at roughly 1 million tokens, far below the 92.9% achieved at 33,000 tokens. Position scaling, attention temperature, and a higher-precision KV cache failed to significantly fix the problem, while retrieval capability was highly concentrated in 18 attention heads.

2026-07-26

AI 安全

QuantiBias Finds Quantized Models Can Pass Short-Form Safety Tests Yet Still Produce Stereotypes in About a Quarter of Open-Ended Responses

QuantiBias holds the model and prompts constant while varying only weight precision, measuring multilingual open-ended bias, refusals, and multiple-choice performance before and after quantization. The authors find that common short-form safety metrics remain broadly stable, yet an independent judge model still classifies about 24% to 27% of open-ended responses as endorsing stereotypes.

2026-07-26

推論系統

Diffusers Natively Integrates Nunchaku Lite, Using W4A4 to Reduce Diffusion Model VRAM Usage and Latency

Hugging Face Diffusers can now directly load Nunchaku Lite checkpoints to run diffusion Transformers with 4-bit weights and activations. Official single-machine tests showed up to roughly half the peak VRAM usage and a 1.8× speedup, but hardware coverage and image-quality validation remain limited.

2026-07-26

代理工具

Alibaba Open-Sources Open Code Review, Using a Deterministic Pipeline to Constrain LLM Review Scope and Comment Placement

Open Code Review delegates file filtering, rule matching, and comment placement to deterministic code, then lets LLM agents search repository context and identify defects. Alibaba’s internal benchmarks show token usage at roughly one-ninth that of Claude Code, but it trades lower recall for higher precision and cannot directly replace existing tests and static analysis.

2026-07-25

最新模型

SGLang-Omni Integrates MOSS-Transcribe-Diarize, Bringing Multi-Speaker Transcription to Continuous Batching

SGLang-Omni has added a deployment path for MOSS-Transcribe-Diarize that returns segment-level timestamps, text, and speaker labels in a single response through an OpenAI-compatible endpoint. This makes the open 0.9B model easier to deploy in online services, although the three evaluation datasets used for the official performance results are not currently fully public.

2026-07-25

最新模型

Solar Open 2 Extends a 250B MoE’s Context to 1 Million Tokens With Hybrid Linear Attention

Upstage has released the weights for Solar Open 2, which uses 250B total parameters and activates 15B parameters per token for long-horizon agentic workflows. The model retains softmax attention and a KV cache in only one-quarter of its layers, but its million-token capability and Korean office-agent results are still based largely on the team’s own evaluations.

2026-07-25

AI 基礎設施

NVIDIA ModelExpress Transfers Weights GPU-to-GPU, Cutting Scale-Out Time for an 806 GiB Model to Under Two Minutes

NVIDIA has detailed ModelExpress’s complete transfer path, allowing new inference replicas to copy weights and compilation caches from existing GPUs, with P2P RDMA as the preferred method. In an eight-B200 test, the company cut DeepSeek-V4-Pro’s time to API readiness from eight minutes to 1 minute 44 seconds, though the result depends heavily on identical GPU models, high-speed networking, and compatible tensor configurations.

2026-07-25

最新模型

Apertus 1.5 Adds Native Image and Audio Understanding, With 8B and 70B Open Models Supporting 262K Tokens

The Swiss AI Initiative has released Apertus 1.5, using continued pretraining to expand its original text models into multimodal models that produce text output. The weights and training information are public, but the formal technical report, comprehensive benchmarks, and upstream inference framework support are still forthcoming.

2026-07-25

最新模型

ReMo Reallocates Visual Information Across Audio and Text, Cutting Omni-LLM Input Tokens by 54%

A KAIST team has proposed ReMo, a training-free method that removes visual tokens at the input stage when their content can be explained by audio, while preserving object semantics through text proxies. In the authors’ own tests, it compressed inputs to the Qwen Omni series by more than half without reducing average scores, but neither the code nor independent reproduction results have been released.

2026-07-25

最新模型

The Stack v3 Opens Nearly 5 Trillion Code Tokens, Fixes Deduplication Error in Previous Version, and Embeds Source Code

Hugging Face and BigCode have released a new generation of open code pretraining data containing approximately 173 million training samples and 4.9 trillion tokens. The new version no longer requires users to retrieve the source code separately, but training teams must still manage licensing, data contamination, and the 15.9 TB storage cost themselves.

2026-07-25