Archive
推論 runtime、量化、 serving、KV cache、成本與效能。
推論系統
FreeBalance uses the previous layer’s residual state to predict expert load in the next layer, overlapping weight transfers with attention computation. The method does not alter the final routing decisions or model outputs, but the evidence so far covers only two MoE models, a single eight-GPU topology, and prefill workloads.
推論系統
DeepSeek has introduced time-of-use pricing for V4 Flash and Pro, with peak rates twice as high as off-peak rates. The greatest impact is not on ordinary input, but on cache-hit tokens heavily used by long conversations and coding agents.
最新模型
Oracle’s August 15 retirement date for Grok 3, Grok 4, and multiple Fast, Mini, and Code endpoints has passed. This is not a transparent model-alias switch: OCI users must change the model ID and revalidate inference parameters, image limits, and regional availability.
推論系統
The new release fixes two compatibility gaps in the model preprocessing layer: WebP images are now transcoded, and the Qwen renderer no longer requires system messages to appear first. Although the changes do not introduce new models or kernels, they can prevent multimodal and agent conversations from failing before they reach the inference core.
最新模型
Rather than comparing only local differences between adjacent denoising steps, GCache estimates the cumulative impact of cache errors on the final image or video. It uses dynamic programming to select cache refresh points, then applies Bayesian optimization to align theoretical weights with actual generation quality.
推論系統
FlashDrive reuses video KV, drafts reasoning tokens in parallel, and caches the less-changing steps in flow matching. The paper reports a 4.7× speedup on an RTX PRO 6000 with virtually unchanged trajectory error, but the results remain limited to simulation and a single model stack.
推論系統
The new release adds support for the Kimi K3 text model, hybrid KDA/MLA attention, latent MoE, and tool-call parsing. The converter directly reorders the MXFP4 bit layout, eliminating the need to first expand the entire 2.8T-parameter model to BF16.
最新模型
The previous LoRA loading path did not verify whether a tensor’s offset plus its size exceeded the GGUF file bounds, allowing corrupted or incompletely downloaded adapters to appear to load successfully. b10451 now fails immediately and confirms that the tensor index actually exists.
GitHub Repo
NInfer’s Ampere branch fits Qwen3.8-27B, MTP3, paged KV, and compatible prefix reuse onto a single 24GB RTX 3090. The author measured aggregate eight-request decoding throughput of 165.33 tokens/s, though the results are limited to a fixed workload with short prompts and long outputs.
推論系統
Community testing indicates that Apple Silicon inference frameworks still cannot reliably provide built-in MTP speculative decoding, hybrid-state prefix caching, and batched serving at the same time. The problem affects more than benchmarks: it can also force long-running coding agents to repay the prefill cost on every conversational turn.
最新模型
Google is rolling out Gemini 3.7 Flash across its API, Antigravity, Android Studio, and enterprise agent platforms, offering a limited-time 50% discount to capture high-volume agent workloads. Early integrations show that multimodal input, tool calling, and cross-API formats are already supported, but reasoning parameters and cost data still pose migration pitfalls.
生成式 AI 工具
ComfyUI 0.33.1 brings MiniMax Music 3 model loading and inference nodes into the core and adds CUDA Graph support for backend use. The release also fixes Music 3 failing to run with fixed VRAM configurations and issues detecting QKV layouts in some MiniMax weights.