推論系統

推論 runtime、量化、 serving、KV cache、成本與效能。

TIDE Corrects Teacher–Student Mismatch, Raising Qwen3 Reasoning Avg@8 from 6.9% to 20.3%

TIDE no longer treats high token-level agreement as direct evidence of successful distillation. Instead, it separately addresses student overgeneration and the omission of teacher-preferred tokens. Under the severe-mismatch setting, the method also reduces the average response length by 3.6×, although reproducing the full training run still requires eight GPUs and the authors have not released a checkpoint.

2026-08-11

最新模型

ICBQ Revisits Quantized Block Boundaries, Cutting Qwen3-8B Ternary-Model Perplexity From 5752.9 to 29.9

Rather than replacing the quantizer, ICBQ jointly recalibrates the boundaries between adjacent Transformer blocks a second time, reducing the accumulation of early errors with depth. The quality improvement comes at the cost of a 21% average increase in quantization time, and the authors have not yet released an implementation.

2026-08-11

最新模型

Matryoshka Embeds Three Models in a Single Set of Weights, Cutting Training Compute by 36%

Matryoshka Language Model Suites links 500M, 1.5B, and 3B submodels in a nested architecture that allows each model to be extracted and run independently, avoiding duplicate training for every size. Sharing early layers and the KV cache also increases speculative decoding throughput by 14% to 26%, though the approach has so far been validated only on base models up to 3B parameters.

2026-08-11

推論系統

Tevatron-Elastic Tunes Depth, Tokens, and Vector Width with a Single Retrieval Checkpoint

Tevatron-Elastic unifies early exit, Matryoshka vectors, and inter-layer token compression into configurable operating points, allowing one checkpoint to serve multiple cost profiles. In the authors’ test environment, the Qwen3‑0.6B reranker retained full quality at layer 16 while achieving a 1.75× speedup.

2026-08-11

推論系統

KVDiagnosis Traces KV-Cache Compression Failures Sample by Sample, Releases 12,520 Correct-to-Wrong Cases

Rather than merely comparing aggregate long-context scores, KVDiagnosis creates a paired FullCache baseline for every compression configuration to track evidence retention, attention, and probability drift. An implementation audit also led the team to exclude 7,800 PyramidKV records with identical results, highlighting how adapter errors can contaminate cross-method evaluations.

2026-08-11

推論系統

Laguna S 2.1 INT4 With DFlash: Four RTX 3090s Tested at 200K Context and a Peak 282.5 tokens/s

A publicly available, reproducible community vLLM recipe deploys Laguna S 2.1 INT4 on four 24GB RTX 3090 GPUs and handles a real 190K-token request using a precision-matched DFlash draft model. The reported 282.5 tokens/s is a peak under a specific configuration, not sustained throughput, while the quality and compatibility of low-bit quantization remain highly dependent on the checkpoint and inference engine.

2026-08-11

最新模型

PTQ4SNN 同時量化權重與膜電位，事件分類僅損失 1 個百分點

PTQ4SNN 把脈衝神經網路長期保留的浮點膜電位納入後訓練量化，按通道配置 2、4 或 8 位元。論文在分類與語意分割上接近浮點模型，但硬體收益仍來自分析模型，且專用程式尚未公開。

2026-08-10

RAG／文件理解

DocMemo Re-queries Long Documents with Bayesian Page Memory, Reaching 71.3% Accuracy on MMLongBench

DocMemo does not treat the top-k pages from the initial retrieval as final. Instead, it feeds reasoning feedback back into the page-relevance distribution and resamples. Its three-layer memory improves long-document question answering, but evaluation relies on GPT-4.1, and the public repository remains quite minimal.

2026-08-10

推論系統

CoBa Dynamically Allocates Inference Compute, Matching Best-of-16 With 58.9% Fewer Weighted Tokens

CoBa initially generates a small number of candidate answers, then uses disagreement and verification scores to decide whether to sample additional candidates or activate a stronger verifier. It achieved 85.13% macro-average accuracy across five reasoning benchmarks, although the results came from offline replay of pre-generated candidate pools rather than testing in a live service.

2026-08-10

最新模型

Meta Open-Sources Muse Glimmer 30B: 17GB Quantized Version with DFlash Hits 233 tokens/s on RTX 5090

Muse Glimmer brings multimodal input, tool calling, and failure retries to a 30B dense model deployable on consumer hardware. Apache 2.0 weights and official GGUF files are now available, although the performance and safety figures still come primarily from Meta’s own evaluations.

2026-08-10

代理工具

A²E Unifies 23 Benchmarks and 9 Agent Harnesses, but Each Evaluation Cell Still Contains Only 5 Tasks

The Shanghai AI Laboratory has released A²E, which uses the Agent Task Protocol to decouple benchmarks, agent harnesses, and trajectory scoring. Its 1,035 paired runs show that the harness affects tool use and token costs, but the sample size is too small to support framework rankings.

2026-08-10

代理工具

Shepherd v0.3.0 Retains Agent Changes as Reviewable Proposals, but the Paper’s Automated Forking Supervision Is Not Yet Fully Delivered

Shepherd records model calls, tool operations, and file changes as persistent execution traces, allowing users to inspect or discard modifications before they enter the workspace. The latest documentation also explicitly states that the meta-agent delegation, in-flight forking, and automated retry interfaces demonstrated in the paper and on the project homepage are not yet directly usable features in v0.3.0.

2026-08-10