Archive
推論 runtime、量化、 serving、KV cache、成本與效能。
AI 研究
The method retains all recurrent steps, using early exploration to narrow the scope of subsequent attention computation. Performance figures cover only attention workloads; the quality drop at 4K context length and the public reproduction materials still require verification.
推論系統
The implementation combines 92 layers in a single Pallas call, improving memory bandwidth utilization through cross-layer weight prefetching. The official comparison is limited to a specific topology and small batches, and speculative decoding performance also depends on the draft acceptance rate.
GitHub Repo
The release bundles Node.js and uv for both Intel and Apple Silicon, selecting the appropriate versions at startup. The desktop app remains an early preview, and full agent workflows and backend configurations require validation in actual deployment environments.
GitHub Repo
The new path reuses ggml Metal kernels to run quantized weights within the Transformers workflow. It currently requires installation from the main branch, with limitations in model coverage, batching, and kernel compatibility.
推論系統
Tests running DeepSeek-R1 on eight B200 GPUs show that confidential computing increases average per-token latency by less than 5%. The results depend on adjustments to memory transfers, kernel timing, and multi-GPU communication, and still require validation under actual serving workloads.
推論系統
A new article details inference testing on eight MI350X GPUs using data collected in July. The README at the specified commit differs from the article on workloads, while differences in engine configurations also limit performance comparisons.
AI 安全
The vulnerability involves the RPyC service on disaggregated prefill/decode nodes and requires an attacker to reach the control port. A public reproduction shows that health checks can still pass after exploitation; the advisory has yet to list a patched version.
推論系統
Physical AI Toolchain adds support for deploying inference on remote GPUs, with two robotic arm examples. Prior research shows that added latency and image bandwidth demands can offset computational gains, so deployments must assess the entire control pipeline.
AI 研究
A new benchmark evaluates coding agents on 53 SGLang tasks, finding that some patches that pass other checks still fail to serve models correctly. The results are limited to a specific test environment, and passing the benchmark does not mean the code is ready for production.
GitHub Repo
Hugging Face combines SIMD-based splitting, buffer reuse, and caching to reduce text encoding costs. The benchmarks cover specific tokenization workloads; Python call overhead and end-to-end service gains still require separate testing.
最新模型
The model offers DROID and SO-101 weights and fine-tuning workflows, with a success rate of about 42.9% on the public RoboLab leaderboard. Applications must still manage the control loop, and the planning horizon must not be mistaken for inference latency.
AI 研究
The study limits the scope of changes and uses evaluation variability and inference costs to screen modifications to agent frameworks. The public implementation enables replication, but the scale of token savings depends on the baseline.