推論系統
HeadWiseKV Configures Long-Context Caching per Attention Head, Extending Qwen3.6-27B’s Usable Length to 161K
HeadWiseKV stops retaining the same amount of history for every attention head. Instead, it uses offline calibration to assign multilevel windows to individual heads and actually reduces physical KV residency. In fixed-model tests, the paper extends the maximum successfully validated context from 114K to 161K, although the higher figures reported by the public repository rely on specific quantization settings and a single RTX 4090D.
Even when long-context models combine local attention with recurrent or linear pathways, a small number of global-attention layers can still cause the KV cache to grow with sequence length, ultimately making GPU memory—rather than compute—the limiting factor for the context length that can be served. HeadWiseKV, released on September 2, shifts the configuration granularity from the entire model or layer down to each physical KV head: heads that depend on long-range information retain more history, while other heads use shorter sliding windows, with residency allocated within an overall cache budget.
Its core calibrator, SeqCalib, formulates configuration as a constrained rate-distortion optimization problem and makes layer-by-layer decisions in actual execution order. When calibrating higher layers, it carries forward the cache policies that lower layers will use in deployment, capturing cross-layer error instead of evaluating each layer independently. At runtime, grouped-cache directly allocates KV storage for the selected heads rather than retaining a full cache and merely applying a mask, so the method saves actual device memory. It requires no model retraining and does not modify the model’s existing local, recurrent, or linear-state pathways.
The paper evaluates quality on four hybrid long-context models and conducts system experiments with Qwen3.6-27B. At a 112K context length, sampled peak device memory falls by 8.59%, while the maximum successfully validated length rises from 114K to 161K, with RULER and LoCoMo performance remaining close to full-KV caching. A public llama.cpp branch separately reports that, on a 24GB RTX 4090D, UD-Q4_K_XL can reach the model’s 262,144-token limit; Q6_K with a q8_0 KV cache reaches 159,744 tokens, compared with 53,248 for the unmodified version under the same conditions.
The latter figures should not be conflated with the paper’s results: they depend on specific GGUF quantization, CUDA full-GPU offload, and configuration settings. The repository also lacks some original predictions, complete capacity-probing logs, GGUF hashes, and driver versions, and it does not perform statistical significance testing. The next priority is to incorporate calibration cost, quality drift across different prompt distributions, and the benefits for pure Transformers, MoE models, and multi-user batched serving into reproducible evaluations.