WISE reuses attention regions in recurrent models, with a measured 1.36× speedup in the attention pipeline

University of Virginia researchers unveiled WISE on September 23, aiming to reduce the cost of repeatedly computing global attention in recurrent language models. The method requires no retraining: it uses early computation to identify important context blocks, then reuses the selected regions in later steps while the model continues updating its internal representations. [Paper abstract](https://arxiv.org/abs/2609.27373)

The public implementation defaults to 32 recurrent steps, retaining global attention for the first 12. The system divides the context into blocks of 32 tokens. At each of steps 9 through 12, it selects blocks whose cumulative attention weight reaches 95%, then merges those selections into the set accessible in subsequent steps. From step 13 onward, only the set of blocks is fixed; Q, K, V, and attention weights within those blocks are still recomputed, preserving the ability to refine representations in later steps. [Repository overview](https://github.com/tbn5pj/WISE_code)

The engineering value of this design lies in turning irregular attention sparsity into a work schedule the GPU can execute. The authors provide Triton kernels that skip unselected key blocks and fuse score computation, causal masking, softmax, and value aggregation, avoiding an approach that merely applies a mask while still scanning every block. [Implementation documentation](https://github.com/tbn5pj/WISE_code/blob/main/docs/REPRODUCING.md)

The authors benchmarked 30 attention workloads from the Huginn model on an RTX A6000. At a 4K context length, the final 20 steps achieved approximately a 1.76× speedup over native FlashAttention. Including the first 12 exploration steps, the full 32-step attention pipeline achieved approximately a 1.36× speedup. These figures include the corresponding schedule construction costs but do not cover inference for the entire model. [Benchmark table](https://arxiv.org/html/2609.27373v1#S7.SS4)

The quality cost cannot be overlooked. The paper’s paired tests detected no decline in mean F1 at context lengths up to 2K, but found a drop of approximately 3.3 points at 4K. The setting with the highest speedup therefore also requires scrutiny of the quality tradeoff; the results cannot be generalized to mean that performance is preserved across all long-context tasks. [Quality and efficiency evaluation](https://arxiv.org/html/2609.27373v1#S7.SS5)

The paper also describes the context quality evaluation as involving 100 examples, while the public reproduction documentation lists 30 examples per context length, totaling 120 prompts. How these samples correspond still needs clarification. Replotting saved result tables is also no substitute for rerunning the original evaluations. [Reproduction documentation](https://github.com/tbn5pj/WISE_code/blob/main/docs/REPRODUCING.md)

The reproduction materials have further gaps: the repository provides kernels, configurations, dataset identifiers, and prompt hashes, but does not include the complete benchmark tensors. The documentation also states that the large-model evaluations and GPU benchmarks were not rerun when the public release was packaged. For those deploying the method, the next step should be to obtain verifiable test inputs, remeasure end-to-end latency in the same hardware and software environment, and separately validate quality for Chinese, multi-turn conversations, and longer contexts. [Public release verification notes](https://github.com/tbn5pj/WISE_code/blob/main/PAPER_CODE_CONSISTENCY.md)