LLM 推論／測試時計算
Prefix Sliding Preserves the Prompt Prefix and Recent Reasoning, Turning KV Memory for Ultra-Long Thinking into a Fixed Budget
During generation, Prefix Sliding evicts intermediate reasoning tokens while retaining the system prompt, task prefix, and the most recent several thousand tokens. It can be applied without retraining. The research reports that overall thinking time for long reasoning can be reduced by roughly threefold, but the current implementation depends on forked versions of vLLM and FlashAttention, with limited benefits for short outputs and tasks that require revisiting early details.
When reasoning models use full attention, generating each new token requires reading an ever-growing KV cache. The longer the chain of thought, the higher the per-step cost and memory requirement. Based on an attention analysis of Qwen3-1.7B, Prefix Sliding divides the context into two permanently visible regions: a prefix containing the system instructions, tool definitions, and original question, and a sliding window containing the most recent several thousand reasoning tokens. Older drafts between these two regions are evicted. With a 100-token prefix and a 4,096-token window, for example, attention processes at most 4,196 tokens, so generation cost no longer increases linearly with the entire reasoning trajectory.
The method retains continuously increasing position IDs, so KV entries that already have RoPE applied do not need to be recomputed as the window moves. The team also modified FlashAttention to compute attention only over the prefix and recent window, using two-stage filtering with intra-tile masking and whole-block skipping. In tests on a single 80GB H100, vLLM generated 1,024 sequences. After the window reached a steady state, throughput remained at approximately 5,000 tokens per second, while full attention continued to slow down. The roughly threefold speedup claimed in the paper is an end-to-end result showing that more reasoning tokens can be generated within the same amount of time; it does not mean that reasoning quality per token improves.
The researchers also applied this structure to reinforcement learning. A 100,000-token rollout does not need to be sent to the trainer in full. In the example, only the final 8,192 tokens are returned: the first 6,144 serve as context, and loss is calculated only over the final 2,048. In controlled tests on a 7B model, the authors report performance approaching that of full attention, but the gradients are a truncated approximation, and scalability to frontier-scale models has not yet been demonstrated.
The limitations are also significant from an engineering perspective. In LiveCodeBench, a model may resume writing earlier code after several thousand tokens; if the window is too small, it can lose the beginning of a function. Large volumes of tool output can also displace useful context almost instantly. The public code still uses older versions of Torch, vLLM, Prime-RL, and FlashAttention, and installing the custom kernels takes about ten hours. Engineering teams should first test window sizes, the proportion of long outputs, and the need to revisit earlier information against their own workloads before deciding whether integration into a production inference engine is worthwhile.