AI 推論系統
vToken Virtualizes the KV Cache at Token Granularity, Up to Doubling Concurrency Under Memory Constraints
vToken adds a logical token table and asynchronous compaction on top of the existing PagedAttention design, allowing per-token eviction to actually release physical blocks. Its vLLM prototype improves SLA-constrained throughput by up to 1.37×, but the code has not yet been released, and the authors have not shown that its data-movement costs are suitable for all hardware and workloads.
Long-context inference often uses per-token policies to evict unimportant KV entries, but vLLM’s PagedAttention allocates memory in fixed-size blocks. As long as a block contains even one valid token, the entire block cannot be reclaimed. The [vToken paper](https://arxiv.org/abs/2608.13263) reports that, with Llama 3.1 8B and a 16K context, strategies such as H2O can leave 40% to 60% of the space within blocks unused, showing that “logical deletion” does not make GPU capacity truly reusable.
vToken inserts a token-level virtualization layer between the eviction policy and the block manager. Each request maintains logical token IDs, physical block/offset locations, and liveness bits. The policy only marks tokens as invalid; once fragmentation crosses a threshold, a reclaimer asynchronously moves the surviving KV entries into more densely packed destination blocks. Data movement is scheduled after the forward pass, with CUDA events ensuring that the next attention operation reads from the new locations. This avoids rewriting the attention kernel while preserving CUDA Graph compatibility.
The authors tested H2O, Random, and Scissorhands on vLLM 0.18.0 and PyTorch 2.10. Compared with Naive-Evict, which waits until an entire block is empty, vToken reduces the number of KV blocks retained per request by 27.2% to 72.3% and improves SLA-constrained throughput by up to 1.37×. Under a fixed active-KV budget, the maximum feasible concurrency is up to twice as high. Integrating a new eviction policy also requires fewer than 50 lines of changes, down from more than 500. The current [vLLM mainline implementation](https://github.com/vllm-project/vllm/blob/main/vllm/v1/worker/block_table.py) still centers on a block table and token-to-slot mapping, indicating that vToken is targeting a real runtime boundary.
The next engineering questions are whether the code will be released, whether it can be upstreamed, and whether it still delivers net gains in multi-tenant environments, with prefix caching, across different block sizes, with FP8 KV caches, and in deployments that disaggregate prefill and decode. The paper’s figures come from the authors’ prototype. Compaction requires temporary destination blocks, so it cannot start when memory is already completely exhausted, and evicting tokens may still degrade model quality.