推論系統
vToken Reclaims KV Cache Through Token-Level Virtualization, Boosting Throughput by Up to 37% on a Single H100
vToken adds a token-to-physical-location mapping layer on top of PagedAttention blocks, compacting scattered KV gaps left after eviction. Its vLLM prototype reduces retained blocks by up to 72.3%, though testing remains limited to a single machine with one GPU and primarily 7B- and 8B-class models.
LLM serving systems typically use PagedAttention to divide the KV cache into fixed-size blocks, avoiding allocator-level fragmentation. Methods such as H2O and Scissorhands, however, decide whether to retain or evict individual tokens. As long as a block contains even one valid token, its remaining empty slots cannot be returned to the resource pool. Preliminary vToken testing showed that after token-level eviction was applied to a 16K context, wasted space within blocks could reach 40%–60%.
vToken inserts a virtualization layer between the eviction policy and physical blocks. A per-request TokenTable records each logical token, its liveness state, and its current block and offset. The policy merely marks tokens as invalid; when memory pressure rises, the runtime moves surviving KV entries from sparsely utilized blocks to new locations. Copies are scheduled on a separate CUDA stream after the forward pass completes, and the next attention iteration waits on a CUDA event. This avoids modifying the PagedAttention kernel or recapturing the CUDA Graph. Mapping metadata for a 16K sequence occupies approximately 256 KB.
The authors evaluated Mistral-7B and Llama-3.1-8B, along with an additional Qwen2.5-14B capacity case, using vLLM 0.18.0, PyTorch 2.10, and a single 80 GB H100. Compared with Naive-Evict, which applies the same eviction decisions without compacting blocks, vToken reduced the number of retained blocks by 27.2%–72.3%. For Mistral-7B, SLA-constrained throughput improved by 9.9%–37.3%, while the maximum feasible concurrency under a constrained KV budget increased by as much as 2×. The integration code required for a new eviction policy also fell from more than 500 lines to fewer than 50.
This does not provide model capacity for free: data movement still competes with decoding for GPU resources, while any quality loss caused by token eviction remains the responsibility of the higher-level policy. The prototype covers only the single-node, single-GPU fast path and conservatively avoids moving shared-prefix blocks. The next question is whether compaction gains can offset synchronization and bandwidth costs under multi-GPU tensor parallelism, cross-device KV migration, and mixed workloads.