推論系統
KVMem Pages Agent History to Memory and NVMe, Letting a 24GB Laptop GPU Address a Million-Token Workspace
Instead of compressing overflow agent history into summaries, KVMem preserves precomputed KV states and loads relevant blocks for each query. The authors demonstrate a logical million-token workspace using Qwen 27B on an RTX 5090 Laptop GPU, though model support and performance figures remain closely tied to their custom runtime.
Long-running coding agents accumulate conversations, tool outputs, and file contents. When that history exceeds the model’s context window or the GPU’s KV cache capacity, common approaches summarize older content or retrieve the original text when needed and run prefill again. KVMem proposes another approach: treating KV blocks whose attention computation has already been completed as virtual-memory pages, storing them across tiers of GPU memory, system memory, and NVMe. It then builds an attention-space index from the current query and moves only the relevant blocks, in chronological order, back into the limited execution window.
The paper evaluates the system on LongMemEval, MemoryAgentBench, AgentLongBench, and DeepSWE long-context tests. The authors report that, in the Qwen3.8-27B DeepSWE configuration, task success increased from 43.8% using compression alone to 48.4%. With NVFP4 weights, MTP, and a 24GB RTX 5090 Laptop GPU, the system can address a million-token workspace—four times the model’s native 256K context window—while generating about 50 tokens per second in a single session. Here, “one million tokens” refers to queryable logical history; it does not mean the model performs full attention over all tokens at once.
The publicly available QW3 runtime includes CUDA kernels, paged KV caching, continuous batching, MTP, and OpenAI-compatible and partially Anthropic-compatible endpoints. Deployers can configure separate budgets for history selection, generation headroom, and NVMe capacity. However, the current implementation primarily targets Qwen3.6, Qwen3.8 27B, and NVIDIA GPUs; generation on AMD GPUs, Metal, and CPUs is not yet supported. The built-in server also lacks authentication and TLS. Areas worth investigating next include portability across models, NVMe tail latency, throughput under multi-tenant batching, and whether query indexing can create a form of memory loss that is harder to detect than summarization when it misses critical evidence from earlier in the history.