推論系統
ReCache Splits Tool Schemas into Reusable KV Blocks, Reducing Allocation Memory by 92.43%
ReCache rewrites attention and positional encoding so that individual tools or skills can reuse their KV caches across requests. Qwen3 experiments show up to a 3.655× improvement in time to first token, but the method requires model fine-tuning and is not yet a drop-in inference plugin.
The combination of APIs, MCP tools, or skills available to a tool-using agent can vary with every request. Even when a schema remains completely unchanged, conventional prefix caching fails if the prefix order differs. The [ReCache paper](https://arxiv.org/abs/2608.19662) proposes encoding each resource independently: it masks attention between different schemas and restarts positional indices locally for each block, making the resulting KV representations independent of neighboring resources and their ordering.
The system reduces costs in two additional ways. Structural routing ranks layers or KV head groups by how much enabling each one individually reduces tool-call loss, allowing only the paths important to call decisions to access resources. Semantic pruning preserves tool names, parameter names, parameter descriptions, and final aggregation tokens instead of storing complete schemas. The authors combined seven datasets, including ToolACE, APIGEN, and WildToolBench, to train and evaluate Qwen3-1.7B and Qwen3-4B.
With resource-level attention alone, in-domain Inv-F1 reached 82.3%, nearly matching the dense baseline’s 82.4%, while TTFT improved by 3.655×. The full ReCache system achieved Inv-F1 scores of 80.3% on in-domain resources and 60.8% on unseen resources, reduced allocated KV tensor memory by 92.43%, and accelerated attention computation by 1.423×. For groups containing more than 10K resource tokens, the authors measured nearly constant prefill latency of approximately 5 ms and resource KV allocation of 0.03 GiB. The [public repository](https://github.com/EIT-NLP/ReCache) includes modified Qwen3 attention implementations, DeepSpeed training configurations, and evaluation code.
This is not a caching layer that can be transparently added to existing servers: the model must adapt to the new attention masks and positional layout, and the experiments focus primarily on Qwen3-4B. Engineers should next examine whether the approach scales to larger models with frozen weights, and whether blocking attention between resources remains safe when tools have cross-dependencies or schemas change frequently.