AI 推論系統
SemPIC Precompiles Documents into Reorderable KV Caches, Achieving Average F1 Close to Full Recomputation on Long-Context Tasks
SemPIC uses a separate Writer model to generate position-independent document KVs offline, while the Reader retains the original model and standard cache interface. Across three models and four tasks, average micro-F1 rises from 0.53 for existing methods to 0.60, but end-to-end latency, cache size, and code have not yet been disclosed.
Long-context RAG systems and agents often reread the same set of documents while changing the query, conversation history, and document order. A conventional prefix cache can only reuse an exactly matching prefix. If each document is independently computed into KVs and then concatenated in an arbitrary order, positional encoding and the absence of future context can shift the attention distribution.
SemPIC separates cache generation from cache use into a Writer and a Reader. The Writer is a model augmented with LoRA that learns through behavioral distillation to compile documents offline into per-layer KVs. The Reader retains the original pretrained decoder and requires no additional transformation layers on cache hits. Rather than merely approximating local KV values, the training objective makes the Reader reproduce the output behavior of full recomputation when using the precompiled states. The study also introduces KV Gradient Checkpointing, which reduces peak training memory while preserving gradients through cached tensors.
Across three models and four long-context tasks, SemPIC achieves an average micro-F1 of 0.60, outperforming KV Packet at 0.53 and coming within two points of full recomputation at 0.62. This suggests that reorderable document caches may be integrated into existing serving stacks without modifying the core online inference engine. The approach is particularly attractive for agents that repeatedly consult a fixed knowledge base.
However, the paper currently demonstrates mainly the feasibility of preserving output quality. Its abstract does not report end-to-end throughput at different cache hit rates, time to first token, offline compilation cost, or per-document storage overhead. The Writer may also need to be retrained for each model. Engineering teams should next look for a public implementation, cross-model generalization results, and the total cost of cache invalidation and knowledge-base updates rather than comparing F1 alone.