推論系統
NVIDIA Transfers KV Caches Across Models With Linear Mapping, Cutting Model-Switching Prefill Time by Up to 25×
The new method maps a KV cache generated by a smaller model to a larger model in the same family, avoiding reprocessing the entire prompt when switching models during long conversations. Only four of six model pairs retained 73% to 98% of the original accuracy, showing that tensor compatibility does not guarantee semantic compatibility.
NVIDIA researchers have proposed cross-model KV cache transfer to eliminate the cost of rerunning prefill on the receiving model during cost-based routing, model upgrades, and dynamic switching in long conversations. Existing prefix caches can typically be reused only by the same model. This method instead creates a separate mapping for each target layer and head when the source and target models have the same number of KV heads and the same per-head dimension.
The researchers first select several source-model layers that best predict each target layer, concatenate their keys and values, and then solve for the transformation matrices using closed-form ridge regression. To prevent the mapping from applying only to positions seen during calibration, RoPE rotations are first removed from the keys. The transformation is performed in position-independent content space, after which the target model’s RoPE is reapplied. Calibration uses only 500 FineWeb-Edu sequences of 1,024 tokens each and requires no gradient-based training.
Across six small-to-large model pairings from Qwen3, Llama 3.1, and Ministral 3, four retained an average of 73% to 98% of the accuracy achieved when the target model performed its own prefill across five benchmarks. Mapping was 2.7× to 25× faster than rerunning prefill. The two weaker pairings retained as little as about 42%. A nonlinear MLP improved HellaSwag retention by as much as 37 percentage points for some failed pairings, but sacrificed the simplicity and training-free benefits of the linear approach.
The engineering constraints are substantial: the linear mapper itself still contains 1.01 billion to 3.36 billion parameters and occupies 4 to 12 GB, while calibrating each model pair takes roughly 47 to 87 minutes on eight H100 GPUs. The experiments also cover only models from the same family that share a tokenizer and have matching KV shapes. The next priorities are integrating the mappings into paged-cache runtimes such as vLLM and SGLang, and validating their real-world, end-to-end benefits under long-context, multi-tenant, and quantized-KV workloads.