推論系統
NVIDIA Transfers KV Caches Across Models With Linear Mapping, Cutting Model-Switching Prefill Time by Up to 25×
The new method maps a KV cache generated by a smaller model to a larger model in the same family, avoiding reprocessing the entire prompt when switching models during long conversations. Only four of six model pairs retained 73% to 98% of the original accuracy, showing that tensor compatibility does not guarantee semantic compatibility.