推論系統與評測
KVDiagnosis Traces KV-Cache Compression Failures Sample by Sample, Releases 12,520 Correct-to-Wrong Cases
Rather than merely comparing aggregate long-context scores, KVDiagnosis creates a paired FullCache baseline for every compression configuration to track evidence retention, attention, and probability drift. An implementation audit also led the team to exclude 7,800 PyramidKV records with identical results, highlighting how adapter errors can contaminate cross-method evaluations.
A team at King Abdullah University of Science and Technology has released KVDiagnosis, a sample-level diagnostic benchmark for KV-cache compression in long-context inference. Conventional evaluations report only compression ratios, memory usage, and aggregate task scores, offering no explanation for why a sample that was originally answered correctly becomes incorrect. More problematically, the “50%” claimed by different methods may refer to retaining half the tokens, pruning channels, reallocating per-layer budgets, or reducing numerical precision—not equivalent byte counts.
KVDiagnosis first catalogs 25 methods across five mechanism categories and empirically validates eight implementations. Each data source is first run with FullCache, after which every method is evaluated at 75%, 50%, and 25% settings using the same model, prompt, tokenizer, decoder, and scorer. Only after the complete matrix is frozen does the benchmark extract C→W records—cases answered correctly with FullCache but incorrectly after compression. This prevents one compressor’s failure set from being used to test other methods and explicitly preserves unsupported, failed, and inapplicable diagnostic runs instead of silently removing them from the denominator.
The public dataset uses a pinned version of Qwen3-8B with deterministic decoding to evaluate RULER-8K, RULER-16K, Qasper, and HotpotQA. It contains 2,600 FullCache controls, 59,800 supported compression runs, and 12,520 C→W records. Of the failures, 63.2% have only low or partial evidence coverage. Across ten diagnostics, stratified AUROC values for distinguishing failures from successful cases range from 0.684 to 0.871. For 96 reproducible low-evidence-attention failures, quadrupling attention to the evidence repairs 29.2%, while a count-matched sham intervention repairs only 6.3%, providing limited but causally suggestive validation.
Notably, the authors excluded PyramidKV following an audit: a legacy adapter mistakenly used the same fixed-budget code path as SnapKV, causing all 7,800 paired runs to have identical retained positions, outputs, scores, and gold NLL. The repository now publishes the data, hash manifests, validation scripts, and audit logs. However, the results remain concentrated on a single 8B model, and the configuration ratios do not represent equal-bit memory comparisons across mechanisms. Engineering teams therefore should not treat the benchmark as a direct ranking of compressor speed or serving cost.