推論系統
ReCo Coordinates KV Cache and Reasoning Length with Step-Level Rewards, Cutting End-to-End Latency by More Than Half
ReCo dynamically compresses the KV cache based on process rewards for each reasoning step, while suppressing repetitive reflection and enabling early stopping. In the authors’ experiments across three reasoning models and six benchmarks, generated tokens fell by 37% to 65%, while latency improved by 2.08× to 2.35×.
The cost of large reasoning models comes not only from an ever-growing KV cache, but also from the additional compensatory reasoning models generate after the cache is compressed. ReCo, submitted on August 5, therefore does not treat “how much cache to delete” as its sole objective. Instead, it uses a lightweight process reward estimator to score each completed reasoning step, then jointly controls the retained context and subsequent generation behavior.
The method comprises three coordinated components. First, reward-adaptive KV-cache compression reduces the cache more aggressively during high-reward steps, when the reasoning state is more stable, while retaining more information during low-reward steps, when the model may still need to revisit earlier context. Second, the system penalizes reflection-related tokens according to reward ranges, reducing restatement, backtracking, and unproductive continuation. Third, it stops early when the estimated confidence is sufficiently high. The authors’ key observation is that, given the same token-deletion budget, deleting tokens during high-reward steps preserves answer accuracy better than deleting them at random positions. They also find that compressing the cache without constraining generation can negate the savings by causing outputs to grow longer.
Across three reasoning models and six benchmarks, ReCo reduced generated tokens by 37% to 65% compared with full chain-of-thought reasoning. End-to-end latency fell to roughly half its original level, corresponding to a reported speedup of 2.08× to 2.35×, while accuracy was broadly maintained. The implication for inference services is that KV cache management and decoding strategies should not be governed by two unrelated sets of rules.
However, the paper is explicitly marked as still under revision, and its page does not provide a public implementation. The computational overhead of the process reward estimator itself, the risk of excessive compression caused by incorrect scores, and tail latency under continuous batching, quantized caches, and high-concurrency serving all still require independent validation. Engineering teams should wait for the code, per-model accuracy results, and hardware configuration to be released before determining whether the twofold speedup can carry over to production environments.