Open WebUI Community Proposes Cache Compression Improvement; Long-Conversation Summaries May Recompute Over 100,000 Tokens

On November 1, the Open WebUI community proposed an improvement to conversation compression, saying the summarization process may reprocess long histories that are already cached. The author reported results using Apple Silicon, oMLX, and Qwen 27B: for a conversation of about 117,000 tokens, a cached request normally began responding in about four seconds, while a compression request was still unfinished after six minutes. These are, respectively, time to first response and an elapsed wait for a request that had not completed; they cannot be directly converted into a speedup multiplier. Original proposal

The author points to _generate_summary as the cause: the process flattens the conversation history into a single user message, then applies a summarization prompt. Even if the text remains similar, the token prefix sent to the model has changed. Prefix caching relies on a shared prefix to reuse KV state; the official vLLM documentation also explains that this mechanism saves input prefill computation and does not directly shorten the time needed to generate new tokens. This provides technical context, but is not independent verification of the oMLX case. Explanation of the caching mechanism

The proposal recommends preserving the original system prompt, tools, and message ordering, and appending the summarization instruction at the end. The author says that after rewriting the request through a local proxy, 116,736 of 117,316 input tokens were cache hits, or about 99.5%, and the summary completed in 99.6 seconds. This remains a measurement from one author in a specific environment; the public issue does not include the complete proxy implementation, nor does it show that all inference services would achieve the same result. Measurements and proposed approach

Another limitation concerns reasoning output: on November 2, another user asked for a way to disable thinking specifically for compression requests, reporting that reasoning text could end up in the summary and that the extra generation prolonged the wait. For local deployments, switching to a dedicated summarization model could also increase memory use or add model-switching costs. Follow-up request

As of this review, both issues remain open and should not be treated as a released performance improvement. Engineering teams should track cache-hit counts for summarization requests, prefill time, token allocation between reasoning and the actual summary, and whether summaries retain task constraints. Measuring ordinary chat speed alone may miss the cost of long conversations entering the compression stage.