Dify Community Discusses Context Compression and Cache Conflicts; Performance Data Not Reproduced in Dify

A performance investigation proposal appeared in the Dify repository on October 26, asking whether an agent’s context compression breaks prefix caching. The reporter said their agent removes an older archived checkpoint when creating a summary request, making the beginning of the prompt differ from previous requests and causing many tokens to be recomputed during prefill. The author later clarified that the measurements came from oMLX running on an Apple M3 Ultra with 256 GB of memory, not from Dify. Source: community proposal

There is a clear inference mechanism behind the concern. vLLM’s prefix caching reuses KV blocks for identical prompt prefixes; each block’s hash includes the previous block’s hash and the current block’s tokens. So even if later text is unchanged, changing earlier content can prevent subsequent blocks from being reused. This type of caching primarily saves input prefill computation; it does not directly reduce the decoding time for generating each new token. Source: vLLM design documentation, feature documentation

The discussion also helps clarify the data. The proposal initially reported a cache hit rate rising from 10.1% to 99.6%, along with a sharp drop in time to first token. The author later noted that statistics from different measurement levels should not be compared directly. In the latest continuous logs provided, cache hit rates for ordinary requests, summary requests, and the following turn were about 99.5%, 96.9%, and 99.6%, respectively. The summary request still had 4,989 uncached tokens and a time to first token of 17.94 seconds. These logs show high cache hit rates when the prefix is retained; by themselves, they do not establish the full cause of the original performance difference. Source: follow-up logs and discussion

Engineering teams can validate this by tracing the turn before compression, the summary request, and the first turn after compression. For each request, record cached tokens, uncached input, and time to first token, then compare the message prefix actually sent to the model. Retaining history can help cache reuse, but it also affects context capacity and the purpose of summarization, so it is not a universal fix. As of the time of review, the issue remained open with no related fix PR; the next step is to watch for a Dify-specific reproduction and regression tests. Source: issue status