Dify Community Reports an Index Deletion Race That May Use Up RAG Retrieval Slots

On October 19, the Dify community reported a knowledge base indexing race: if a document is deleted while it is still being indexed, a background job may later write index entries that are no longer associated with the document. These entries can still match queries and use up top_k candidate slots, but are discarded when document chunks are loaded, leaving the application with fewer reference chunks. The reporter reproduced the issue in a self-hosted Dify 1.17.1 Docker environment. Issue report

According to the report, the cleanup process first retrieves the index IDs for the document’s chunks and deletes them. Because indexing has not finished at that point, there is nothing to delete. The background write completes later, and only then does the indexing process discover that the document is gone. The existing handler only logs a message and does not remove the entries it just wrote. The reporter deliberately paused keyword writes to control the timing and observed three orphaned entries; after re-uploading the same content, a query with top_k=3 returned only two chunks. The vector path was examined through code review and unit tests only; it has not been tested end to end with an embedding service. Reproduction steps and limitations

The patch proposed that same day has the indexing process retain the chunks passed to the write backend. If the write completes and the process then receives an exception indicating that the document has been deleted, it calls a new discard cleanup interface. For high-quality indexing, it removes text or parent-child document chunk vectors by ID. In economy mode, it removes keyword entries using the existing locking mechanism. The author also added race-condition tests covering the real indexing flow and cleanup timing. The proposal is still awaiting review; it does not cover vectors for multimodal attachments and will not reclaim orphaned data created before the change. Patch proposal

For RAG operations, this means that a reduced number of retrieved chunks may involve a consistency issue between the index and document data. An engineering inference: if the issue appears mainly when the same document is re-uploaded after deletion, it may be useful to compare the IDs of index matches with the chunks that can actually be loaded. Dify’s official documentation says retrieval testing and regular retrieval share an API endpoint, and the test page can be used to compare queries, but settings changed there apply only to that test. Follow-up should track the patch’s merge, a release containing it, and a cleanup plan for existing orphaned entries. The report does not establish that all vector backends are affected. Official retrieval testing documentation