GitHub Repo
Dify Community Proposes a Knowledge Chunking Fix: Character-Level Splitting May Remove Chinese Text and List Line Breaks
On October 15, the Dify community reported that a fallback to character-by-character splitting in the knowledge base can remove single line breaks, joining rows that were originally separate. On October 16, a contributor reported reproducing the issue and proposed a fix, but the proposal has not been merged, and its impact on retrieval has yet to be measured.
Dify’s knowledge base chunking process is losing text structure. A report filed on October 15 says that in self-hosted version 1.17.1 and the main branch at the time, recursive chunking can remove all single line breaks when it falls back to character-by-character splitting. This can join adjacent rows of product IDs, URL lists, and some Chinese, Japanese, and Korean text. Another contributor commented on October 16 that they had reproduced the behavior in both chunkers. Issue report
The issue occurs when text exceeds the chunk size and none of the configured separators—such as double line breaks, periods, or spaces—is found. The chunker eventually uses an empty string as the separator and splits the content into individual characters; a later filtering step then drops the newline fragments. The report provides a Python reproduction that requires no model or database: join 20 SKU IDs with single line breaks, set the chunk size to 50 and overlap to zero, and the output loses the boundaries between rows. Reproduction steps
A fix proposed on October 16 in PR #44179 preserves line breaks in the empty-separator fallback path for both recursive chunkers, while keeping the existing behavior when splitting explicitly on line breaks. The author added a regression test using single-line-break input and reported that 107 tests passed and 4 were skipped in the relevant test suites. These are validation results reported by the proposal’s author; the change is still awaiting code review. Proposed fix
Technically, this means a RAG data quality issue can occur before indexing. Dify’s official documentation explains that text exceeding the maximum chunk size is forcibly split, and the retrieved results are then used as model context. From this, it follows that if row boundaries have already been lost, later prompt adjustments or switching models may not restore the structure. However, the public discussion has not provided quantitative comparisons of recall or answer accuracy. Chunking settings documentation
Engineering teams can start by checking the chunking preview with real Chinese documents and line-numbered data, then comparing the text before and after indexing. Next, they should track whether the fix is merged, which version includes it, and whether affected existing knowledge bases need to be reprocessed. This proposal should not yet be treated as an officially released fix.