資訊檢索／RAG
DEPT Keeps Document Vectors Fixed While Training Query Expansion End to End, Raising Qwen3-4B’s Average BEIR Score to 42.59
DEPT uses the same decoder to generate query expansions and encode documents, while a preservation loss prevents the existing vector index from drifting during fine-tuning. It outperformed staged baselines on five BEIR benchmarks, although the results are still based on single runs with two small models.
Released on August 18, DEPT addresses a practical conflict in using the same LLM for both query expansion and vector retrieval: although retrieval loss can improve the query side, it also shifts document vectors, invalidating the existing index and creating a continuously moving training target.
The researchers applied LoRA to Qwen3-4B-Instruct-2507 and LLaMA-3.2-3B-Instruct. The model first generates expansion text and then encodes the original query plus the expansion, using straight-through soft decoding to propagate retrieval gradients back to the generation logits. On the document side, a Document Embedding Preservation loss constrains new vectors to remain close to their initially cached values. Fixed PCA whitening standardizes the vector geometry, while the cached index is also used for online hard-negative mining. This allows query behavior to be updated while keeping the document index approximately stable.
Across five BEIR zero-shot retrieval benchmarks—SciFact, ArguAna, NFCorpus, FiQA, and SCIDOCS—the Qwen version achieved an average nDCG@10 of 42.59, compared with 41.09 for ExpandR. The LLaMA version scored 39.60, versus 38.17 for the same baseline. The shorter DEPT-K variant generates only about nine tokens but still reached average scores of 41.45 and 38.73, respectively. In ablations across three datasets, the full model averaged 50.82; removing the preservation loss reduced the score to 46.95, while removing whitening lowered it further to 42.09. On general generation benchmarks, the average score declined only slightly, from 77.56 for the base model to 76.28. By comparison, standard contrastive learning reduced it to 9.32, suggesting that the preservation mechanism also helps prevent the collapse of generation capabilities.
For RAG engineering, the main point is not a one- or two-point leaderboard gain, but the possibility of continuing to use an existing document index after fine-tuning the query strategy. However, each result currently comes from only one random seed, the training and evaluation scope is limited, and the codebase has only just been released, with no independent reproduction yet. The upfront costs of building the document cache, computing whitening statistics, and maintaining an online negative-sampling index must also be included when assessing deployment on large corpora.