Q2D-Web Tests Agentic RAG on 190 Million Documents, Finds Supporting Queries a Retrieval Weakness

The Perplexity research team released [Q2D-Web](https://arxiv.org/abs/2609.08887) on September 8 in an effort to close the gap between existing retrieval benchmarks and production traffic from agentic RAG systems. The data comes from nine months of real-world queries filtered to remove personally identifiable information. It includes 12,365 primary queries that restate the main intent and 57,356 supporting queries generated by agents to gather background information, switch languages, or investigate entities. In total, the dataset contains 69,721 queries across ten languages. The researchers then combined the top 5,000 results for each query and deduplicated them using MinHash-LSH, producing a candidate corpus of approximately 190 million documents.

The evaluation focuses not on the final ten results shown to users, but on the first stage of a multistage search pipeline. If relevant evidence does not appear within the top 1,000 results, downstream rerankers and answer models never have an opportunity to use it. Q2D-Web therefore uses Recall@1000 as its primary metric, while also reporting Recall@100 and nDCG@10. It provides three sets of relevance labels based on citation logs, production rankings, and a combined set augmented with LLM judgments. The relative ordering of 13 BM25, dense embedding, and late-interaction retrievers remains broadly consistent across label sources, but changes substantially by language, domain, and query type.

Under the combined labels, pplx-embed-v1-4b achieves the highest Recall@1000, while Nemotron-3-Embed-8B leads on Recall@100 and nDCG@10. This shows that “retrieving more evidence” and “ranking evidence closer to the top” are distinct optimization objectives. Every neural retriever performs markedly worse on supporting queries than on primary queries. BM25 has the lowest overall recall, yet retrieves the largest number of unique relevant documents missed entirely by the other models. For RAG engineering, these findings support hybrid retrieval and demonstrate that embedding models should not be evaluated solely on human-written search queries.

A complete run of a 4B model requires approximately 4,608 H200 GPU-hours. Using reciprocal rank fusion across multiple retrievers, the researchers selected 31.7% of the corpus while preserving the system ordering observed on the full corpus. However, this subset inflates Recall@1000 by an average of 5.1 percentage points. It is therefore suitable for rapid screening but cannot replace final validation on the complete corpus. In addition, the dataset is not publicly released to prevent training contamination. Instead, open-weight models can be evaluated only through a [hosted leaderboard](https://huggingface.co/spaces/perplexity-ai/q2d-web-leaderboard). Documents are encoded using only their first 512 tokens, and the labels combine Perplexity’s proprietary ranking signals with LLM judges, making comprehensive auditing or offline reproduction difficult for external teams.