AI 基礎設施與評測
MLPerf Measures the Full RAG Pipeline for the First Time, but Performance Mode Fixes the Retrieval Path
MLCommons has introduced an end-to-end RAG inference benchmark that separately measures vector database construction and multi-hop question-answering throughput. To control uncertainty in generation, performance tests replay prerecorded inputs for each stage, so they do not fully represent the dynamic behavior of online RAG systems.
MLCommons unveiled the first MLPerf end-to-end RAG inference benchmark on August 26, expanding beyond measurements centered on a single model and a single prompt to cover data ingestion, chunking, embedding, vector indexing, retrieval, reranking, multi-hop query rewriting, evidence sufficiency assessment, and answer generation. The benchmark is divided into `e2e-rag-db` and `e2e-rag-qna`: the former evaluates index construction by the number of documents processed per second, while the latter evaluates the entire question-answering pipeline by the number of tasks completed per second.
The reference workload uses 824 questions from FRAMES and a fixed version of 2,515 Wikipedia HTML documents, split into approximately 107,000 overlapping chunks. The system uses E5-base-v2 to create 768-dimensional vectors, FAISS HNSW for storage, and ColBERTv2 for reranking. GPT-OSS-20B/120B handles document assessment, query rewriting, and generation, with up to five retrieval rounds per question. This design allows participants to compare model colocation, different numerical precisions, CPU/GPU configurations, cross-stage batch scheduling, and prefix caching instead of optimizing only the large-model core.
However, the first release supports only the Offline scenario and does not cover online arrival rates, tail latency, or concurrency jitter. To improve reproducibility, performance mode replays prerecorded inputs for each round, fixing the retrieved documents and number of iterations; the models still perform the computations, but their outputs are discarded. This means its throughput is closer to a system stress test with a known execution graph and cannot be used to directly infer dynamic retrieval quality. The reference answer accuracy is also only 35%, and the compliance threshold is 97% of that result. Current audits mainly check the final generated output length and do not yet comprehensively prevent participants from reducing intermediate retrieval or assessment work. Engineering teams should wait for the first cross-hardware submissions and watch for a future Server scenario and more comprehensive per-stage compliance testing.