MLPerf Inference 6.1 Adds End-to-End RAG and Edge Agent Tests as Benchmark Expands to Full Inference Pipelines

MLCommons released MLPerf Inference 6.1 on September 16. The most significant change is not that a particular accelerator set a new record, but that the unit of evaluation is shifting from a single model to a complete application pipeline. The new end-to-end RAG benchmark measures document ingestion and online question answering separately. The former covers embedding generation and vector database construction, while the latter sequentially performs query embedding, retrieval, reranking, and inference with one or more LLMs. This makes vector indexing, data movement, and inter-component scheduling part of the standard score for the first time, better exposing systems that optimize the generation core but are slowed down by the retrieval layer.

Another new benchmark, Edge Agentic Inference, simulates a coding-agent workload for a single user. The context grows over multiple interaction turns, requiring the model to alternate between gathering evidence and reasoning. The evaluation includes both time-bounded accuracy and deterministic tasks with built-in correctness checks. Interactive scenarios in the GPT-OSS benchmark now also formally permit speculative decoding, enabling compliant comparisons of complete implementations that combine draft and verification models.

Inference 6.1 received submissions from 30 organizations, covering clusters with as many as 512 accelerators, heterogeneous systems spanning multiple vendors, and distributed systems operating across the Pacific. More than half of the submissions used the new API-based harness, which sends requests through an actual client/server interface. MLCommons says MLPerf Endpoints will eventually replace the data center edition of MLPerf Inference. This will force vendors to address serialization, queuing, networking, and server-side scheduling rather than merely demonstrating in-process kernel throughput.

However, the new benchmarks still use controlled workloads and do not capture the data quality, recall, or agent-tool reliability of real-world RAG systems. Engineering teams comparing results should also distinguish between the Closed and Open divisions, latency constraints, accuracy thresholds, and cluster sizes. Improvements of up to 5.7× across benchmark rounds cannot be attributed directly to a single chip. The next questions are whether third parties can reproduce the raw results and whether MLPerf Endpoints will publish enough metrics to compare cost, tail latency, and multi-tenant interference.