基準與基礎設施
MLPerf Endpoints v0.7 Launches, Adding Continuous Submissions and a Comparable Results Pipeline for Agentic Inference
MLCommons has released Endpoints v0.7, transforming inference testing from fixed-batch submissions into endpoint benchmarking infrastructure with automated review and continuous updates. The upcoming multi-turn agentic workloads will measure both per-user progress and aggregate throughput, rather than counting only tokens/s for isolated requests.
MLCommons released MLPerf Endpoints v0.7 on July 28, completing an automated submission pipeline, continuous review tools, and dynamic results visualizations. Rather than being another static leaderboard, it enables inference services exposed as endpoints to run under the same workloads and rules, laying the groundwork for rolling submissions and agentic workloads in v1.0. The reference implementation is available in a public GitHub repository; a server only needs to expose a compatible interface to connect to the load generator and results-packaging workflow.
This infrastructure update matters because conventional single-turn tokens/s does a poor job of characterizing agents. Each agent turn depends on the preceding turn, while the continually expanding conversation history places simultaneous pressure on prefill, KV cache capacity, prefix reuse, and conversation-aware routing. For the forthcoming Agentic Inference benchmark, MLCommons has defined a two-axis Pareto curve: one axis measures output throughput across the entire system, while the other measures the rate of progress for each user or trajectory. This directly shows the tradeoff between aggregate throughput and individual agent completion time as concurrency increases.
The associated agentic workloads also introduce fixed session IDs, cache salting, deterministic prompt reconstruction, and inline accuracy checks within the same performance configuration. These safeguards prevent submitters from claiming artificial speedups through cross-task caching, truncated responses, or altered output distributions. This is especially useful for developers of vLLM, SGLang, TensorRT-LLM, and model gateways: whether a scheduler preserves conversational KV locality will become a measurable system capability.
For now, v0.7 remains a foundational release. Buyer-oriented normalization rules, complete agentic results, and open rolling submissions are expected in v1.0. Engineering teams should first ensure that benchmark configurations lock the model version, quantization scheme, and accuracy threshold; otherwise, cross-vendor Pareto curves may still compare outputs of differing quality.