AMD details MLPerf inference optimizations, boosting eight-GPU GPT-OSS serving throughput by about 38%

On September 17, AMD published an engineering analysis of MLPerf Inference 6.1, explaining how it improved GPT-OSS-120B inference on the same eight MI355X GPUs. The company reported that throughput in the Server scenario rose from approximately 82,100 tokens per second in the previous round to 113,200, an increase of about 38%. This is a whole-system result under specific benchmark conditions. [Technical analysis](https://rocm.blogs.amd.com/artificial-intelligence/mlperf-inf-v6.1/README.html)

One key change was splitting the previously shared attention kernel into separate prefill and decode paths. Prefill processes chunks of input and tends to be compute-intensive, while decode reads the paged KV cache step by step and is more constrained by memory bandwidth. AMD uses AITER’s variable-length attention for prefill and a separate decode kernel written in Gluon, reducing stalls through data layout changes and overlapped loads. The company claims a 25% to 28% improvement in attention performance. [Kernel design](https://rocm.blogs.amd.com/artificial-intelligence/mlperf-inf-v6.1/README.html)

The scheduler also incorporates service targets of three seconds to the first token and 80 milliseconds per subsequent token, aligning request scheduling with latency constraints. The technical implication is that faster kernels must be accompanied by control over the workload entering the system. Improvements to individual kernels and overall throughput use different denominators and cannot simply be added together. [Scheduling details](https://rocm.blogs.amd.com/artificial-intelligence/mlperf-inf-v6.1/README.html)

The quantization configuration must also be held fixed. The reproduction guide links to AMD’s public weights, and the quantization script in the model card uses MXFP4 weights and FP8 activations, excluding components such as attention, routers, and the output layer from quantization. The model card’s overview describes activation precision differently, so anyone reproducing the results should cross-check the actual model configuration, scripts, and container version rather than infer execution precision from the model name alone. [Model card and scripts](https://huggingface.co/amd/gpt-oss-120b-w-mxfp4-a-fp8-Mlperf)

The quality figures in the model card come from evaluations on AIME25 and GPQA at low reasoning effort. They do not directly represent performance with larger reasoning budgets or on Chinese-language workloads. Teams that rely on long-form code generation, tool calling, or Chinese-language question answering still need to assess post-quantization error patterns and output quality before applying the benchmark results to capacity planning. [Evaluation conditions](https://huggingface.co/amd/gpt-oss-120b-w-mxfp4-a-fp8-Mlperf)

AMD has listed containers and commands for performance and quality testing, but the guide still contains a ROCm version placeholder, and the multi-node interconnect specifications and launcher details remain incomplete. This provides an actionable starting point, but the public documentation cannot yet be treated as a complete, unambiguous recipe for reproducing the results. [Reproduction guide](https://rocm.blogs.amd.com/artificial-intelligence/mlperf-inf_v6.1-repro/README.html)

MLCommons’ Closed division requires the same reference model while allowing optimizations within its rules. Throughput is also no substitute for measurements of whole-system power consumption. For deployment teams, the next step is to rerun tests using their own input lengths, concurrency levels, and tail-latency targets while holding the quality threshold fixed, then determine whether these changes reduce the cost per request. [MLCommons rules overview](https://mlcommons.org/benchmarks/inference-datacenter/)