AIPerf Reworks LLM Benchmarking with Multiple Processes, but Streaming Metrics Still Require Token-Counting Checks

On September 18, NVIDIA detailed AIPerf’s architecture and deployment testing methods, positioning it as the successor to GenAI-Perf. The central change removes the dependency on Perf Analyzer, instead using multiple processes to generate requests and process results, coordinated through ZMQ. This reduces the likelihood that high-concurrency tests will hit the client’s processing limits first. [Technical article](https://developer.nvidia.com/blog/benchmarking-llm-inference-at-scale-with-aiperf/)

This design affects measurement reliability: if the load generator cannot sustain the intended traffic, the observed throughput ceiling may reflect the client’s capacity. AIPerf distributes work across independent services and offers concurrency, request-rate, and trace-replay modes. The public repository also supports custom data and endpoint extensions, allowing the same measurement workflow to be used across different inference services. [Project documentation](https://github.com/ai-dynamo/aiperf)

Traffic patterns are another focus. In addition to fixed intervals, the tool can generate Poisson and gamma arrival distributions, adjust burstiness, gradually ramp up load, vary input and output lengths, and replay existing service traces. From an engineering perspective, this helps reveal tail latency caused by batch scheduling, mixes of long and short prompts, and cache contention. A single average tokens-per-second figure cannot capture these differences. [Load configuration details](https://developer.nvidia.com/blog/benchmarking-llm-inference-at-scale-with-aiperf/)

Metrics still need to be interpreted according to their definitions. Time to first token includes network latency, queuing, and prompt processing. Default inter-token latency (ITL) is calculated by dividing the elapsed time after the first chunk by the output length minus one; it is not based on individual token timestamps. If the server bundles multiple tokens into the first response chunk, single-user throughput may be overestimated. The documentation provides a correction based on per-chunk usage, but this requires a compatible endpoint, streaming mode, and server-reported token counts. [Metric definitions](https://github.com/ai-dynamo/aiperf/blob/main/docs/metrics-reference.md)

Existing GenAI-Perf scripts also need review. The migration guide provides worker-process control through `--workers-max`, removes the old pass-through argument separator, and notes that some `analyze` functionality is not yet supported. Teams should first establish a baseline comparison using pinned versions, identical data, and the same stopping criteria, then introduce real-world traffic replay. The article does not validate measurement discrepancies across tools, so it does not establish that client-side bottlenecks have been eliminated for every workload. [Migration guide](https://github.com/ai-dynamo/aiperf/blob/main/docs/migrating.md)