AgentX 1.0 Replays Million-Token Inference Workloads from 393 Real-World Agent Traces

SemiAnalysis has released AgentX 1.0, adding replayable agentic programming workloads to InferenceX. Traditional inference benchmarks typically fix input and output lengths and then measure tokens per second. AgentX instead reproduces multi-turn conversations, pauses caused by tool calls, bursts of concurrent subagents, and rising KV-cache hit rates as conversation history accumulates. Together, these factors strain HBM capacity, prefix routing, cross-node KV transfers, and DRAM/SSD offloading, making the benchmark more representative of how long-running coding agents are served.

The first Apache 2.0 dataset contains 393 anonymized Claude Code agent traces, comprising 56,798 primary-agent turns, 1,697 subagent launches, and 98,827 model requests. According to the dataset card, it includes approximately 21.6 billion input tokens and 106 million output tokens; 44% of sessions launched at least one subagent. The full version retains requests approaching one million tokens, while a derived dataset capped at 256K tokens per request is also available for testing models with shorter context windows.

The replayer uses AIPerf to issue requests according to their original relative timing and sweeps across different numbers of concurrent users. It measures per-user output rate, time to first token, end-to-end latency, per-chip throughput, power consumption, and prefix-cache behavior. A public dashboard, REST API, configurations, execution logs, and result-processing code allow engineers to trace the model, hardware, precision, and serving framework combination behind any individual data point. This is better suited than peak tokens/s alone for validating session-aware routing, disaggregated prefill/decode, and hierarchical KV caches.

The limitations are also clear: the traces come primarily from SemiAnalysis’s Claude Code agent traffic and therefore cannot represent every research, browsing, or enterprise agent. The content was anonymized and stored in 64-token KV blocks, so input lengths are estimates based on cache blocks; for extremely long contexts, they may overstate the actual tokenizer counts. More importantly, the replayer does not re-execute the original programming tasks. Its results can therefore answer only how fast and efficient a serving system is, not whether the model writes correct code. Future developments worth watching include public traces from other harnesses, independent cross-vendor reproduction, and whether cache hit rates remain aligned with real-world task completion times.