OpenAI Jalapeño Delivers 1.5–1.9× Higher Throughput per Watt on Three Open Models, but Long-Context Agent Workloads Remain Untested

OpenAI has released the first InferenceX benchmark results for Jalapeño, its first-generation in-house inference chip. On single-turn workloads with 8K input and 1K output tokens across GPT-OSS-120B, DeepSeek R1 670B, and Kimi K2.5 1T, Jalapeño used single-token prediction without multi-token prediction, speculative decoding, or prefill/decode disaggregation. Even so, its peak throughput per watt was 1.5–1.9× higher than the comparison systems, while end-to-end latency was 1.7–3.6× lower.

For DeepSeek R1, OpenAI reports that Jalapeño processed 19,641 mixed tokens/s per kilowatt, compared with 11,781 for GB300; end-to-end latency was 1.65 seconds versus 5.99 seconds. These figures were normalized using the chips’ rated power: 700W for Jalapeño and 1,400W for GB300. OpenAI says Jalapeño’s sustained power consumption did not exceed 550W during testing.

The architecture’s key feature is not the fixed assignment of prefill and decode to separate chip pools. Instead, it reduces data movement through homogeneous compute resources, sharded HBM, localized KV cache, and a dedicated collective network. SemiAnalysis reports that the production version will use HBM4, provide 15.4TB/s of memory bandwidth per package, and scale through a cross-rack interconnect domain spanning 2,048 XPUs. OpenAI also says Codex helped port three models that were not originally planned for the platform, with some attention and MoE block kernels running 1.5–1.8× faster than manually written versions. This is not an overall model-level speedup.

The limitations are significant. Although SemiAnalysis observed and verified the runs in the lab, the raw data was still supplied by OpenAI, and the full benchmark suite was not run. The current 8K/1K single-turn workloads are not representative of AgentX, which will stress routers, prefix caches, KV offload, and concurrent multi-turn execution. The comparison systems also differ in software maturity and HBM generation. Engineering teams should next evaluate production B0 silicon, long-context agent trajectories, failure recovery, and full-rack power consumption—not just package TDP.