AI 基礎設施
NVIDIA Groq 3 LPX Enters Mass Production: Static Scheduling Delivers 3,431 Output Tokens per Second at 100K Context
Groq 3 LPX integrates 256 LP30 chips, 128GB of SRAM, and compiler-prescheduled inter-chip transfers into a low-latency decoding system. Third-party testing shows it leading existing public endpoints in long-context speed, but cost, concurrent throughput, and production-service performance remain undisclosed.
NVIDIA announced that Groq 3 LPX has entered mass production. It is positioned not as a replacement for Vera Rubin NVL72, but as a complement for the low-batch decoding stage—the hardest part of agentic inference to accelerate. The system can use Rubin GPUs for prefill, long-context processing, and the KV cache, then hand off token-by-token generation to LPX. In another configuration, attention remains on the GPU while LPX executes the FFN; LPX can also serve as an external draft model for speculative decoding.
An LPX rack consists of 256 LP30 LPUs providing a combined 128GB of SRAM. Each chip has 96 point-to-point links, each running at 112Gbps. The key is not merely greater bandwidth, but a deterministic execution model that allows the compiler to schedule every computation and data transfer in advance, eliminating runtime arbitration. Matrix multiplication results can be divided into 320-byte vectors and transmitted across chips as they are computed, reducing the impact of fixed communication latency that can otherwise negate the benefits of tensor parallelism at small batch sizes.
Artificial Analysis tested Gemma 4 31B in an NVIDIA data center. With a 100K-token input context, median output speed reached 3,431 tokens per second, compared with 870 for the fastest public endpoint available at the time; at 10K context, it reached 3,382 tokens per second. The test used the same `o200K_base` tokenizer for token counting and reported no loss in output precision or model quality. NVIDIA’s own SPEED-Bench testing recorded a separate median of 4,767 tokens per second, but this was not an equivalent independent test.
The design is particularly attractive for agents that require dozens or hundreds of tool calls: decode can remain interactive even as the context continually grows. However, the current figures reflect low-batch performance for a single model on a specific rack configuration. They do not address multi-tenant concurrency, the cost of handing off the KV cache after prefill, power consumption, or pricing per million tokens. Engineering teams should next examine latency percentiles from production services operated by Nebius, Groq, and others, as well as whether the same advantage holds across different MoE architectures, quantization formats, and large batch sizes.