Fast Gemma Verified Build Breaks 510 Tokens per Second on a Single A10G by Combining Sliding-Window Attention and MTP

The Fast Gemma Challenge requires participants to serve `google/gemma-4-E4B-it` on a fixed single NVIDIA A10G, without replacing the model or disabling features, while keeping perplexity below approximately 2.42. On August 3, VIDRAFT published its complete configuration: all 128 reruns on private prompts passed, with throughput reaching 510.58 tokens/s and PPL at 2.3930. A self-reported result reached 535.91 tokens/s, but it was not verified because it exceeded the quality threshold.

This configuration does not rely on a single core optimization. It reduces the attention window to 188 tokens to lower the memory bandwidth required to read the KV cache during decoding. It also uses a multi-token prediction drafter with seven speculative tokens to accelerate each decoding step. The inference path additionally incorporates INT4 weights, a pruned LM head, CUDA graphs, fused sparse argmax, split-KV verification, deferred detokenization, and other adjustments. Before testing, the system processes 64 synthetic single-token prompts to complete JIT compilation and CUDA graph capture in advance, which the team estimates avoids a timing-period loss of roughly 15 tokens/s.

From an engineering perspective, the configuration notably disables benchmark precaching that could not be reproduced in private reruns. VIDRAFT also published the `manifest.json`, serving code, and weight bucket, making the figures more auditable than a simple leaderboard screenshot. However, the 188-token sliding window may alter long-range dependency behavior, and the PPL threshold represents only the quality proxy metric specified by the organizers. The next step should be to measure TTFT, tail latency, VRAM usage, and output quality across different prompt lengths, concurrency levels, and GPUs to determine whether these techniques can be transferred from a single-stream competition benchmark to real-world serving.