AI 推論基礎設施
Cloudflare Separates Prefill and Decode Precision, Boosting GLM 5.2 Decode Throughput by Up to 55%
Cloudflare uses an FP8 KV cache to expand Kimi K2.6’s concurrency capacity and switches to INT4 weights for GLM 5.2 decoding. The system also adds page-generation tags to the shared KV cache to prevent multi-tenant requests from reading incorrect cached data.
Cloudflare has published the production configuration it uses to serve large sparse models on Workers AI. Rather than applying uniform quantization across the entire inference pipeline, its approach selects precision separately according to the bottlenecks in prefill and decode. All experiments were run with the open-source SGLang framework using a disaggregated two-stage deployment on H200 GPUs.
For Kimi K2.6, Cloudflare changed the decode-stage KV cache from BF16 to FP8 e4m3, increasing the total context capacity from roughly 686,000 to 1.37 million tokens. At the same concurrency level, the FP8 kernel is actually slightly slower—for example, single-request throughput falls from 137 to 125 tokens per second. The real benefit comes from memory capacity. BF16 runs out of memory beyond 32 concurrent requests, while FP8 can sustain 64 requests and 2,192 tokens per second. That is about 41% higher than BF16’s peak throughput, while the cost per token reportedly falls by approximately 30%.
For GLM 5.2, the weights were compressed from FP8 to INT4, reducing the checkpoint size from 705 GB to 421 GB. With eight-way tensor parallelism, per-GPU weight usage falls from roughly 88 GB to 52 GB. Because decode is memory-bandwidth-bound, single-request throughput rises from 60 to 92 tokens per second. However, prefill—where weights must first be dequantized—slows from 10,160 to 8,660 tokens per second. Cloudflare therefore uses INT4 only for decode while retaining FP8 for prefill.
Higher density also increases the isolation risks associated with a shared cache. The new mechanism assigns each physical KV page a tag that changes whenever the page is reallocated. Before decoding, the system verifies that the page and tag match those expected by the request; if they do not, the request is aborted. Cloudflare measured an overhead of less than 1% for both throughput and p95 latency. However, its quality conclusions are based primarily on benchmarks selected by Cloudflare, some of which are internal tests. Engineering teams should still validate quantization error and isolation behavior using their own long-context, tool-calling, and multi-tenant stress tests.