GitHub Repo
SGLang 0.5.22 Adds Deferred KV Cache Allocation Option to Improve Certain Mixed Short- and Long-Prompt Workloads
Released on October 19, SGLang 0.5.22 lets decode nodes in disaggregated deployments wait until prefill is complete before allocating KV space for a request. Official tests on specific workloads showed higher throughput but worse tail inter-token latency; the feature remains optional.
SGLang released 0.5.22 on October 19, adding a memory scheduling option for prefill/decode disaggregated deployments: defer reserving KV cache space for each request on the decode node. The release also raises the minimum Python requirement to 3.11, so check your runtime environment before upgrading. Official release notes
The existing default policy reserves space on the decode side before prefill has finished. While a long prompt is still being processed, those slots may be temporarily unavailable to requests that are ready to generate. The new prefill_complete policy instead waits for the signal that the final prefill result is ready before allocating destination slots and starting the transfer handshake. It changes when space is reserved; the total KV pool size remains unchanged. Official technical documentation
Historical tests included in the merged proposal used GLM-5.2-NVFP4 with 12 Rubin GPUs and deliberately constrained decode KV capacity. Under a mixed workload of long and short prompts, two test runs showed a 26.81% increase in average system output throughput and a 48.87% decrease in p95 time to first token. The tradeoff was a 92.77% increase in p99 inter-token latency. A separate comparison using Qwen3-8B showed no clear benefit. These results predate a subsequent rebase and are not independent validation of the final release version. Test configuration and results
To enable the feature, set --disaggregation-decode-allocation-policy prefill_complete on both sides. The initial version requires Mooncake transport and the Python bootstrap registry; attention TP, CP, and pipeline sizes must all be one. Combinations with the decode radix cache, HiSparse, and other features are still unsupported. The documentation says to upgrade all workers under the default early policy first, then switch them over together. Deployment limitations
The engineering tradeoff to watch is between capacity utilization and streaming stability: deferred allocation keeps KV on the source side for longer and gives up the opportunity to overlap chunked prefill with transfer. For long-context retrieval or agent services, compare throughput, time to first token, and inter-token latency using your actual prompt-length distribution to determine whether enabling it is worthwhile.