推論系統
SGLang 0.5.18 Overlaps Weight Loading with CUDA Graph Capture, Cutting Qwen3-32B Cold Start to 35.6 Seconds
The new release processes checkpoint staging and CUDA Graph capture concurrently during startup, delivering a 2.38× speedup over the standard default workflow in official H100 testing. It also rewrites parts of the tensor-parallel communication path and adds seven model families, but the Torch upgrade and cache-path consolidation introduce migration costs.
SGLang 0.5.18 treats model cold starts as an I/O and GPU initialization problem whose operations can overlap: while checkpoint pages are loaded from storage and staged, the system begins capturing CUDA Graphs instead of waiting for the two phases to run sequentially. With --startup-weight-load-mode overlap, official testing with Qwen3-32B on a single H100 measured a startup time of 35.6 seconds, compared with 84.8 seconds for the standard default workflow. However, compared with a sequential workflow that already has prefetching enabled, the improvement is only 8.6% to 11.7%; the “2.38×” figure cannot be directly extrapolated to clusters that already use weight caching or different storage backends.
The decoding path also includes two more concrete changes. In pure data-parallel attention configurations, the TP LMHead previously required an all-gather followed by a scatter; these operations can now be combined into a single all-to-all. On a B200, DeepSeek-V4-Pro’s LMHead time fell from 320 microseconds to 169 microseconds, but overall TPOT decreased only from 36.97 milliseconds to 35.67 milliseconds. In TP4 Blackwell testing of DeepSeek-V4-Flash, FlashInfer MNNVL replaced some NCCL all-reduce operations, improving small-batch decoding performance by up to 6.9%. These figures show that operator-level speedups do not necessarily translate proportionally into lower end-to-end latency.
Version 0.5.18 also adds seven autoregressive or diffusion model families, including Muse Glimmer, SANA-Video, and LTX-2.5, while integrating DSpark logprob output and multiple fixes for DeepSeek, Qwen, and Kimi. Deployers should still account for breaking changes before upgrading: the CUDA stack moves to Torch 2.13 and Triton 3.7.1, while torchao integration has been removed. Compilation caches for Triton, FlashInfer, Inductor, DeepGEMM, and other components have also been consolidated under SGLANG_CACHE_DIR, so the first startup will trigger recompilation. For inference platforms that rely on autoscaling, the next step should be to retest cold-start performance with their own models, network file systems, and warm-up strategies rather than relying solely on the official H100 results.