AI 推論基礎設施
SGLang 0.5.18 Overlaps Weight Loading with CUDA Graph Capture, Cutting Qwen3-32B Cold Start to 35.6 Seconds
The new release processes checkpoint staging and CUDA Graph capture concurrently during startup, delivering a 2.38× speedup over the standard default workflow in official H100 testing. It also rewrites parts of the tensor-parallel communication path and adds seven model families, but the Torch upgrade and cache-path consolidation introduce migration costs.
SGLang 0.5.18 treats model cold starts as an I/O and GPU initialization problem whose operations can overlap: while checkpoint pages are loaded from storage and staged, the system begins capturing CUDA Graphs instead of waiting for the two phases to run sequentially. With `--startup-weight-load-mode overlap`, official testing with Qwen3-32B on a single H100 measured a startup time of 35.6 seconds, compared with 84.8 seconds for the standard default workflow. However, compared with a sequential workflow that already has prefetching enabled, the improvement is only 8.6% to 11.7%; the “2.38×” figure cannot be directly extrapolated to clusters that already use weight caching or different storage backends.
The decoding path also includes two more concrete changes. In pure data-parallel attention configurations, the TP LMHead previously required an all-gather followed by a scatter; these operations can now be combined into a single all-to-all. On a B200, DeepSeek-V4-Pro’s LMHead time fell from 320 microseconds to 169 microseconds, but overall TPOT decreased only from 36.97 milliseconds to 35.67 milliseconds. In TP4 Blackwell testing of DeepSeek-V4-Flash, FlashInfer MNNVL replaced some NCCL all-reduce operations, improving small-batch decoding performance by up to 6.9%. These figures show that operator-level speedups do not necessarily translate proportionally into lower end-to-end latency.
Version 0.5.18 also adds seven autoregressive or diffusion model families, including Muse Glimmer, SANA-Video, and LTX-2.5, while integrating DSpark logprob output and multiple fixes for DeepSeek, Qwen, and Kimi. Deployers should still account for breaking changes before upgrading: the CUDA stack moves to Torch 2.13 and Triton 3.7.1, while `torchao` integration has been removed. Compilation caches for Triton, FlashInfer, Inductor, DeepGEMM, and other components have also been consolidated under `SGLANG_CACHE_DIR`, so the first startup will trigger recompilation. For inference platforms that rely on autoscaling, the next step should be to retest cold-start performance with their own models, network file systems, and warm-up strategies rather than relying solely on the official H100 results.