推論與模型服務
SGLang 0.5.16 Dynamically Adjusts Speculative Verification Length Based on Confidence and Overhauls Caching for Hybrid Architectures
The new SGLang release introduces DSpark, which changes the verification window for each round based on the draft model’s confidence, and makes UnifiedRadixTree the default cache for sliding-window, Mamba, and sparse-attention models. The release also removes several quantization backends and changes the rollout transport format, so compatibility must be checked before upgrading existing services.
The headline feature in SGLang 0.5.16 is DSpark. The draft model first generates candidate tokens in blocks using a semi-autoregressive approach, then dynamically determines the verification window sent to the target model based on its own confidence, instead of using a fixed draft length for every round. In official single-request testing of DeepSeek-V4-Pro on a B300 with TP8, SGLang reported 383.7 tokens/s at an acceptance length of approximately 5. Deployment requires selecting the `DSPARK` algorithm, enabling compact ragged verification, and tuning the block size for the model. Because these results come from a specific hardware and batching configuration, they cannot be directly extrapolated to multi-tenant throughput or tail latency.
The new release also makes UnifiedRadixTree the default cache structure for sliding-window attention, Mamba, and DSA models, allowing prefix hits and the recurrent state of state-space models to use the same management path. With context-parallel prefill, GLM-5.2 can partition DSA KV and indexer cache layers by rank. In official testing with 8,192 tokens, 78 layers, and `cp_size=4`, per-rank cache usage fell from 0.77 GB to 0.20 GB, a reduction of approximately 74%. ReplaySSM’s ring-based speculative verification also eliminates the need for a complete SSM snapshot for every draft, reducing temporary storage from 11.5 GB to 1.8 GB in a Qwen3.5-35B-A3B TP1 test.
Model support now includes Inkling, LongCat 2.0, Mellum v2, Pi0.5, and LongLive 2.0. Inkling, which has 975B parameters and a one-million-token context window, has been validated on Blackwell, H200, and MI350X/MI355X hardware. Although the necessary code for MiniMax-M3 has been merged, the official release notes still indicate that it is not yet functional end to end.
Upgrade risks are equally important. QServe W4A8, FBGEMM FP8, some CUTLASS paths, and the built-in NVFP4 JIT kernels have been removed, while NVFP4 GEMM now depends on FlashInfer. Diffusion-model RL rollout responses have also changed from JSON/base64 to raw MsgPack bytes. Other known issues include nondeterminism at temperature 0 and a reverted Mamba fix. Engineering teams should rerun correctness and latency tests using their own batch configurations, GPUs, models, and identical prompts rather than upgrading solely on the basis of official microbenchmarks.