推論系統
SGLang 0.5.19 Adds Beam Search and Restructures Inference Paths Across NVIDIA and AMD
The latest SGLang release can return multiple candidate sequences from the same service using `beam_width` and makes the unified radix tree the default cache for all models. The update also expands MoE, long-context, and ROCm kernels, but introduces dependency and configuration behavior changes that require revalidation before upgrading.
The most immediately visible change in SGLang 0.5.19 is native Beam Search support in the inference server: requests can pass `beam_width` to obtain multiple higher-scoring candidate sequences, while coexisting with standard sampling requests. This is useful for speech recognition, translation, code generation, and workflows that use a post-processing verifier for reranking. The current limitation is that it cannot be used together with speculative decoding, prefill/decode disaggregation, DP attention, or HiCache.
On the runtime side, the unified radix tree is now the default KV cache for all configurations, rather than serving only hybrid architectures. It allows decode workers for sliding-window models to reuse prefixes and supports attaching or detaching an L3 storage tier while the server is running. The MoE path adds DeepEP v2 ElasticBuffer, enabling FP8 cross-node communication for DeepSeek-V3/V4 and Qwen3-MoE to work with CUDA Graph using fixed buffers. For W4A8 MoE on Hopper, official measurements show an approximately 12% increase in output throughput for DeepSeek-V4-Flash, although the result has not yet been independently reproduced.
The hardware optimizations are not limited to NVIDIA. MI300X and MI355X gain persistent Lean Attention, which redistributes decode workloads with uneven sequence lengths to idle compute units. Release data claims up to 1.52× higher throughput and a 3.62× reduction in inter-token latency on MI355X. Blackwell’s MLA path also adds decode context parallelism, primarily targeting long contexts such as 128K.
The upgrade risks are equally concrete: FlashInfer 0.6.18 is now a required dependency for certain features; programs that construct `ServerArgs` directly must explicitly call `resolve_once()`; and the Spark3 model and tool parser names have been changed to Spark2.5. The team has also introduced count and length limits for stop strings and regular-expression configurations. Production environments should rerun accuracy, cache hit rate, TTFT, TPOT, and mixed-request tests separately rather than judging the benefits solely from an official benchmark on a single hardware platform.