推論基礎設施
SGLang Adds GPU-Free Simulator to Official Documentation for Replaying Inference Traffic with Latency Predictions
SGLang Simulator intercepts runtime components on the CPU to simulate scheduling, batching, and multi-tier KV caches without actually loading models or executing GPU kernels. The team claims errors below 5% on key metrics for two Qwen3 models running on H20 GPUs, but hardware and model coverage remain limited.
SGLang updated its documentation on September 5, listing the Simulator—previously developed through a public roadmap—as an advanced feature. Rather than shrinking models or performing slow inference on CPUs, it uses class and module hooks at startup to replace the Scheduler, ModelRunner, TokenizerManager, RadixCache, HiCacheController, storage backends, and GPU kernel loading process. Actual matrix computations are instead handled by AIConfigurator, which predicts the execution time of each batch based on hardware specifications and pre-profiled operator data.
The simulator preserves SGLang’s HTTP interface and `bench_serving` metrics format, while using a mock memory pool to track KV quotas. Existing load-testing clients and production traffic traces can therefore be replayed directly. Current modes cover no caching, HBM L1, DRAM L2, and disk L3. It also provides a blocking mode that waits according to real elapsed time, as well as an offline mode that the project says can run 10 to 1,000 times faster. In practice, platform teams can first sweep request rates, batch configurations, and combinations of RadixAttention and HiCache, then submit a small number of candidate configurations for costly testing on real hardware.
Public validation has primarily focused on Qwen3-8B and Qwen3-32B-FP8 running on H20 GPUs. Official tables show mean absolute percentage error (MAPE) below 5% for average time to first token (TTFT), time per output token (TPOT), inter-token latency (ITL), throughput, and total execution time. However, these figures are still provided by the development team, and the predictor depends on operator profiling. Its error rates cannot be directly extrapolated to mixture-of-experts (MoE) or multimodal models, new GPUs, cross-node communication, or disaggregated prefill and decode. The roadmap also continues to list multi-instance routing, HybridRadixCache, parallel data transfers, and more sophisticated L2 writeback and eviction behavior as pending work. Engineers should next watch for the release of cross-hardware calibration data and compare simulated rankings against results from real hardware using their own traces. For now, the simulator is better suited to configuration search than to capacity commitments or procurement decisions.