LLM 推論系統
TEMPO Assigns MoE Experts Based on Measured Runtime, Cutting Qwen3-235B p99 Token Latency by About 15.6%
Rather than balancing only tokens or the number of active experts, TEMPO estimates both HBM weight-loading costs and GEMM tile padding. An SGLang prototype improves throughput by 4% to 6% on some Qwen3-235B workloads, but adds only overhead on the communication-bound DeepSeek-V3.
In expert-parallel MoE inference, every layer must wait for the slowest GPU, so common dispatchers balance either the number of tokens or the number of active experts per GPU. TEMPO's measurements show that both proxy metrics overlook the piecewise cost structure of grouped GEMM: when an expert receives fewer than roughly 156 to 168 tokens, most of the time is spent loading weights from HBM; above that threshold, computation is rounded up in 128-token M-tiles. Splitting the same expert across more replicas can therefore both duplicate weight transfers and introduce padded computation.
The team models runtime as `t=max(a+bG, c+βN)`, where G is the number of active expert replicas on a GPU and N is the number of tokens, with an additional tile term for larger prefill workloads. Real-world decode batches typically contain a small number of hot experts alongside many cold experts. Across four workload traces, 92% to 100% of batches simultaneously included experts in both the memory-bound and compute-bound regions, meaning any single balancing metric has regimes in which it fails.
TEMPO formulates per-batch assignment as a fixed-cost makespan problem and uses a heuristic solver that runs in milliseconds. Solving is moved to a separate process, while dispatch and counting are fused into a kernel inside the CUDA graph and integrated with SGLang. When Qwen3-235B-FP8 operates within the predicted effective region, end-to-end throughput increases by about 4% to 6%; under a medium Poisson load, p99 per-token latency falls from 226 ms to 191 ms. Because DeepSeek-V3 has more experts per GPU and a higher share of communication overhead, every adaptive strategy trails static assignment by about 2% to 3%, demonstrating that TEMPO is not a universal accelerator.
The most practical engineering lesson is not to adopt a fixed strategy, but to first measure the kernel, model shape, batch characteristics, and network topology, then determine whether any recoverable imbalance exists. The paper anonymizes the GPU model, and some results at EP32 and above still come from calibrated simulation. Driver or kernel updates also require recalibration, and the current system does not yet jointly optimize expert placement and per-batch assignment.