推論系統
FreeBalance Predicts Popular MoE Experts Early, Cutting Prefill Latency by Up to 13.1% on Eight A800 GPUs
FreeBalance uses the previous layer’s residual state to predict expert load in the next layer, overlapping weight transfers with attention computation. The method does not alter the final routing decisions or model outputs, but the evidence so far covers only two MoE models, a single eight-GPU topology, and prefill workloads.
MoE serving systems typically distribute different experts across multiple GPUs. When the router directs a large number of tokens to the same rank, the other devices must wait for the slowest node even if they finish earlier. Conventional online load balancing waits for routing results before moving expert weights, placing the transfer on the inference critical path. The [FreeBalance paper](https://arxiv.org/abs/2608.14205) instead uses the residual hidden state output by layer ℓ−1 to invoke layer ℓ’s existing router early and estimate how many tokens each expert is about to receive. This prediction determines only the physical placement; the actual MoE computation still uses the original routing results, so it neither approximates logits nor changes the model’s semantics.
Based on the predicted load, the system schedules paired expert swaps between the busiest and least busy ranks. Its cost model uses measured attention time, interconnect bandwidth, and startup latency to limit the number of swaps, aiming to hide weight transfers beneath the next layer’s attention computation. Paired swaps also preserve the number of experts and memory footprint on each GPU. Every rank independently reconstructs the plan using identical statistics and deterministic sorting, eliminating the need to broadcast the full placement separately. The paper evaluates Qwen3-30B-A3B and Moonlight-16B-A3B on eight NVIDIA A800-SXM4 GPUs with EP=8, batch size 16, and 8K-token inputs. Qwen3-30B-A3B itself has [128 experts, with 8 activated per token](https://huggingface.co/Qwen/Qwen3-30B-A3B-Instruct-2507).
The authors report a 32.8% reduction in the maximum-to-average rank load ratio, up to a 13.1% reduction in end-to-end prefill latency, and the ability to hide the swapping cost of 5.1 experts per layer on average. Gains vary across LongBench subsets, while mixed workloads deliver less improvement than the best-case results. The next engineering questions include the public implementation, the decode phase, small batches, cross-node InfiniBand, and hardware with different compute-to-communication ratios, such as H100 and B200 GPUs. Although inaccurate predictions do not affect the answers, they may still move the wrong experts and waste bandwidth.