推論系統
MoE Router Cuts Cache Misses by 60% but Fails the 1% Perplexity-Degradation Threshold
A preregistered study attempted to train expert locality directly into an MoE router, but found a clear trade-off between cache performance and model quality. Combining training-based and training-free rerouting reduced misses further, yet still did not demonstrate a proportional end-to-end speedup.
When GPU memory is far smaller than the model weights, an MoE model’s theoretically low number of activated parameters does not translate into low data movement: the experts selected for each token may still need to be loaded from RAM or an SSD into the GPU. In a new study, researchers ran a 134GB, Q4_K_M-quantized Qwen3-235B model on an 8GB GPU and measured post-warmup decoding at just 0.44 token/s, consistent with a model based on bytes required per token divided by measured bandwidth. An initial plan to amortize SSD scans across a batch instead collapsed under paging thrash at batch size 32.
The authors also developed `llama-moe-trace`, which captures Qwen3-30B routing decisions without modifying the model. Measurements showed that adjacent tokens reuse the same expert at roughly twice the random baseline rate. Meanwhile, 95% of traffic was concentrated among 52.5% of the experts, and an LRU cache large enough to hold only 13.4% of the experts could serve 66% of requests. This indicates that the router already exhibits exploitable locality, but not enough to eliminate the SSD bandwidth wall.
The study then trained a 137M-parameter MoE under preregistered conditions, adding locality and domain-routing losses. The mechanism did reduce cache misses by as much as 60%, while some static fixed configurations even achieved a 99% cache hit rate. However, every configuration failed the quality threshold requiring perplexity degradation of no more than 1%. Experiments with a 340M-parameter model likewise provided no evidence that scaling would eliminate the cost. Combining trained locality with cache-aware rerouting at inference time reduced misses by about 80% at both model scales, at the cost of up to 3.4% perplexity degradation.
This is a valuable negative result: edge MoE deployments cannot assume that a router can be made more cache-friendly for free. However, the training experiments used models far smaller than the measured 30B and 235B models, and the reduction in cache misses was not accompanied by comprehensive end-to-end results for latency, energy consumption, or different SSDs. In practice, engineers still need to compare alternatives such as tiered placement, prefetching, quantization, and direct CPU computation.