vLLM Community Traces H20 MoE Scheduling Inefficiency to Expert-Load Kernel Selection

On September 25, the vLLM community filed a report on H20 inference performance, noting that FlashInfer’s SM90 FP8 mixture-of-experts (MoE) path may select an inefficient matrix-multiplication kernel because of an inaccurate workload estimate. The author proposed a fix to FlashInfer; as of September 27, it had not been merged, and there was no evidence that the regression was introduced by a specific vLLM release. [Community report](https://github.com/vllm-project/vllm/issues/58799)

FlashInfer’s `cutlass_fused_moe` uses CUTLASS to run fused expert computations. Its API supports expert-parallel size and rank, as well as FP8 block scaling. The issue affects this existing execution path and is especially relevant to teams deploying sparse models across multiple GPUs. [Official API documentation](https://docs.flashinfer.ai/generated/flashinfer.fused_moe.cutlass_fused_moe.html)

According to the report, the scheduler originally used the number of input tokens as its workload estimate, but kernel-branch and matrix-tile selection require the number of rows assigned to each expert. In the test configuration, 288 tokens each selected six experts. Distributed across 400 physical expert slots, that came to an average of only 4.32 rows per expert. The patch changes the estimate to “number of tokens × top-k ÷ total physical experts,” rounded up and clamped to at least one. It applies this estimate after calculating the worst-case workspace capacity. [Proposed patch](https://github.com/flashinfer-ai/flashinfer/pull/5560)

This case shows that total batch size alone does not describe the matrix shapes experts actually process. The average also cannot fully capture routing imbalance: when a few experts receive most of the data while other ranks have no work, the best kernel choice may differ. Results from evenly distributed and concentrated workloads should therefore be interpreted separately.

On eight H20 GPUs with uniform routing, the author measured local MoE latency for 288 tokens falling from 1.808 ms to 0.564 ms, a speedup of about 3.21×. The test included five warmup runs and 24 samples, but timing excluded cross-GPU communication, and GPU clocks were not locked. These figures cannot be directly translated into online serving throughput. The author also provided a synthetic reproduction script that does not depend on model weights, so other developers can inspect operator behavior. [Test method and results](https://github.com/vllm-project/vllm/issues/58799)

Validation gaps remain: the benchmark used a controlled source-code backport, while a full upstream build and test run using the original setup are still pending. One independent reference comparison also failed on both the baseline and patched versions, for an unknown reason. Under highly concentrated routing, one matrix stage was about 7.6% slower. Engineering teams should track upstream review and compare end-to-end latency and output correctness with the model, quantization method, and concurrency conditions held constant. Current evidence is insufficient to generalize the results to other GPUs or expert-parallel sizes. [Validation limitations](https://github.com/flashinfer-ai/flashinfer/pull/5560)