GitHub Repo
Transformers Community Proposes MoE Compilation Optimization, Cutting Post-Warmup Training Time by 10–24% in Three Tests
A Transformers MoE optimization proposal moved to review on October 29. It aims to eliminate repeated compilation caused by collecting router outputs. The author reports faster training in three tests, but the patch has not been merged and the performance results await independent verification.
The Transformers community is reviewing an optimization for training mixture-of-experts (MoE) models. Contributor Titus-von-Koeller submitted PR #50102 on October 28, and it moved to review on October 29. The author reports that, after warm-up, training time fell by about 10–24% in three tests using regional compilation. As of the time of this check, the proposal had not been merged. Proposal and test summary
Regional compilation works by compiling recurring blocks in a model so other identical blocks can reuse the computation graph. PyTorch’s official documentation explains that this narrows the compilation scope and reduces cold-start costs. However, some operations remain outside the compiled regions, and there is a tradeoff in optimization opportunities compared with compiling the whole model. PyTorch documentation
The author found that each decoder layer appends router outputs to a shared list. As the list grows with the number of layers, the compiler repeatedly generates new graphs, eventually reaching its recompilation limit and falling back to eager execution. The proposal makes each layer’s collection work independent of how many outputs have already accumulated, then combines the results outside the compiled regions. The author says this preserves output order and gradients, including those needed for auxiliary loss. Technical explanation
The three comparisons were Qwen3-30B-A3B on a single B300, Granite on a single RTX 4090, and Qwen3 on eight H100 GPUs with FSDP2 and checkpointing. After warm-up, training time fell by about 24%, 10%, and 21%, respectively. Including warm-up, the overall reductions were about 11%, 13%, and 20%. The eight-GPU result used an earlier prototype, so it should not be treated as full validation of the current patch. Author’s test table
For training engineers, this points to a bottleneck worth checking: after enabling compilation, verify that recurring blocks continue to reuse computation graphs. Next, watch for maintainer review and independent reproductions, and track recompilation counts, total job time, and loss curves in your own setup. The current data is not enough to conclude that all MoE models will see similar gains.