訓練系統
SLAI T-Rex Achieves 34.22% MFU in Full-Parameter Post-Training of a Trillion-Scale MoE on Ascend SuperPOD
The research team redesigned parallelization strategies, communication scheduling, kernels, and operator fusion for the DeepSeek-V4 family, bringing Ascend training efficiency to 2.93× the open-source baseline. The system was also used to train an operations research model on solver-verified data, though more complete public reproduction of the code, data, and hardware measurements is still needed.
The SLAI T-Rex technical report focuses on a difficult systems problem: how to perform full-parameter continual pre-training and supervised fine-tuning of trillion-parameter DeepSeek-V4 MoE models on the Ascend CloudMatrix384 SuperPOD. Such workloads are constrained not only by the memory pressure from model weights and optimizer states, but also by cross-node communication introduced by expert parallelism, sparse attention kernels, and numerous fragmented operators that can leave NPUs idle.
The team applied optimizations at three levels. At the model level, it reorganized data, tensor, pipeline, and expert parallelism while coordinating computation and communication. At the kernel level, it used AuraKernel together with profiling to rewrite sparse attention, normalization, and other time-consuming kernels. At the execution level, it fused sequences of small eager operators into AscendC kernels, reducing launch overhead, intermediate tensors, and global memory traffic. The report states that the final model achieved a Model FLOPs Utilization (MFU) of 34.22%, a 2.93× improvement over the open-source baseline recipe, while maintaining training stability.
The researchers also applied this infrastructure to post-training for operations research. The data pipeline used a solver to verify synthetic mathematical programming documents, creating 10,000 supervised fine-tuning (SFT) examples spanning four task categories and three problem representations. In the team’s evaluation, a specialized model based on DeepSeek-V4-Flash achieved an average zero-shot Pass@1 of 71.81%, outperforming GPT-5.4-Mini by 3.98 percentage points and the untuned base model by 11.27 points.
This work is significant because it addresses both a non-CUDA large-scale training stack and verifiable domain-specific data, rather than presenting only a microbenchmark for a single kernel. However, MFU depends heavily on the model configuration, numerical precision, sequence length, and cluster topology. The comparison with GPT-5.4-Mini also cannot, by itself, demonstrate broad reasoning capability. The engineering community should next examine whether the kernels and training recipes are fully released, whether the results can be reproduced across different Ascend clusters, and whether solver verification truly rules out answers that are correctly formatted but mathematically mis-modeled.