AI 研究
Microsoft Publishes MAI-Base-2 Pretraining Details, Using Multi-Round MoE Scheduling to Control Memory Use
On October 15, Microsoft published a technical report on MAI-Base-2, a model with 2.8 trillion parameters and about 80 billion active parameters per token. The report details MoE memory management and training efficiency improvements, but the model remains in post-training, with no public weights or service date announced.
Microsoft AI published its MAI-Base-2 pretraining results on October 15. The mixture-of-experts (MoE) foundation model has 2.8 trillion parameters, with about 80 billion active parameters per token. It was trained for 38 days in August and September this year using 16,000 NVIDIA GB200 GPUs. The model is still in post-training. Microsoft plans to license its weights to trusted partners through Foundry, but has not announced a public service date. Release and timeline
A major engineering change in the report addresses memory waste caused by uneven loads across MoE experts. Experts receive different numbers of tokens. Reserving space for peak loads can leave memory idle much of the time, while setting a capacity limit and dropping overflow tokens can hurt training quality. Microsoft instead uses multi-round scheduling: each expert group has a fixed capacity, excess tokens are handled in later rounds, and busy experts can borrow unused capacity from others in the same group. Smaller capacity can save memory, but may require more computation rounds. Official technical report
The team also grouped expert computation and communication, reduced synchronization between hosts and GPUs, and overlapped communication with computation. Microsoft says these improvements raised model FLOPs utilization in MoE layers from 24% to 35%, making it possible to use MoE in every layer. Average utilization over the full training run was 21%; the two figures cover different measurement scopes. These details show that the practical gains from increased sparsity are still constrained by memory allocation, communication, and kernel launch costs. Engineering details
For evaluation, Microsoft used bits per byte (BPB) to compare predictive loss on retained text, reducing interference from tokenizer differences. Based on scaling curves, it estimates that the FLOPs needed to reach the same loss as DeepSeek-V4-Pro-Base could improve by a factor of 4.2 to 12.3. This is an estimated efficiency gain for the training recipe; it cannot be translated into inference speedups or lower API costs. Some comparison models have also undergone post-training. Engineering teams should next focus on reproducible evaluation, post-training capabilities, and latency and memory requirements at deployment. Evaluation methods and limitations