NVIDIA Publishes MoE Training Benchmarks; Best Eight-B200 Configuration Reaches 2.21× Baseline

NVIDIA published its BioNeMo mixture-of-experts (MoE) training workflow and benchmarks on September 24. Running Mixtral-8x7B on eight B200 GPUs, the best configuration achieved 9,050 tokens per second per GPU, approximately 2.21 times the Hugging Face baseline cited in the article. These are official throughput results for a specific training configuration; they do not yet demonstrate a proportional reduction in the time required to reach target quality on biological tasks. [Official benchmarks](https://developer.nvidia.com/blog/efficient-moe-training-for-biological-foundation-models/)

The comparison table also shows why gains from different changes must be distinguished. The baseline uses BF16 and fully sharded data parallelism across eight GPUs, delivering a throughput of 4,096 tokens per second per GPU. Switching to Transformer Engine while retaining the same precision and parallelism settings raises throughput to 4,447, an increase of about 8.6%. The configuration reaching 9,050 also switches to expert parallelism and MXFP8, so the 2.21× result reflects differences across the entire configuration and cannot be attributed solely to kernel fusion. [Comparison conditions](https://developer.nvidia.com/blog/efficient-moe-training-for-biological-foundation-models/)

Although MoE selects only a subset of experts for each token, numerous small kernel launches and exchanges between GPUs can still slow training. BioNeMo’s model implementation processes experts using grouped linear operations, then shards expert weights across GPUs along the expert dimension. Data parallelism and expert parallelism can be combined into a two-dimensional configuration. The default dispatcher exchanges tokens through NCCL, while a more advanced path additionally requires DeepEP and NVLink access between GPUs. [Model documentation](https://docs.nvidia.com/bionemo-recipes/latest/main/recipes/models/mixtral/)

MXFP8 shares one scaling factor across every 32 values to improve the numerical range of the eight-bit representation. Values use four exponent bits and three mantissa bits, while scaling factors are stored in eight bits and represent powers of two. Transposition requires more than rearranging bits: blocks in different orientations must be requantized, so Transformer Engine creates copies in both orientations from the original high-precision input. This also explains why eight-bit computation does not directly halve memory usage across the entire training process. [Precision and transposition details](https://nvidia.github.io/TransformerEngine/examples/fp8_primer.html)

The model documentation also allows precision to be selected layer by layer and provides workflows for converting weight formats and comparing outputs. Researchers can first check results before and after conversion, then progressively enable low-precision computation and cross-GPU features to narrow the scope when investigating numerical discrepancies. [Conversion and validation](https://docs.nvidia.com/bionemo-recipes/latest/main/recipes/models/mixtral/)

The public repository describes these workflows as reference implementations that can be modified. Hardware support also has limits: the support table lists MXFP8 support for compute capabilities 10.0 and 10.3, with 12.0 support still pending. When reproducing the results, engineering teams should pin the commit and package versions, hold sequence length, batch size, and routing settings constant, and track loss curves, validation quality, and the share of runtime spent on communication. Only by measuring both throughput and the number of steps required to reach target quality can teams determine whether low precision and expert sharding reduce total training costs. [Repository and support table](https://github.com/NVIDIA-BioNeMo/bionemo-recipes)