AI 基礎設施／低精度訓練
Transformer Engine 2.19 Adds Rubin and 2D MXFP8 Support, Blocks a cuBLAS Algorithm That Could Silently Produce Incorrect Results
NVIDIA’s low-precision training library adds Rubin SM107a support, two-dimensional MXFP8 weight scaling, and MoE communication paths. The release also disables a cuBLAS 13.7 algorithm that can silently corrupt data in grouped GEMM operations on B300 and Rubin.
NVIDIA Transformer Engine 2.19 extends software support for the next-generation Rubin GPU into the training core: builds compiled with CUDA 13.4 or later can recognize SM107a, while the PyTorch interface adds experimental “directional mixed quantization,” allowing the rowwise and columnwise matrix paths to use different quantizers. This provides greater flexibility than applying a single precision strategy to an entire tensor, particularly for workloads whose forward, backward, or communication stages have different error tolerances.
The more concrete new interface is `MXFP8BlockScaling(enable_2d_quantization=True)`. Standard MXFP8 stores one E8M0 scale for every 32 consecutive values; 2D mode goes further by assigning block scales to weights, with the goal of reducing errors caused by directional quantization. Users must explicitly enable it. Version 2.19 also brings MXFP8 to NCCL-EP’s dispatch forward and combine backward paths, supporting pre-quantized `GroupedLinear` and fused grouped MLP operations. These changes directly target expert communication costs in large-scale MoE training.
The attention path has also changed: cuDNN FP8 fused attention now supports packed THD inputs and context parallelism, and an experimental `GatedDeltaNetAttention` kernel has been added. Some FlashAttention backends can now operate under `torch.compile(fullgraph=True)`. These features do not provide unconditional acceleration: FP8 THD sink-softmax backward requires cuDNN 9.26 or later, and GatedDeltaNet has a known issue in which backward propagation may produce NaNs when `head_dim=32`.
The safety aspects of the upgrade are equally important. NVIDIA has disabled a cuBLAS 13.7 grouped-GEMM algorithm on B300 and Rubin that could cause silent data corruption. Teams validating new hardware should therefore treat version pinning and numerical regression testing as essential. There are also breaking changes to NCCL-EP’s `ep_bootstrap` and `EpBuffer` interfaces and to the output type of `mhc_fused_projection`. NVIDIA has not provided overall model-throughput or convergence benchmarks for version 2.19, so engineering teams must still measure the benefits using their own matrix shapes, communication topologies, and model-quality criteria.