Transformers Community Reports Batch-Size Differences in DiffusionGemma, Same Inputs May Generate Different Text

A new technical report in Transformers points to batch size as a factor in a DiffusionGemma reproducibility issue. On October 22, the reporter said that in a controlled experiment with fixed logical inputs and the same content copied across one to four rows, 22 of 24 generations changed their final text and 19 changed their stopping position. The tests used Transformers 5.15.0, BF16, and H100, with additional experiments on A100; all of these results currently come from the reporter. Source: GitHub issue

The behavior is related to the model’s generation process. According to the official documentation, DiffusionGemma first uses an encoder to build a KV cache for the prompt, then repeatedly denoises an entire token block with a decoder. The output logits at each step also serve as a self-conditioning signal for the next step. So, in addition to checking numerical differences in a single forward pass, it is necessary to observe the text and stopping behavior across the full set of iterations. Source: model documentation

On H100, the reporter first observed differences in the MoE routing projection: only 0.051% of routing scores changed, but the effects spread to all block positions within two layers. Switching the routing projection to row-by-row computation made the first denoising round consistent, but later rounds still diverged in the self-conditioning probability-to-embedding matrix multiplication. Only after both operations were computed row by row did a four-row batch match a single row byte for byte across 293 denoising steps. On A100, the first divergence appeared in an attention layer, indicating that the same approach cannot yet be assumed to apply to other hardware. Source: experiments and proposed fix

For deployment teams, this report points to a risk that still needs validation: if a service adjusts batch sizes in response to traffic, the same request may produce different results depending on how it is batched, affecting regression tests and experiment comparisons. Engineers can start by fixing the model revision, software environment, and batch conditions, then comparing single-row and batched results. The issue remains open, with no related fix listed; the throughput cost of the row-by-row approach, its performance on other backends, and whether newer versions are affected also remain to be verified. Source: issue status