GitHub Repo
Transformers Community Reports BatchNorm Momentum Semantic Mismatch, Some Vision Models’ Fine-Tuning Susceptible to Final Batch
On November 8, the Transformers community reported that some vision models pass Keras-convention BatchNorm momentum values directly to PyTorch, causing running statistics to overweight the latest batch. The report includes reproduction code and fine-tuning experiments, but the proposed fix remains under discussion and the scope of impact still needs verification.
A new issue report in Transformers says BatchNorm settings in EfficientNet, ALIGN, MobileNetV1, and MobileNetV2 may have a cross-framework semantic mismatch: a momentum value originally meaning “how much historical statistics to retain” is passed to PyTorch as “how much of the new batch statistics to use.” The report was filed on November 8 and remained open at the time of review, with no fix yet accepted. Issue report
The key is the update direction of the same-named parameter. Keras uses “old value multiplied by m, plus batch value multiplied by 1−m”; PyTorch uses “old value multiplied by 1−m, plus batch value multiplied by m.” So 0.99 means slow updates in the former, but makes the latest batch account for 99% in the latter. The difference is confirmed directly in the two frameworks’ documentation. Keras documentation, PyTorch documentation
This can affect the transition between training and evaluation. During training, BatchNorm uses statistics from the current batch; by default, during evaluation it switches to accumulated means and variances. If those accumulated values mostly reflect the final batch, a smaller or distribution-shifted tail batch may change evaluation outputs. This also explains why the issue may not show up as abnormal training loss. PyTorch documentation
The reporter ran short fine-tuning experiments on CIFAR-10 and observed a marked accuracy drop in two tested models when the tail batch shrank from 32 images to 3. However, changing momentum to 1−m did not improve accuracy in every short-run setting; slower updates to historical statistics can also affect how quickly a pretrained model adapts to new data. These results come from the reporter’s experiments and cannot be generalized to all checkpoints. Experiments and reproduction code
As an engineering step, check the actual BatchNorm momentum, tail batch size, and whether evaluation uses running statistics. Next, watch how maintainers define configuration compatibility and seek validation across datasets and training durations; this report should not yet be treated as a released fix.