Transformers Community Reports Float16 Overflow in ModernBERT; Long-Text Mean Pooling May Produce NaNs

On November 1, the Transformers community reported a numerical issue in long-text classification: when ModernBertForSequenceClassification is configured with classifier_pooling="mean" and run in pure float16, it may produce NaN classification scores even when the encoder outputs remain normal. The reported environment used Transformers 5.22.1 and PyTorch 2.15.0, with testing performed on the CPU of an Apple silicon computer. Issue report

The trigger is the order of operations in mean pooling: the hidden states are first multiplied by the attention mask, then summed across the token dimension, and finally divided by the number of valid tokens. The author notes that once the intermediate sum exceeds the representable range of float16, the subsequent division cannot restore a valid value. In the published test cases, mmBERT-small produced NaNs at 2,048 tokens, while ModernBERT-base did so at 6,144 tokens. These lengths were observed with specific synthetic English inputs and should not be treated as fixed thresholds for all text. Reproduction code and output

The report has practical implications for long-document workflows. The official documentation lists ModernBERT as supporting up to 8,192 tokens, with cls and mean as pooling options for classification. This means numerical stability in the pooling stage should be checked separately, even when inputs are within the model's length limit. An engineering implication is that checking only whether encoder outputs are finite may miss anomalies before the classification head. ModernBERT documentation

In the author's comparison, performing the sum in float32 and converting back to the original precision after division restored finite outputs in the test cases. The PyTorch documentation also explicitly states that the dtype argument to torch.sum converts the input before the operation, which can help avoid overflow. This offers a way to adjust pooling precision locally, though performance and output error should still be evaluated before deployment. PyTorch API documentation

As of the report's review, only other participants had expressed interest in taking it on; no maintainer confirmation or linked fix was available. The author did not test CUDA, and the classification head was newly initialized, so the results do not represent accuracy evaluations on existing classification tasks. Follow-up should track formal regression tests and differences between pure half-precision and automatic mixed-precision execution paths. Discussion status