GitHub Repo
Transformers Community Reports Qwen4-Exp Attention Anomaly; Left-Padded Batches May Produce NaNs
A Transformers community report dated October 23 says that with Qwen4-Exp and eager attention, outputs for valid tokens may contain NaNs in left-padded batches. A small CPU test did not show the same issue with the SDPA path, but the impact on official weights and GPUs remains unverified.
A new technical report in Transformers points to a risk of numerical contamination with Qwen4-Exp under a specific attention backend. On October 23, the reporter provided a reproduction: when a batch contains both left-padded and unpadded sequences and attn_implementation="eager" is set, logits at valid positions in the left-padded sequence contain NaNs, while the unpadded sequence remains normal. With the same weights and inputs using SDPA, values remain finite. Issue report
This condition overlaps directly with the official recommendations. The Qwen4-Exp documentation calls for left padding during batched generation and lists support for eager and SDPA. Its architecture combines Qwen Sparse Attention (QSA) with GatedDeltaNet: QSA selects relevant token blocks, while GatedDeltaNet handles linear attention. Whether anomalies at masked positions can be isolated by subsequent layers is therefore an important implementation boundary. Official architecture and usage documentation
According to the reporter’s investigation, adding the QSA selection mask to the causal/padding mask can cause two minimum finite floating-point values to overflow to negative infinity. If an entire padded row has no visible keys, softmax produces NaNs. Even when GatedDeltaNet later multiplies by a zero mask, it cannot remove the NaNs and may propagate the anomaly to valid positions. This is the reporter’s proposed root-cause analysis and has not yet been confirmed by maintainers. Analysis and reproduction
The test used Transformers 5.19.0, CPU, FP32, and a small randomly initialized model, without loading pretrained weights. The reporter proposed a patch that replaces mask addition with overwriting blocked positions. A local test restored finite outputs, but this should not be considered an official fix. Test scope and proposed patch
For engineering teams, this report suggests including batches of varying sequence lengths, checks for finite values at valid positions, and attention-backend comparisons in validation. SDPA can serve as a comparison baseline for now; whether it is suitable for production deployments still needs to be retested with the weights, precision, and hardware used in each environment. Teams should track maintainer confirmation, regression tests, and a fix release, and should not interpret this result as evidence that all Qwen4-Exp inference paths are failing.