GitHub Repo
Transformers report says Gemma 2 attention soft cap is skipped; switching backends may change model computation
A report filed on October 21 says Gemma 2 may silently skip its attention-logit soft cap when using default SDPA or some Flash Attention kernels. It includes numerical comparisons and reproduction code, but no official fix version has been confirmed, and the findings do not establish quality changes across all tasks.
Hugging Face Transformers received a report on October 21 about inference reproducibility: Gemma 2’s attention soft cap may be skipped when switching attention backends, without any runtime warning. The reported environment used Transformers 5.20.2, PyTorch 2.13.0, and a single H100, and the report includes reproduction code. Issue report
Gemma 2 applies the transformation 50 × tanh(x / 50) to compress larger attention scores before subsequent computation. The official configuration documentation lists 50 as the default value for attn_logit_softcapping. There is also a separate soft cap for the final output logits; the report concerns the former, and the two should be checked separately. Gemma 2 documentation
According to the reporter’s analysis, eager and flex_attention apply this transformation, but sdpa, the default for Gemma 2, ignores the parameter it receives. The Flash Attention path depends on whether the kernel accepts softcap; the report specifically names ROCm’s kernels-community/aiter-flash-attn, which does not accept this parameter. The issue therefore should not be generalized to all Flash Attention kernels. Backend analysis
The report compares Gemma 2 2B in FP32 on the WikiText validation set. For a single 2,048-token segment, the loss difference was small, but the maximum absolute difference between eager and SDPA output logits was about 2.443. With the soft cap disabled, the eager and SDPA outputs had zero difference at the displayed precision. This supports the conclusion that the backend skips the operation, but it is not enough to infer widespread degradation on downstream tasks. Reproduction results
For engineering teams, this means that when switching to an accelerated backend, they should check that model computations are preserved, alongside throughput and memory use. Evaluation records should capture the backend, kernel, and version, and teams should compare numerical outputs on fixed inputs. The next step is to track whether upstream adds capability checks, precise warnings, and regression tests. The issue remains open, and the official fix version has yet to be confirmed.