Back Home

GitHub Repo

Transformers confirms a Gemma 3 attention compatibility issue: the Flash Attention path drops the image bidirectional mask

On November 13, a Transformers project member confirmed that a specific Flash Attention path for Gemma 3 ignores bidirectional attention for images without notifying users. Maintainers support disabling the support flag for the affected multimodal classes, but the timeline for a formal release remains unclear.

Marcus Qwertyus · Public domain · Image source
zh-Hant

On November 13, Transformers received a report about multimodal inference correctness: when Gemma 3 uses kernels-community/flash-attn2, the bidirectional attention that image tokens should retain is reduced to causal attention, with no warning during execution. Project members agreed in the discussion that the relevant support flag should be disabled. As of this review, the issue remains open, so it cannot yet be claimed that a formal release has fixed it. Issue and maintainer responses

This discrepancy affects the model’s computational semantics. Google’s architecture explanation says that Gemma 3 lets image tokens attend bidirectionally to one another, allowing each position to read other parts of the same image; generated text, by contrast, uses causal attention. Applying causal constraints to the image block would therefore prevent later image information from being read by earlier positions, deviating from the intended attention design. Google’s architecture explanation

The report used Transformers 5.22.1, an H100, and BF16 to run a small test model, comparing SDPA with the Flash Attention path above. The maximum absolute difference between the two sets of logits was 0.1846; after removing token_type_ids from the SDPA input, the difference fell to 0.0098. The author used this to pinpoint the issue: SDPA builds the image mask from this input, while the Flash path retains only the padding mask. These are numerical results from a minimal reproduction and are not enough to estimate any loss in visual question-answering accuracy for a production model. Reproduction code and results

For deployment teams, this means that when switching attention backends, image output correctness should be checked alongside latency and memory use. Hugging Face’s current Gemma 3 documentation demonstrates text and image generation with attn_implementation="sdpa" and provides an attention-mask visualization tool, which can serve as a starting point for comparisons. As a practical step, teams can hold the model, image, and prompt constant, compare outputs across backends, and then run evaluations on their actual tasks. They should also track changes to the support flag, regression tests, and formal release notes to confirm the scope of the fix. Official Gemma 3 integration documentation

Sources

  1. Gemma3 under Flash Attention silently drops the bidirectional attention over image tokens
  2. Gemma explained: What’s new in Gemma 3
  3. Transformers documentation: Gemma 3