Transformers Proposes FlashAttention Fix to Restore Packed-Sequence Isolation in Ministral3 and Mistral4

On November 6, the Transformers community submitted PR #50450 to fix parameter passing for unpadded packed sequences in Ministral3 and Mistral4 when using FlashAttention. According to the author, the attention modules in both models receive position_ids but do not pass them on to the attention function. As a result, the backend cannot identify sequence boundaries, potentially allowing attention to cross between documents. At the time of review, the proposal was still open. Proposed fix

The issue involves an efficiency optimization commonly used for fine-tuning. In a typical batch, short samples are padded to the same length. Unpadded processing instead concatenates samples, reducing computation spent on unnecessary tokens. Hugging Face's technical article explains that this FlashAttention 2 path relies on position information to preserve sample boundaries and uses variable-length attention to process each sequence. If boundary information is not passed through, concatenated data may be treated as content that can attend across samples. Official technical explanation

The author says the problem occurs because, after explicitly receiving the position parameter to scale queries, the model does not forward it to attention_interface. The proposal adds this forwarding and a test to check that the attention function receives the parameter. The author also notes that an existing unpadding test stopped catching the issue after an earlier change. The public page still shows that maintainer approval is pending before CI can run, so the official package cannot yet be considered fixed. Issue analysis and validation status

For engineering teams, the key concern is the independence of training samples. Based on the mechanism described above, if samples that should be isolated provide context to one another, loss values and convergence may differ from expectations. The proposal does not yet include a complete comparison of training quality or define all affected versions. TRL documentation also lists packing and unpadding as distinct memory-optimization techniques; the conditions that trigger this issue still need to be checked against the data collator and attention settings. TRL documentation

Teams fine-tuning the affected models can first compare outputs from running samples individually versus concatenated, checking whether one document affects the next. They should then monitor maintainer review, actual FlashAttention regression results, and the release that includes the fix. Current evidence is insufficient to quantify the impact on existing training jobs.