Transformers Community Proposes Sequence Isolation Fix After Missing Acceleration Packages May Change Model Outputs

On October 9, the Transformers community disclosed a document-packing correctness issue: when some optional acceleration packages are unavailable and execution falls back to PyTorch reference implementations, convolution and recurrent state may cross document boundaries, causing the output for a later document to be affected by preceding text. The report provides a minimal reproduction with Qwen3.5-0.8B, tested against main-branch commit 4cc2aa8 on an H100 with float32. This does not establish that every released version is affected. Issue report

Document packing concatenates multiple samples into a long sequence to reduce padding computation. The official documentation says DataCollatorWithFlattening can output a seq_idx identifying which sequence each token belongs to, along with boundary parameters used by attention operations. It also says to confirm that the model can handle flattened inputs. In other words, after the data collator provides boundary information, the underlying operators still need to enforce isolation semantics. Data collator documentation

The report points to fallback wrappers that discard arguments not declared by the reference function; the relevant functions do not accept seq_idx or cu_seqlens. The test compares the token log probabilities for the second document when packed with those when run separately. The maximum difference was about 0.0007 in a fully accelerated environment, rising to 5.7380 after causal-conv1d was disabled. These figures describe output differences in a specific case; they are not percentages of model quality loss, and there is no evidence that the issue has caused cross-user data leakage. Reproduction and measurements

Proposed fix #49477 masks convolution windows across sequences and runs the gated delta rule and Mamba2 scan separately for each sequence. The author reports that, in a separate Qwen3.5-0.8B test, the maximum log-probability difference fell from 6.8 to about 0.000011. However, per-sequence processing requires host synchronization and carries a performance cost. As of this review, the proposal remains open. Engineering teams should validate their actual environments by comparing packed results with results from running each sample separately, record the optional packages and kernel paths in use, and track the proposal’s merge and release status. Any effects on training loss and evaluation reproducibility remain inferences that require further measurement. Proposed fix