Transformers Community Investigates Packed-Training Boundary Gap; Linear-Attention Fallback May Mix in Previous Documents

A document-packing issue in Transformers training prompted discussion of implementation trade-offs on October 12–13. The original report, filed on October 9, said that some linear attention, causal convolution, and Mamba scan operations fall back to PyTorch reference implementations when optional acceleration packages are missing, but discard sequence-boundary parameters such as seq_idx and cu_seqlens. Contributors said that supporting a pure PyTorch path would be too complex and inefficient; the reporter emphasized that the current behavior silently changes results. Discussion

Packed training concatenates multiple samples into a single sequence to reduce the computational cost of padding, but each document must remain independent. The official documentation explains that Gated DeltaNet models such as Qwen3.5, as well as convolutional models, cannot isolate samples just by resetting position_ids: the data collator must provide boundaries, and the corresponding kernels must be used. A reference path without those kernels may mix content from different samples. Official documentation

The reporter compared next-token log probabilities for the same document run alone and after another document, using an H100, a development branch, and Qwen3.5-0.8B. In an environment with the full set of packages, the maximum difference was about 0.0007; when the causal-convolution package was simulated as missing, it rose to 5.7380. These are reproduction results for a specific forward pass and cannot yet be translated into a loss in training quality. Reproduction code and results

Fix proposal #49477 masks convolution-window positions across documents according to sequence IDs, and runs the delta rule and Mamba2 scan one sequence at a time. In the author's tests, the maximum log-probability difference for Qwen3.5-0.8B fell from 6.8 to about 0.000011, but per-sequence processing increases host synchronization costs. Fix proposal

A separate draft, #49474, instead raises an error when a fallback implementation cannot preserve boundaries, prompting users to install the required package. Neither proposal has been merged. Engineering teams can first verify their actual kernel and data-collator configuration and add a check comparing packed results with per-sample results. Next, watch which strategy maintainers adopt and which models and releases the fix covers. Error-handling proposal