Transformers Community Reports Falcon-H1 Cache Reordering Bug: Convolutional and Recurrent States Are Not Reordered with the Batch

On November 7, the Transformers community reported a Falcon-H1 cache consistency issue: after calling DynamicCache.batch_select_indices to swap batch order, the attention keys and values move with the batch, but the convolutional and recurrent states remain in place. The reporter reproduced the issue on CPU with Transformers 5.22.2 and PyTorch 2.9.0. As of this review, the issue remains open, with no related fix listed. Issue report

This discrepancy concerns state management during generation. The official documentation defines batch_select_indices as selecting batch indices from a cache and reorder_cache as reordering a cache for beam search; DynamicCache creates the corresponding cache structure based on the model configuration. For models that hold multiple types of state, all states must preserve the same mapping to their samples. Otherwise, even if tensor shapes remain valid, the contents may no longer correspond to the same input. Generation and cache documentation

The reported test does not require downloading model weights. It creates a two-row cache, marks four types of state with distinct values, and swaps the rows using indices [1, 0]. The result shows that batch selection swaps only the attention states; passing the same indices to reorder_cache produces the expected result for all four types of state. This pinpoints the issue at the cache-operation level, but does not yet establish how much actual text output might differ. Reproduction code and comparison results

Based on these results, engineers who reorder Falcon-H1 caches through this interface in custom decoding or batch-management workflows should check that all states are synchronized and compare the next-step output after reordering with per-item inference results. The available evidence does not show that all generation modes are affected, nor is it sufficient to conclude that beam search has the same bug. Follow-up points include whether a fix covers all hybrid-cache state types, whether tests for slicing and repeated indices are added, and the validation results for full-model and GPU paths.