GitHub Repo
Transformers Community Proposes Repetition Penalty Fix to Avoid Harming EOS with Left Padding
A Transformers patch proposal submitted on November 1 says generation may include masked padding tokens in repetition penalties, affecting end-of-sequence token scores. The author has proposed passing the attention mask and reported test results, but the PR had not been merged or confirmed through maintainer review as of the time of checking.
The Transformers community submitted PR #50215 on November 1, proposing a fix for repetition penalties in text generation. The author says RepetitionPenaltyLogitsProcessor may include padding tokens excluded by the attention mask in its history. When the pad_token_id used for left padding is the same as the eos_token_id, EOS generation scores may be penalized unnecessarily. The proposal remained open as of the time of checking. Patch proposal
This use case is not uncommon. The official tutorial recommends left padding when sending batches of prompts with different lengths to a generation model, and its example sets the padding token to EOS. The documentation also explains that a repetition_penalty greater than 1 can discourage repetition, while EOS triggers generation to stop. Therefore, if padding positions affect EOS scores, stopping time and output length are engineering considerations worth checking. This is a risk inferred from the mechanism, not a result of published large-scale quality measurements. Official generation tutorial
The proposal would let the penalty processor accept an optional attention_mask, exclude masked positions, and pass the mask through the sampling and beam search flows. The test design covers EOS tokens that genuinely appear in the input, different padding lengths, and multiple returned sequences. This is intended to ensure the fix does not address padding while also ignoring valid historical tokens. The author reports that 53 tests for the relevant processors passed, along with 3 generation integration tests. Changes and test details
For teams operating batched inference services, the proposal offers a practical validation approach: hold the model, prompt, and decoding parameters constant, then compare EOS scores and stopping steps for a prompt run on its own versus in a batch with a longer prompt, both with and without repetition penalties. Next, watch for whether maintainers accept the mask-passing approach and for the official CI results. The public page still showed CI awaiting maintainer approval, so the author’s reported tests should not be treated as project validation or as confirmation that the issue is fixed in a release. Proposal status