Transformers Proposes Top-H Sampling Fix to Align Entropy Budget and Candidate Filtering

On November 8, the Transformers community submitted PR #50471 to correct how TopHLogitsWarper calculates entropy. The fix responds to an issue reported on November 1: when users generate text with do_sample=True and top_h, the candidate token set selected by the existing implementation may differ from the rule described in the documentation. As of this check, the proposal remains open, so the official package cannot be considered fixed. Proposed fix

Top-H is a decoding method that uses an entropy constraint to determine the sampling range. The research formulates truncation sampling as an entropy-constrained optimization problem and then approximates a solution with a greedy algorithm, aiming to balance diversity and coherence in generation. Because it uses information from the probability distribution to determine which candidates to retain, the parameter’s practical meaning depends on how entropy is calculated. The paper predates this fix; it is not a new model or study published this week. Top-H paper

The issue has two parts: the original implementation calculates the entropy budget using only the 100 highest-probability candidates, ignoring the long tail of the full vocabulary distribution; when determining the retention range, it also uses partial entropy without renormalizing. The reporter provided a reproduction in Transformers 5.22.1 that does not require downloading a model. With the same top_h=0.4 setting, a 50-token Zipf distribution retains 6 tokens, while the reference rule should retain 4; a 49,152-token case retains only 8, compared with a reference value of 25. The direction of the deviation varies with the distribution. Issue and reproduction code

The proposal calculates the budget from the full distribution, then computes renormalized entropy based on the cumulative probability mass of each candidate prefix, while retaining the existing limit of 100 candidates. The author reported 62 passing tests in the relevant test file, covering different distributions, masks, and half precision. For engineering teams, this means the same sampling parameters may produce different results before and after the fix. Evaluations should record the package version and compare candidate sets, output quality, and latency. Current evidence mainly supports consistency with the stated rule; it is not yet enough to claim improvements in Chinese generation or downstream task performance. Test details