GitHub Repo
Transformers Community Reports Speculative Sampling Boundary Bug: Zero-Probability Tokens May Pass Validation
A Transformers community report filed October 23 says speculative sampling may accept a candidate token with zero target probability when the validation random value is exactly zero. The report includes a CPU comparison test and a proposed change, but the real-world generation rate, impact on production models, and fix version remain unconfirmed.
Hugging Face Transformers has a report about a boundary condition affecting speculative sampling correctness. An issue filed October 23 says the internal function _speculative_sampling uses r_i <= probability_ratio to decide whether to accept a candidate token. When the target probability is zero, the draft probability is one, and the validation random value is also zero, a candidate that should have been excluded can still pass. The test used a CPU and development-version source code, so it does not establish that every stable release is affected. Issue and reproduction notes
Speculative decoding has a smaller assistant model propose several tokens, which the main model then verifies in a single forward pass, reducing costly step-by-step inference. The official documentation describes standard speculative decoding as preserving the target output distribution, so the acceptance test’s boundary behavior is also a correctness requirement. This has a different technical meaning from the separately documented option that deliberately mixes the main-model and draft distributions to increase the acceptance rate. Official decoding documentation
The reporter tested both the internal function directly and the public generate() interface with two small, randomly initialized Llama models. When only the validation random value was fixed at zero, the test accepted a token excluded by the target distribution; changing the comparison to a strict less-than returned the result to the control output. These experiments did not download pretrained weights or measure effects on real workloads. Experiment setup
A technical point to track is whether zero probabilities after sampling filters are still strictly respected by the verifier. The official documentation shows sampling usage with assistant_model and do_sample=True. Engineering teams using this path can include zero-probability candidates and random-value endpoints in regression checks, and test the acceptance and rejection branches separately. Official usage
As of the time of review, the issue remained open, with no maintainer confirmation or linked fix visible on the page. Although the report records finding a zero in a real random-number sequence, that value came from replaying a selected state and cannot be treated as the rate at which it occurs naturally. Follow-up should track an official fix, affected versions, and test results on pretrained models and different devices. Current status and limitations