vLLM Community Reports CPU Sampling Bias; Fixed Random Table May Make Some Tokens Unselectable

On October 2, the vLLM community reported a CPU inference correctness issue: the V1 sampler uses a fixed random table that may cause generated results to deviate from the probability distribution output by the model. The report includes reproduction code for both the core layer and the engine layer, with tests covering Linux x86_64 and Apple M1 Pro. The evidence currently comes from a specified commit on the main branch, so it cannot be taken to mean that all released versions are affected. Issue report

The author says fused_gumbel_argmax gets perturbations from a fixed Gumbel table containing 2²⁰ elements. Each time, it uses a seed to determine a starting point, then takes consecutive values according to token index. This limits the random combinations for an entire vocabulary row to finite windows of the same table. The author enumerated all windows and, using a manually configured Zipf distribution over 151,936 tokens, found that 79,078 tokens with nonzero theoretical probability never won. Together, they account for about 3.75% of the original distribution’s probability mass. Test method and results

The engine-level test used a small Llama with random weights. A custom logits processor fixed the distribution at each step, and the test then issued 20,000 single-token requests. V1 did not select any of the tokens mentioned above, while Model Runner V2 selected them 768 times, close to the theoretical expectation of 749. This is an artificial experiment designed to isolate sampler behavior; it does not show that half the vocabulary in real models is permanently unavailable, or that generation quality declines by a fixed amount. End-to-end comparison

The official design document provides context on the two paths: Model Runner V2 uses a Triton Gumbel kernel that generates stateless random numbers inside the kernel based on a seed, and avoids explicitly constructing the full softmax. The official configuration code also requires Triton for V2, so deployers should first confirm platform support and the actual execution path; switching to V2 should not be treated as a universal solution. Design document, configuration validation

For engineering teams, this report is a reminder that sampling tests should cover real vocabulary sizes and tail probabilities, rather than checking only whether outputs are fluent. The author notes that existing chi-squared tests capped the vocabulary at 100 tokens and did not catch this case. Areas to watch include whether large-vocabulary statistical tests are added upstream, how random number generation is adjusted, and the correctness and performance of any fix across CPU platforms. Maintainers still need to confirm the full scope of the impact. Testing blind spots and potential fixes