Transformers Maintainers Confirm a Sampling Gap in Serve; Positive Temperature May Still Use Greedy Decoding

A gap in how generation parameters are passed through Hugging Face Transformers’ lightweight inference service has surfaced. A report filed on October 19 showed that ten calls to Mistral-7B-Instruct-v0.3 through transformers serve returned exactly the same answer to the same prompt, even with temperature set to 1.5. The reported environment used 5.21.0.dev0 on Apple MPS, with continuous batching disabled; these results cannot be directly generalized to all stable releases and execution backends. Issue report

The key is how the API’s temperature parameter maps to the underlying generation mode. The proposal’s author noted that the service sets do_sample=False when it encounters zero temperature, but does not turn sampling back on for a positive temperature. If do_sample is not enabled in the model’s generation configuration, generation still uses greedy decoding. The official documentation likewise explains that random sampling must be enabled with do_sample=True; temperature alone does not select the generation mode. Generation strategies documentation

A fix proposed the same day would enable sampling for nonzero temperatures and add a configuration-construction test. The author showed that a local modification produced ten different answers across ten calls. Maintainer SunMarc then confirmed that the current defaults can indeed continue using greedy decoding after the temperature is changed, but said parameter handling is more complex than the proposal suggests and that they would rework the approach to align it with other inference engines. At the time of review, the original proposal had not been merged, so the local results do not mean an upstream fix is complete. Proposed fix and maintainer response

This has direct implications for evaluation and agent experiments. The official documentation positions serve for local, self-hosted, evaluation, and medium-load use, and provides compatible chat endpoints. From this, it follows that engineering teams relying on higher temperatures to generate multiple candidate answers may find that this gap prevents the expected exploration across generations, complicating assessments of answer diversity. Serving documentation

Next, watch for a replacement fix, a stable release, and clarification of the precedence between top-level temperature and model generation settings. When validating a deployment, check the sampling flag actually passed in and compare output distributions across multiple prompts; repeating a single prompt alone is not enough to prove that sampling is disabled.