Back Home

GitHub Repo

Transformers EOS Configuration Reportedly Removes BOS, Affecting Llama, Gemma, and Mistral Tokenization

A November 19 report says Transformers 5.23.0 may unexpectedly remove the beginning-of-sequence token when configuring the end-of-sequence token, with the anomaly persisting after saving. Another user reproduced it the following day; as of the time of review, no official fix version had been confirmed.

zh-Hant

Hugging Face Transformers has a new tokenizer compatibility report: for some Llama, Gemma, and Mistral checkpoints loaded through AutoTokenizer, configuring add_eos_token may also remove the existing BOS, or beginning-of-sequence token. The issue was reported on November 19 and covers version 5.23.0 and the main branch at the time. On the 20th, another user reproduced the same behavior with Llama 2. Issue report

The report’s minimal example compares the encoding of “hello”: the older version preserves both the beginning and ending tokens, while the newer version, after EOS is enabled, leaves only the text and ending token. The issue can be triggered either through loading arguments or by changing the attribute afterward. Saving with save_pretrained and reloading does not restore the BOS. The anomaly may therefore persist in tokenizer artifacts and carry into later workflows, rather than being limited to a single run. Reproduction code and output

The reporter points to inconsistent state between two sources: the post-processor in tokenizer.json originally adds BOS, while the Python-side flag is initialized as disabled. When EOS is changed, the code rebuilds the template based on that flag, removing BOS as well. The official documentation lists adding BOS and EOS as separate options and explains that the post-processor handles the beginning and ending tokens, indicating that this issue involves synchronization between the configuration and the actual template. Official tokenizer documentation

The engineering risk is that input sequences may change silently. By inference from this mechanism, fine-tuning data, offline evaluations, and deployment may become incomparable if they use different tokenization results. The report has not quantified any model-quality impact, and the issue should not be generalized to every model in these families. Teams preparing to upgrade can fix a small set of text samples and compare token IDs, beginning and ending tokens, and results before and after saving and reloading.

A fix will also need to address conflicts between model configuration files and existing post-processors; simply prioritizing one source could change default encoding for other checkpoints. Follow-up should check whether the official fix covers the loading-argument, attribute-change, and serialization paths, and whether it includes regression testing across models. Discussion of fix tradeoffs

Sources

  1. v5: add_eos_token drops the BOS token for Llama, Gemma and Mistral tokenizers — Issue #50985
  2. Transformers Llama tokenizer documentation