Back Home

GitHub Repo

Transformers Maintainers Reproduce T5Tokenizer Compatibility Issue: Unknown Pieces May Be Encoded as the End-of-Sequence Token

A November 16 report says Transformers v5 can encode unknown pieces as the end-of-sequence token when loading certain SentencePiece vocabularies. A maintainer confirmed reproduction the next day, but no official fix was available as of the time of review.

zh-Hant

A tokenizer compatibility issue in Transformers has been reproduced by a maintainer. A report submitted on November 16 said that T5Tokenizer in v5 may output an incorrect token ID when loading a SentencePiece model whose unknown-piece ID differs from that of the original T5. On the 17th, project member Rocketknight1 said they had reproduced the issue locally. Issue and maintainer response

The case uses rinna/japanese-gpt2-small. Its official configuration does specify T5Tokenizer and defines <unk> and </s> separately; the model card also says its vocabulary was trained with SentencePiece. The issue therefore concerns loading compatibility for existing models that reuse this tokenizer class, and cannot be assessed solely by whether the model name contains “T5.” Official tokenizer configuration, model card

In the report’s minimal example, the input “ABCの本を読んだ。” contains the unknown piece “AB,” which should map to ID 0 but instead becomes ID 2, the end-of-sequence token in that vocabulary. The encoding remains incorrect even though querying unk_token_id returns the correct value, 0. The reporter reproduced the issue in versions 5.0.0, 5.22.1, and 5.23.0, and said that version 4.57.6 produces the correct result. Reproduction code and output

The report attributes the cause to a hard-coded unk_id=2 when creating the Unigram backend. This matches the ordering in the original T5 vocabulary, but does not apply to every model that shares the tokenizer class. The reported workaround rebuilds the backend using the original SentencePiece pieces, scores, and unknown-piece ID. The reporter said the result matches the older version, though it has only been verified for this case so far. Cause analysis and workaround

For engineering teams, this shows that after an upgrade, confirming only that a model loads and its special-token attributes are correct is not enough to validate input semantics. Based on the mechanism described above, an incorrect ID may change the sequence received by the model, but the available evidence does not quantify any effect on generation quality and does not establish that all requests will stop early. Teams should track a fix that obtains the unknown-piece ID from the vocabulary, along with regression tests for nonstandard vocabularies. For now, deployments can compare native SentencePiece encoding with the actual encoding before and after the upgrade. Current status

Sources

  1. T5Tokenizer hardcodes unk_id=2 in v5 — Issue #50851
  2. rinna/japanese-gpt2-small 官方分詞器設定
  3. rinna/japanese-gpt2-small 模型卡