Transformers Community Reports DeBERTa Normalization Regression; Full-Width Characters and Chinese Radicals May Become Unknown Tokens

On October 15, the Transformers community reported a DeBERTa tokenization compatibility issue: the same model weights and input may produce different tokens in versions 5.19.0 and 4.57.6. In the reported examples, full-width Latin letters, circled numbers, and ligatures become [UNK] in the newer version, while the older version converts them into recognizable words. Another participant added that the Kangxi radical “⾊,” which may appear in Chinese PDF extraction, also does not convert to “色” as it does in the reference implementation. Issue and reproduction code

The reporter attributes the cause to the v5 loading path: although SentencePiece’s precompiled character map is passed in, DebertaV2Tokenizer does not add it to the normalization pipeline. The retained NFC normalization cannot replace the model’s original nmt_nfkc rule. Unicode distinguishes canonical equivalence from compatibility equivalence; NFKC handles compatibility differences such as circled forms and character width, while NFC preserves those distinctions. So a character that looks reasonable does not necessarily mean the tokenization input still matches what the model saw during training. Technical analysis, Unicode Normalization Specification

For Chinese document pipelines, this means errors may occur before model inference. An engineering inference is that if a classifier, reranker, or prompt-injection detector uses an affected tokenizer, an increase in unknown tokens could change its decisions. No downstream task measurements are available yet, so the report does not establish that accuracy has declined. The Microsoft model card also shows that DeBERTa-v3 uses a fixed vocabulary and a downstream fine-tuning workflow, making input encoding an important part of deployment validation. Official model card

The reporter proposed restoring the precompiled normalizer and said local tests pass. As of this check, the issue remains open and no official fix has been released. Whether maintainers accept the proposed fix, whether consistency is preserved after saving and reloading, and how tokenization and task scores differ on real-world Chinese corpora are key follow-up questions. Deployers can first compare outputs from the model’s original SentencePiece tokenizer and the current tokenizer, and include full-width letters, circled numbers, and PDF-extracted radicals in upgrade checks. Fix proposal and discussion