GitHub Repo
Transformers community reports lost tokenizer normalization, possible extra newline token in WeMM embeddings
A Transformers report on October 13 says a newer tokenizer loading path may ignore the model’s existing normalization settings, adding a trailing newline token to WeMM-Embedding-2B inputs. The report includes a cross-version minimal reproduction, but maintainers have not confirmed it and retrieval quality has not been measured.
On October 13, the Transformers community reported an embedding model compatibility issue: when loading tencent/WeMM-Embedding-2B, a change in tokenizer class may cause the normalization settings in tokenizer.json to stop taking effect. The reporter says this leaves a trailing newline that would previously have been removed, causing the same text to produce a different token sequence after an upgrade. At the time of review, the issue remained open, with no maintainer confirmation. Issue report
The minimal reproduction loads the model with AutoTokenizer.from_pretrained and encodes text containing chat markers with add_special_tokens=False. The author reports five tokens with version 5.17, while version 5.19 adds a trailing token with ID 198; the report identifies 5.18 as the point where the change began and attributes it to the loading class changing from a generic fast tokenizer to Qwen3_5Tokenizer. These version differences are currently results from the author's reproduction and cannot be generalized to all models. Reproduction code and output
The technical risk concerns which token is used to produce the embedding. The reporter says this model reads the last token, so an extra trailing newline changes the embedding. Tencent’s model card says WeMM-Embedding-2B is built on Qwen3.5, supports text, images, video, and visual documents, and outputs L2-normalized 2,048-dimensional vectors; its installation example explicitly pins transformers==5.2.0. The model card does not declare compatibility with newer versions or provide quality tests for this issue. Official model card
For engineering teams maintaining vector search, the potential risk is that if query-side and document-indexing-side preprocessing behave differently, they may produce inconsistent vectors even with identical weights. There is currently no evidence quantifying any recall or ranking loss. Upgrade validation should first compare token IDs, embedding similarity, and retrieval results on a fixed sample before deciding whether to update the index. Follow-up should track whether maintainers can reproduce the issue, how the normalization settings are preserved, and whether a fix covers the save-and-reload workflow.