Back Home

GitHub Repo

Transformers Community Proposes ERNIE 5.5 Adaptation; Full Model Validation and Upstream Integration Still Pending

On November 16, a Transformers contributor reported completing an adaptation for ERNIE 5.5 Flash Base and comparing inference results against a partial set of real weights. The work addresses a current loading compatibility gap but has not yet become officially supported in a release.

zh-Hant

On November 15, the Transformers community opened a request for ERNIE 5.5 Flash Base support, noting that AutoConfig in version 5.23.0 could not recognize the ernie5_5_moe architecture declared by the checkpoint. On the 16th, the proposer said they had completed an implementation that can load BF16 weights directly and published a development branch. A project member said they would check with the ERNIE team about integration plans to avoid duplicate work. As of the time of review, the discussion had not listed an upstream merge request. Source: support discussion

This gap makes it less convenient to bring the model into existing fine-tuning toolchains. Although the weights use the Hugging Face format, running them still requires a corresponding model class. According to the report, the checkpoint provides neither an auto_map nor a model implementation, so it cannot be loaded through the existing Auto-class pathway. Training workflows that depend on Transformers still need an architecture adaptation before teams can assess further integration. Source: issue description

Baidu’s model card describes a text-only base model with 280 billion total parameters and 14 billion parameters activated per token; it has not undergone mid-training or post-training. Its 43-layer backbone combines 37 layers of compressed attention with 6 layers of MLA sparse attention, and uses latent-space MoE. Differences in attention dimensions and cache behavior across layers mean the adaptation involves actual compute paths; adding an architecture-name mapping alone is not sufficient. Source: official model card

Validation remains limited. The contributor said they compared three-layer slices using real weights against vLLM-ERNIE and confirmed consistency in block outputs, expert routing, and cache decoding. These results come from the submitter and do not yet establish correctness for the full model, long contexts, or training gradients. Source: validation report

Teams that need to deploy now can refer to the SGLang and vLLM-ERNIE paths listed in Baidu’s model card. Items to track include the upstream implementation, numerical comparisons using the full weights, and FP8 loading validation. Fine-tuning teams should also independently confirm the license terms and memory requirements; the number of parameters activated per token should not be treated as the storage cost of the full weights. Source: deployment and license information

Sources

  1. Add ERNIE 5.5 — Transformers Issue #50791
  2. ERNIE-5.5-Flash-Base-BF16 官方模型卡