Baidu Releases ERNIE 5.5 Flash Base Weights, Using Hybrid Compressed Attention to Reduce Long-Context Costs

On November 5, Baidu released ERNIE-5.5-Flash-Base weights in FP8 and BF16. Trained from scratch on 33.5T tokens, this text base has not undergone mid-training or post-training, making it suitable for research on domain adaptation, instruction fine-tuning, and reinforcement learning. The official materials distinguish between a maximum pretraining length of 128K tokens and support for 1M tokens during fine-tuning and inference. Evaluations should treat training length and actual long-context capability separately. Official model card

The architecture’s key idea is to reallocate the cache budget for long contexts. Of its 43 layers, 37 use HCA, compressing every 128 tokens into one cache entry while retaining a recent, uncompressed window. The other six layers use MLA to preserve per-token historical representations, then use a DSA indexer to select attention positions. This keeps cache costs low in most layers, while a few layers handle detailed retrieval. However, descriptions of core attention compute cannot be taken to mean that the overall inference service maintains fixed costs at one million tokens. Architecture description

In the MoE component, hidden representations are projected from 4096 dimensions into a 2048-dimensional latent space before being sent to experts. Each layer has 512 routed experts, selects 10 per token, and also includes one shared expert; multiple independently initialized routers jointly score the experts. Baidu says the latent-space design can reduce expert computation and dispatch communication. However, 14B active parameters per token does not mean the full model’s weights require storage for only 14B parameters. Technical description

For deployment, the model card provides examples using the completions interfaces in SGLang and vLLM; inputs must retain the BOS token. Engineering teams can first validate loading, long-context retrieval, and throughput with their target framework, then decide on a post-training workflow. The weights are under a custom license, so users should also check the terms of use. The official benchmarks use standardized prompts and few-shot settings, but remain vendor-reported results. Independent reproduction, long-context quality, and differences between FP8 and BF16 under real workloads are worth watching next. Deployment and evaluation details