Baidu’s ERNIE 5.5 Base Targets Serious Model Research

Baidu released a serious research baseline. On November 5, 2026, the company’s ERNIE Team announced ERNIE-5.5-Flash-Base, a text-only mixture-of-experts model with downloadable weights and 280 billion total parameters. Each token activates 14 billion parameters, giving developers a fixed model for further training rather than another polished chatbot demanding applause.

The checkpoint ends at pre-training, before mid-training or post-training. That makes it a common starting point for domain adaptation, instruction tuning and reinforcement-learning research, while also giving teams a fixed baseline for controlled comparisons of training methods. Baidu said a post-trained ERNIE 5.5 Flash model will follow this base version.

A large model built around selective computation

The model card lists 43 layers, a hidden size of 4,096 and 64 attention heads. Thirty-seven layers use Heavily Compressed Attention, whose cache stores one entry for each 128-token block alongside a short window of recent uncompressed tokens.

Six Multi-head Latent Attention layers preserve token-level access to the full context. They use an indexer to select the highest-scoring earlier positions, and Baidu says this sparse selection keeps per-query core-attention cost constant as context grows.

Every layer except the first uses a mixture-of-experts block with 512 routed experts. Ten experts activate for each token, alongside one shared expert, while the experts operate in a 2,048-dimensional latent space instead of the model’s full hidden width.

Baidu says that narrower expert space cuts expert dispatch and combination traffic, as well as expert FLOPs, by roughly half compared with full-width experts. The architecture also includes four parallel residual streams through manifold-constrained hyper-connections and a MultiMax output head with a learnable piecewise temperature.

Training scale, software tools and evaluation

Pre-training used 33.5 trillion tokens, the Muon optimizer and FP8 mixed precision. The training context expanded in stages to 128K tokens, while fine-tuning and inference support 1 million tokens—because apparently ordinary long context now counts as a modest ambition.

The corpus combines multilingual web data with code and software-development records, including commits, issues, pull requests, reviews, generated tests and CI-verified fixes. Curated long documents, sequences containing semantically similar distractors, and synthetic retrieval and question-answering data provide additional long-context training.

Those choices point the base release toward repository-level reasoning, agentic software engineering and extended inputs. The model is not presented as a finished assistant; its purpose is to give developers a substantial foundation they can adapt, test and compare.

The accompanying FlashMask repository documents attention kernels for Hopper and Blackwell GPUs. Blackwell configurations support head dimensions up to 576 for query and key and 512 for value, while Baidu reports speed gains of up to 175% in the forward pass and 74% in the backward pass over FlashAttention-4 MaskMod on SM100 hardware.

The distributed Muon system also combines dense and mixture-of-experts parameter synchronization into one collective operation. It spreads optimizer state more evenly across nodes without changing the optimizer’s updates, addressing the sort of infrastructure detail that rarely makes launch headlines but often decides whether training remains manageable.

Baidu evaluated ERNIE against the publicly released DeepSeek-V4-Flash-Base, Hy3-preview-Base and MiMo-V2.5-Base checkpoints using the same harness, prompts, few-shot examples and LLM judge for free-form question answering. The selected results in Baidu’s table show where ERNIE 5.5 Flash leads and where it trails, keeping the comparison tied to a shared evaluation setup rather than a collection of convenient demonstrations.

The immediate value of ERNIE-5.5-Flash-Base is therefore its role as a controlled starting point. Downloadable weights, detailed architecture, long-context support and training infrastructure give researchers room to test what happens after pre-training—where the interesting work usually begins and the marketing usually ends.

Stay connected via Google News

Follow us for the latest AI News