baidu/ERNIE-5.5-Flash

Deployment

Pick your hardware, checkpoint and strategy in the builder above to generate the launch command. Every recipe runs on the recommended (smallest) GPU count with vLLM's defaults and the 128K training context (--max-model-len 131072); size --max-num-seqs and --max-model-len to your own workload.

• Low-Latency. ECHO speculative decoding through vLLM's MTP path (2 draft steps; DSpark for ERNIE-5.5-Flash). Fastest replies for a single user or light load.
• High-Throughput. Attention DP without a draft. Most tokens per second across many concurrent requests.
• PD Cluster. One prefill and one decode GB200 tray, KV over Mooncake RDMA.

All recipes run attention data parallel and the MoE expert parallel (--data-parallel-size N --enable-expert-parallel).

1. Model introduction

ERNIE-5.5-Flash is an efficient sparse Mixture-of-Experts language model with 280B total and 14B activated parameters per token, pre-trained from scratch on 33.5T tokens with a training context of up to 128K. It comes as two models: ERNIE-5.5-Flash-Base (FP8 and BF16), taken at the end of pre-training before any mid- or post-training, a starting point for domain adaptation, instruction tuning and reinforcement-learning research, served through the completions endpoint; and ERNIE-5.5-Flash (FP8 / FP4 / BF16, coming soon), with reasoning, tool calling and a bundled DSpark draft.

Its 43 layers are 1 dense FFN layer plus 42 Multi-Router Latent MoE layers. Attention is hybrid: 37 layers use Heavily Compressed Attention (HCA, one cached entry per 128 tokens plus a 128-token sliding window, 512-dim heads) and one in seven, 6 layers, keep full-context access through MLA with DSA sparse attention (256-dim heads). Every attention layer adds a low-rank output gate, Virtual Head Attention (a low-rank post-mix across heads) and QK-norm. Each MoE layer has 512 routed experts (top-10) plus 1 shared expert in a 2048-d latent space, selected by several sigmoid routers. Manifold-constrained hyper-connections (expansion rate 4) replace the residual stream, a MultiMax head produces the logits, and ECHO reuses the final layer as a one-step speculative draft, which vLLM runs through its MTP path.

All checkpoints share the architecture above: 280B total / 14B activated parameters.

Total parameters include the input embedding and the output head; activated parameters count the output head but not the embedding lookup. Both exclude the 0.36B ECHO parameters. The model was trained with a context of up to 128K tokens (131,072); fine-tuning and inference support up to 1M (1,048,576); pass --max-model-len 1048576 to use it (80 GB GPUs such as H100 reach 262,144).

The FP8 checkpoint uses 128x128 blocks with UE8M0 scales. It keeps the embedding, the output head, the MoE routers, the HCA compressor, the indexer, the VHA post-mix in BF16, and the mHC parameters in FP32; every other linear layer and the routed experts are FP8.

Sampling Defaults. temperature=1.0, top_p=0.95 (from the checkpoint's generation_config.json), with do_sample=false for ERNIE-5.5-Flash-Base and do_sample=true for ERNIE-5.5-Flash. vLLM reads temperature and top_p from it but not do_sample, so both models sample at temperature=1.0 by default; pass temperature=0 for greedy decoding.

License. ERNIE 5.5 Flash Base Model License Agreement v1.0 (the LICENSE file in the checkpoint).

vLLM serves the model through the out-of-tree vLLM-ERNIE plugin on the released vLLM 0.32.0 wheel, without modifying vLLM. Set VLLM_USE_MODELSCOPE=True to download from ModelScope; the repo names are the same on both hubs.

2. Configuration tips

• Parallelism. Data + Expert Parallel holds the most KV cache and gives the highest throughput. Tensor + Expert Parallel (TP 2, 4 or 8) lowers latency at low concurrency but holds less KV cache, since the single KV head is replicated on every TP rank.
• MoE Kernel. The FP8 recipes on Blackwell pass --moe-backend humming, which supports SiTU with FP8 weights; on Hopper they keep vLLM's default.
• Long Context. Every recipe passes --max-model-len 131072, the 128K training length. The checkpoint allows 1,048,576: on Blackwell pass --max-model-len 1048576, lower --max-num-seqs so each rank's KV cache holds whole 1M-token requests, and pass --gpu-memory-utilization 0.85, since a 1M-token prefill needs tens of GB of temporary memory on top. On 80 GB GPUs 1M does not fit and vLLM fails at startup; H100 reaches 262,144 at --gpu-memory-utilization 0.93.
• KV Cache. Every recipe passes --kv-cache-dtype fp8, which stores the KV cache in FP8 and is what was tested; auto keeps it in BF16.

3. Advanced usage

3.1 Completions (Base)

As a base model, ERNIE-5.5-Flash-Base continues text: send prompts to /v1/completions. It has no chat, reasoning or tool-call training, so chat requests, --reasoning-parser and --tool-call-parser do not apply. For few-shot use, put the examples in the prompt and stop at the end of one answer.

curl http://localhost:8000/v1/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "baidu/ERNIE-5.5-Flash-Base",
    "prompt": "Question: What is 15% of 240?\nAnswer:",
    "max_tokens": 64,
    "temperature": 0,
    "stop": ["\n"]
  }'

The completion text in the response is 36.

3.2 Reasoning (ERNIE-5.5-Flash, coming soon)

ERNIE-5.5-Flash thinks before it answers. Start the server with --reasoning-parser ernie5 (the Reasoning toggle above); thinking then lands in reasoning_content and the answer in content. Thinking is on by default; to turn it off for a request, pass enable_thinking: false through chat_template_kwargs (the chat template reads enable_thinking, not thinking).

# Thinking is on by default.
curl http://localhost:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "baidu/ERNIE-5.5-Flash",
    "messages": [{"role": "user", "content": "What is 15% of 240?"}]
  }'

# Turn it off for one request.
curl http://localhost:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "baidu/ERNIE-5.5-Flash",
    "messages": [{"role": "user", "content": "What is 15% of 240?"}],
    "chat_template_kwargs": {"enable_thinking": false}
  }'

3.3 Tool calling (ERNIE-5.5-Flash, coming soon)

Start the server with --enable-auto-tool-choice --tool-call-parser ernie5 (the Tool Calling toggle above), together with --reasoning-parser ernie5. Tool calls are returned in message.tool_calls.

3.4 KV cache offload

Long shared prefixes (system prompts, agent histories) can stay outside GPU memory after they leave the GPU cache and be loaded back instead of recomputed. Pick one on the KV Offload row:

• Simple. Host memory per DP rank (SimpleCPUOffloadConnector). cpu_bytes_to_use_per_rank is pinned per rank, so the rank count times this value must stay below the node's RAM. On a GB200 tray (4 ranks) the default 220 GiB per rank is too much; set it to 128 GiB ("cpu_bytes_to_use_per_rank": 137438953472).
• CPU + Filesystem. A small CPU staging tier in front of files under root_dir. Point it at fast local NVMe with plenty of space: ERNIE 5.5's KV blocks took about 280 KB per token on disk in our runs.
• Mooncake (Distributed). Each GPU contributes host memory to a Mooncake pool. Install mooncake-transfer-engine-cuda13 (vLLM 0.32.0 ships CUDA 13), and set MOONCAKE_DEVICE_NAME to the InfiniBand HCAs only, for example mlx5_0,mlx5_1,mlx5_4,mlx5_5 (ibdev2netdev lists them); with the empty default, Ethernet ports are picked too and transfers fail.

3.5 Speculative decoding

ECHO (MTP). The checkpoint carries one ECHO layer after the main stack: the backbone's own final layer, trained to predict future tokens, doubles as the draft. vLLM runs it through its MTP path: pick MTP, 1 / 2 / 3 steps under the Spec Decoding toggle, which sets --speculative-config '{"method":"mtp","num_speculative_tokens":N}'. The layer was trained for one future token and can draft several steps by running again; the Low-Latency preset uses 2 steps. Measured on GSM8K at batch size 1 with greedy decoding, the mean accept length is 1.98 / 2.39 / 2.49 tokens per step for 1 / 2 / 3 draft steps, with unchanged accuracy; 2 steps take most of the gain.

DSpark (ERNIE-5.5-Flash, Coming Soon). The ERNIE-5.5-Flash checkpoint bundles a DSpark draft. Pick the Flash checkpoint above and the DSpark mode: --speculative-config '{"method":"dspark","draft_sample_method":"probabilistic"}'. The draft length comes from the checkpoint's block size, so the config leaves out num_speculative_tokens. Our runs use draft_sample_method: probabilistic; vLLM's default is greedy. The Base checkpoints carry no DSpark draft.

3.6 Prefill / decode disaggregation

Pick PD Cluster as the strategy. The tested setup is one prefill and one decode GB200 tray with MooncakeConnector over RDMA, each DP4 + EP4. Install mooncake-transfer-engine-cuda13 on both trays, and set device_name in --kv-transfer-config to the InfiniBand HCAs only. Start prefill, then decode, then the router (tested with vllm-router==0.1.16), and send requests to the router. With speculative decoding, pass the same --speculative-config to the prefill and decode servers.

References

• vLLM-ERNIE plugin
• Hugging Face
• ModelScope
• Technical report