# baidu/ERNIE-5.5-Flash — vLLM serving recipe

> Deploy ERNIE-5.5-Flash with vLLM: Baidu's 280B-total / 14B-active Mixture-of-Experts model with hybrid HCA + MLA-DSA attention, ECHO (MTP) speculative decoding and up to 1M context, served through the vLLM-ERNIE plugin. FP8 and BF16 recipes for B300, GB300, B200, GB200 and H100, with KV cache offload and PD disaggregation.

Configuration: hardware `gb200` (4× GB200 NVL4) · strategy `single_node_dep` (Data + Expert Parallel) · variant `default` (Base · FP8) · features `spec_decoding` · preset `gb200_low_latency`

- Interactive page: https://recipes.vllm.ai/baidu/ERNIE-5.5-Flash
- This configuration as JSON: https://recipes.vllm.ai/baidu/ERNIE-5.5-Flash.json?hardware=gb200

The commands below were rendered for exactly this configuration and take precedence over any example in the guide.

## Other configurations

Change the query string (the same parameters as the page URL) and fetch this URL again. Unknown or incompatible values fall back to the recipe default, so check the Configuration line.

- `hardware`: `h100`, `b200`, `gb200`, `b300`, `gb300`
- `variant`: `default`, `bf16`, `flash`
- `strategy`: `single_node_dep`, `single_node_tep`, `multi_node_dep`, `pd_cluster` (multi-node strategies also need `nodes=<n>`)
- `features`: comma-separated list of `spec_decoding`, `reasoning`, `tool_calling` (default: none). Use `features=none` to turn them all off.
- `preset`: a tuned, evidence-backed configuration (`baseline` opts out of a default one). Each sets its own hardware and strategy:
  - `gb200_low_latency` — Low-Latency (gb200 · DEP · default): `hardware=gb200&strategy=single_node_dep&preset=gb200_low_latency`
  - `gb200_high_throughput` — High-Throughput (gb200 · DEP · opt-in): `hardware=gb200&strategy=single_node_dep&preset=gb200_high_throughput`
  - `gb200_bf16_low_latency` — Low-Latency (gb200 · Multi-node DEP · default): `hardware=gb200&variant=bf16&strategy=multi_node_dep&preset=gb200_bf16_low_latency`
  - `gb200_bf16_high_throughput` — High-Throughput (gb200 · Multi-node DEP · opt-in): `hardware=gb200&variant=bf16&strategy=multi_node_dep&preset=gb200_bf16_high_throughput`
  - `b200_low_latency` — Low-Latency (b200 · DEP · default): `hardware=b200&strategy=single_node_dep&preset=b200_low_latency`
  - `b200_high_throughput` — High-Throughput (b200 · DEP · opt-in): `hardware=b200&strategy=single_node_dep&preset=b200_high_throughput`
  - `b200_bf16_low_latency` — Low-Latency (b200 · DEP · default): `hardware=b200&variant=bf16&strategy=single_node_dep&preset=b200_bf16_low_latency`
  - `b200_bf16_high_throughput` — High-Throughput (b200 · DEP · opt-in): `hardware=b200&variant=bf16&strategy=single_node_dep&preset=b200_bf16_high_throughput`
  - `gb300_low_latency` — Low-Latency (gb300 · DEP · default): `hardware=gb300&strategy=single_node_dep&preset=gb300_low_latency`
  - `gb300_high_throughput` — High-Throughput (gb300 · DEP · opt-in): `hardware=gb300&strategy=single_node_dep&preset=gb300_high_throughput`
  - `gb300_bf16_low_latency` — Low-Latency (gb300 · DEP · default): `hardware=gb300&variant=bf16&strategy=single_node_dep&preset=gb300_bf16_low_latency`
  - `gb300_bf16_high_throughput` — High-Throughput (gb300 · DEP · opt-in): `hardware=gb300&variant=bf16&strategy=single_node_dep&preset=gb300_bf16_high_throughput`
  - `b300_low_latency` — Low-Latency (b300 · DEP · default): `hardware=b300&strategy=single_node_dep&preset=b300_low_latency`
  - `b300_high_throughput` — High-Throughput (b300 · DEP · opt-in): `hardware=b300&strategy=single_node_dep&preset=b300_high_throughput`
  - `b300_bf16_low_latency` — Low-Latency (b300 · DEP · default): `hardware=b300&variant=bf16&strategy=single_node_dep&preset=b300_bf16_low_latency`
  - `b300_bf16_high_throughput` — High-Throughput (b300 · DEP · opt-in): `hardware=b300&variant=bf16&strategy=single_node_dep&preset=b300_bf16_high_throughput`
  - `h100_low_latency` — Low-Latency (h100 · DEP · default): `hardware=h100&strategy=single_node_dep&preset=h100_low_latency`
  - `h100_high_throughput` — High-Throughput (h100 · DEP · opt-in): `hardware=h100&strategy=single_node_dep&preset=h100_high_throughput`
  - `h100_bf16_low_latency` — Low-Latency (h100 · Multi-node DEP · default): `hardware=h100&variant=bf16&strategy=multi_node_dep&preset=h100_bf16_low_latency`
  - `h100_bf16_high_throughput` — High-Throughput (h100 · Multi-node DEP · opt-in): `hardware=h100&variant=bf16&strategy=multi_node_dep&preset=h100_bf16_high_throughput`
  - `gb200_pd_mooncake` — 1P1D Mooncake (gb200 · PD · default): `hardware=gb200&strategy=pd_cluster&preset=gb200_pd_mooncake`

## Install

Requires vLLM >= 0.32.0.

ERNIE 5.5 runs through the vLLM-ERNIE plugin, which is pinned to vLLM 0.32.0

```bash
uv venv --python 3.12
source .venv/bin/activate
uv pip install vllm==0.32.0
uv pip install git+https://github.com/baidu/vLLM-ERNIE.git
```

## Serve

```bash
export VLLM_ENGINE_READY_TIMEOUT_S=3600
vllm serve baidu/ERNIE-5.5-Flash-Base \
  --kv-cache-dtype fp8 \
  --max-model-len 131072 \
  --enable-expert-parallel \
  --port 8100 \
  --data-parallel-size 4 \
  --moe-backend humming \
  --speculative-config '{"method":"mtp","num_speculative_tokens":2}'
```

### vLLM Router (start after vLLM)

Routes client requests across the DP ranks of the single engine process. --intra-node-data-parallel-size must match the engine's --data-parallel-size; consistent_hash keeps a conversation pinned to the rank that already holds its prefix. Requires: uv pip install vllm-router

```bash
vllm-router \
  --host 0.0.0.0 \
  --port 8000 \
  --policy consistent_hash \
  --intra-node-data-parallel-size 4 \
  --worker-urls http://localhost:8100 \
  --worker-startup-timeout-secs 1800
```

## Tuned preset: Low-Latency

Speculative decoding on (MTP, 2 draft steps; DSpark for ERNIE-5.5-Flash). Fastest replies for a single user or light load.

- Key: `gb200_low_latency` · applied by default on this hardware and strategy (`preset=baseline` drops it)
- Qualified on: hardware `gb200` · strategy `single_node_dep` (Data + Expert Parallel) · variants `default`, `flash`
- Builder choices it implies (already in the command above): `spec_decoding` on
- Every point rendered as JSON: https://recipes.vllm.ai/baidu/ERNIE-5.5-Flash/presets/gb200_low_latency.json

## Notes for this configuration

### 1M context

The commands use the 128K training length (`--max-model-len 131072`).
For the full 1M, pass `--max-model-len 1048576`: each rank's KV cache
must hold whole 1M-token requests, so lower `--max-num-seqs` to fit, and
a 1M-token prefill needs extra temporary memory: add
`--gpu-memory-utilization 0.85`.

## Known issues

### Architecture Ernie5_5_MoeForCausalLM is not supported

The vLLM-ERNIE plugin is not installed, or `VLLM_PLUGINS` excludes it.
Install it into the same environment as vLLM 0.32.0 and leave
`VLLM_PLUGINS` unset (or include `ernie`).

### Timed out waiting for engine core processes to start

Loading the weights and warming up the kernels takes longer than the
default timeout. Set `VLLM_ENGINE_READY_TIMEOUT_S=3600`.

### Out of memory during a 1M-token prefill

Lower `--gpu-memory-utilization` to 0.85 to leave room for the
temporary prefill buffers, and lower `--max-num-seqs`, or lower
`--max-model-len`.

## Guide

> Background reference. Commands in it are examples and may target another configuration — use the Serve section above.

### Deployment

Pick your hardware, checkpoint and strategy in the builder above to
generate the launch command. Every recipe runs on the recommended
(smallest) GPU count with vLLM's defaults and the 128K training context
(`--max-model-len 131072`); size `--max-num-seqs` and `--max-model-len` to
your own workload.

- **Low-Latency.** ECHO speculative decoding through vLLM's MTP path (2
  draft steps; DSpark for ERNIE-5.5-Flash). Fastest replies for a single
  user or light load.
- **High-Throughput.** Attention DP without a draft. Most tokens per
  second across many concurrent requests.
- **PD Cluster.** One prefill and one decode GB200 tray, KV over Mooncake
  RDMA.

| GPU | FP8 (~280 GB weights) | BF16 (~560 GB weights) |
| :-- | :-- | :-- |
| B300 / GB300 (288 GB) | 2 GPUs | 4 GPUs |
| B200 (192 GB) | 4 GPUs | 8 GPUs |
| GB200 (192 GB) | 4 GPUs | 2 trays × 4 GPUs |
| H100 (80 GB) | 8 GPUs | 2 nodes × 8 GPUs |

All recipes run attention data parallel
and the MoE expert parallel (`--data-parallel-size N --enable-expert-parallel`).

### 1. Model introduction

**ERNIE-5.5-Flash** is an efficient sparse Mixture-of-Experts language
model with **280B total and 14B
activated parameters per token**, pre-trained from scratch on 33.5T tokens
with a training context of up to 128K. It comes as two models:
**ERNIE-5.5-Flash-Base** (FP8 and BF16), taken at the end of pre-training
before any mid- or post-training, a starting point for domain adaptation,
instruction tuning and reinforcement-learning research, served through the
completions endpoint; and **ERNIE-5.5-Flash** (FP8 / FP4 / BF16, coming
soon), with reasoning, tool calling and a bundled DSpark draft.

Its 43 layers are 1 dense FFN layer plus 42 Multi-Router Latent MoE layers.
Attention is hybrid: 37 layers use Heavily Compressed Attention (HCA, one
cached entry per 128 tokens plus a 128-token sliding window, 512-dim heads)
and one in seven, 6 layers, keep full-context access through MLA with DSA
sparse attention (256-dim heads). Every attention layer adds a low-rank
output gate, Virtual Head Attention (a low-rank post-mix across heads) and
QK-norm. Each MoE layer has 512 routed experts (top-10) plus 1 shared expert
in a 2048-d latent space, selected by several sigmoid routers.
Manifold-constrained hyper-connections (expansion rate 4) replace the
residual stream, a MultiMax head produces the logits, and ECHO reuses the
final layer as a one-step speculative draft, which vLLM runs through its
MTP path.

| Model | Weights | Use |
| :-- | :-- | :-- |
| **ERNIE-5.5-Flash-Base** | [FP8](https://huggingface.co/baidu/ERNIE-5.5-Flash-Base) / [BF16](https://huggingface.co/baidu/ERNIE-5.5-Flash-Base-BF16) | Base model, completions. Released |
| **ERNIE-5.5-Flash** | FP8 / FP4 / BF16 | Chat, reasoning, tool calling, DSpark. Coming soon |

All checkpoints share the architecture above: 280B total / 14B activated
parameters.

Total parameters include the input embedding and the output head; activated
parameters count the output head but not the embedding lookup. Both exclude
the 0.36B ECHO parameters. The model was trained with a context of up to
128K tokens (131,072); fine-tuning and inference support up to 1M
(1,048,576); pass `--max-model-len 1048576` to use it (80 GB GPUs such as
H100 reach 262,144).

The FP8 checkpoint uses 128x128 blocks with UE8M0 scales. It keeps the
embedding, the output head, the MoE routers, the HCA compressor, the
indexer, the VHA post-mix in BF16, and the mHC parameters in FP32; every other linear layer and the
routed experts are FP8.

**Sampling Defaults.** `temperature=1.0`, `top_p=0.95` (from the
checkpoint's `generation_config.json`), with `do_sample=false` for
ERNIE-5.5-Flash-Base and `do_sample=true` for ERNIE-5.5-Flash. vLLM reads
`temperature` and `top_p` from it but not `do_sample`, so both models
sample at `temperature=1.0` by default; pass `temperature=0` for greedy
decoding.

**License.** ERNIE 5.5 Flash Base Model License Agreement v1.0 (the
`LICENSE` file in the checkpoint).

vLLM serves the model through the out-of-tree
[vLLM-ERNIE](https://github.com/baidu/vLLM-ERNIE) plugin on the released
vLLM 0.32.0 wheel, without modifying vLLM. Set `VLLM_USE_MODELSCOPE=True`
to download from ModelScope; the repo names are the same on both hubs.

### 2. Configuration tips

- **Parallelism.** Data + Expert Parallel holds the most KV cache and gives
  the highest throughput. Tensor + Expert Parallel (TP 2, 4 or 8) lowers
  latency at low concurrency but holds less KV cache, since the single KV
  head is replicated on every TP rank.
- **MoE Kernel.** The FP8 recipes on Blackwell pass `--moe-backend humming`,
  which supports SiTU with FP8 weights; on Hopper they keep vLLM's default.
- **Long Context.** Every recipe passes `--max-model-len 131072`, the
  128K training length. The checkpoint allows 1,048,576: on Blackwell pass
  `--max-model-len 1048576`, lower `--max-num-seqs` so each rank's KV
  cache holds whole 1M-token requests, and pass
  `--gpu-memory-utilization 0.85`, since a 1M-token prefill needs tens of
  GB of temporary memory on top. On 80 GB GPUs 1M does not fit and vLLM
  fails at startup; H100 reaches 262,144 at `--gpu-memory-utilization 0.93`.
- **KV Cache.** Every recipe passes `--kv-cache-dtype fp8`, which stores
  the KV cache in FP8 and is what was tested; `auto` keeps it in BF16.

### 3. Advanced usage

#### 3.1 Completions (Base)

As a base model, ERNIE-5.5-Flash-Base continues text: send prompts to
`/v1/completions`. It has no chat, reasoning or tool-call training, so chat
requests, `--reasoning-parser` and `--tool-call-parser` do not apply. For
few-shot use, put the examples in the prompt and stop at the end of one
answer.

```bash
curl http://localhost:8000/v1/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "baidu/ERNIE-5.5-Flash-Base",
    "prompt": "Question: What is 15% of 240?\nAnswer:",
    "max_tokens": 64,
    "temperature": 0,
    "stop": ["\n"]
  }'
```

The completion text in the response is ` 36`.

#### 3.2 Reasoning (ERNIE-5.5-Flash, coming soon)

ERNIE-5.5-Flash thinks before it answers. Start the server with
`--reasoning-parser ernie5` (the **Reasoning** toggle above); thinking then
lands in `reasoning_content` and the answer in `content`. Thinking is on by
default; to turn it off for a request, pass `enable_thinking: false`
through `chat_template_kwargs` (the chat template reads `enable_thinking`,
not `thinking`).

```bash
# Thinking is on by default.
curl http://localhost:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "baidu/ERNIE-5.5-Flash",
    "messages": [{"role": "user", "content": "What is 15% of 240?"}]
  }'

# Turn it off for one request.
curl http://localhost:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "baidu/ERNIE-5.5-Flash",
    "messages": [{"role": "user", "content": "What is 15% of 240?"}],
    "chat_template_kwargs": {"enable_thinking": false}
  }'
```

#### 3.3 Tool calling (ERNIE-5.5-Flash, coming soon)

Start the server with `--enable-auto-tool-choice --tool-call-parser ernie5`
(the **Tool Calling** toggle above), together with
`--reasoning-parser ernie5`. Tool calls are returned in
`message.tool_calls`.

#### 3.4 KV cache offload

Long shared prefixes (system prompts, agent histories) can stay outside
GPU memory after they leave the GPU cache and be loaded back instead of
recomputed. Pick one on the **KV Offload** row:

- **Simple.** Host memory per DP rank (`SimpleCPUOffloadConnector`).
  `cpu_bytes_to_use_per_rank` is pinned per rank, so the rank count times
  this value must stay below the node's RAM. On a GB200 tray (4 ranks) the
  default 220 GiB per rank is too much; set it to 128 GiB
  (`"cpu_bytes_to_use_per_rank": 137438953472`).
- **CPU + Filesystem.** A small CPU staging tier in front of files under
  `root_dir`. Point it at fast local NVMe with plenty of space: ERNIE 5.5's
  KV blocks took about 280 KB per token on disk in our runs.
- **Mooncake (Distributed).** Each GPU contributes host memory to a
  Mooncake pool. Install `mooncake-transfer-engine-cuda13` (vLLM 0.32.0
  ships CUDA 13), and set `MOONCAKE_DEVICE_NAME` to the InfiniBand HCAs only,
  for example `mlx5_0,mlx5_1,mlx5_4,mlx5_5` (`ibdev2netdev` lists them);
  with the empty default, Ethernet ports are picked too and transfers fail.

#### 3.5 Speculative decoding

**ECHO (MTP).** The checkpoint carries one ECHO layer after the main stack:
the backbone's own final layer, trained to predict future tokens, doubles
as the draft. vLLM runs it through its MTP path: pick **MTP, 1 / 2 / 3
steps** under the **Spec Decoding** toggle, which sets
`--speculative-config '{"method":"mtp","num_speculative_tokens":N}'`. The
layer was trained for one future token and can draft several steps by
running again; the Low-Latency preset uses 2 steps. Measured on GSM8K at
batch size 1 with greedy decoding, the mean accept length is
1.98 / 2.39 / 2.49 tokens per step for 1 / 2 / 3 draft steps, with
unchanged accuracy; 2 steps take most of the gain.

**DSpark (ERNIE-5.5-Flash, Coming Soon).** The ERNIE-5.5-Flash checkpoint
bundles a DSpark draft. Pick the **Flash** checkpoint above and the
**DSpark** mode:
`--speculative-config '{"method":"dspark","draft_sample_method":"probabilistic"}'`.
The draft length comes from the checkpoint's block size, so the config
leaves out `num_speculative_tokens`. Our runs use
`draft_sample_method: probabilistic`; vLLM's default is `greedy`. The Base
checkpoints carry no DSpark draft.

#### 3.6 Prefill / decode disaggregation

Pick **PD Cluster** as the strategy. The tested setup is one prefill and one
decode GB200 tray with `MooncakeConnector` over RDMA, each DP4 + EP4.
Install `mooncake-transfer-engine-cuda13` on both trays, and set
`device_name` in `--kv-transfer-config` to the InfiniBand HCAs only. Start
prefill, then decode, then the router (tested with `vllm-router==0.1.16`),
and send requests to the router. With speculative decoding, pass the same
`--speculative-config` to the prefill and decode servers.

### References

- [vLLM-ERNIE plugin](https://github.com/baidu/vLLM-ERNIE)
- [Hugging Face](https://huggingface.co/baidu/ERNIE-5.5-Flash-Base)
- [ModelScope](https://modelscope.cn/models/baidu/ERNIE-5.5-Flash-Base)
- [Technical report](https://ernie.baidu.com/release/ernie-5-5-flash-base/)