# XiaomiMiMo/MiMo-V2.6-Pro-RL — vLLM serving recipe

> Xiaomi's flagship omnimodal MoE (1.02T total / 42B active) with hybrid attention, FP8-compute/mxfp4-stored weights, 1M context, and a DFlash speculative decoder — the default MOPD checkpoint fixes the RL release's tool-call repetition

Configuration: hardware `h200` (8× H200) · strategy `single_node_tp` (Tensor Parallel) · variant `default` (MOPD) · features `tool_calling`, `reasoning`

- Interactive page: https://recipes.vllm.ai/XiaomiMiMo/MiMo-V2.6-Pro-RL
- This configuration as JSON: https://recipes.vllm.ai/XiaomiMiMo/MiMo-V2.6-Pro-RL.json?hardware=h200

The commands below were rendered for exactly this configuration and take precedence over any example in the guide.

## Other configurations

Change the query string (the same parameters as the page URL) and fetch this URL again. Unknown or incompatible values fall back to the recipe default, so check the Configuration line.

- `hardware`: `h100`, `h200`, `b200`, `gb200`, `b300`, `gb300`, `mi300x`, `mi325x`, `mi350x`, `mi355x`
- `variant`: `default`, `rl`
- `strategy`: `single_node_tp`, `single_node_tep`, `pd_cluster` (multi-node strategies also need `nodes=<n>`)
- `features`: comma-separated list of `tool_calling`, `reasoning`, `spec_decoding` (default: `tool_calling`, `reasoning`)

## Install

Requires vLLM >= 0.31.0.

```bash
uv venv
source .venv/bin/activate
uv pip install -U vllm --torch-backend auto
```

Docker image: `vllm/vllm-openai:v0.31.0`

## Serve

```bash
vllm serve XiaomiMiMo/MiMo-V2.6-Pro-MOPD \
  --trust-remote-code \
  --max-model-len auto \
  --generation-config vllm \
  --gpu-memory-utilization 0.95 \
  --tensor-parallel-size 8 \
  --tool-call-parser mimo \
  --enable-auto-tool-choice \
  --reasoning-parser mimo
```

Docker equivalent:

```bash
docker run --gpus all \
  --privileged --ipc=host -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai:v0.31.0 XiaomiMiMo/MiMo-V2.6-Pro-MOPD \
  --trust-remote-code \
  --max-model-len auto \
  --generation-config vllm \
  --gpu-memory-utilization 0.95 \
  --tensor-parallel-size 8 \
  --tool-call-parser mimo \
  --enable-auto-tool-choice \
  --reasoning-parser mimo
```

## Guide

> Background reference. Commands in it are examples and may target another configuration — use the Serve section above.

### Overview

This recipe serves the MiMo-V2.6-Pro family from two checkpoints on the
Variant row:

- **MOPD** (default) — [XiaomiMiMo/MiMo-V2.6-Pro-MOPD](https://huggingface.co/XiaomiMiMo/MiMo-V2.6-Pro-MOPD),
  the MOPD2-distilled upgrade and the recommended starting point for new
  deployments. It fixes the RL release's most visible agentic failure mode —
  tool-call repetition, where the model keeps re-issuing the same or highly
  similar tool calls while making no progress — with benchmark quality
  unchanged, folded into a regular MOPD2 pass via a short specialized-teacher
  run ([diagnosis blog](https://mimo.xiaomi.com/blog/mimo-v2-6-tool-call-repetition)).
- **RL (original)** — [XiaomiMiMo/MiMo-V2.6-Pro-RL](https://huggingface.co/XiaomiMiMo/MiMo-V2.6-Pro-RL),
  the 2026-09-22 RL release, kept for reproducing earlier results. It can get
  stuck re-issuing the same tool calls in agentic workloads; prefer **MOPD**
  unless you are reproducing a specific earlier run.

MOPD2 (Technical Report §5.6) is multi-teacher on-policy distillation:
domain-specialized mixRL and SFT teachers supervise the student on three
on-policy streams (standard MOPD, teacher-prefix OPD, and SFT-prefix OPD).

The architecture is shared by both checkpoints: a sparse MoE model with
1.02T total parameters and 42B active per token — 70 layers (1 dense + 69 MoE)
with 384 routed experts (top-8), hybrid attention (sliding-window 128 and
global attention at a 6:1 ratio), and a 5-layer DFlash-style MTP drafter that
predicts 7 tokens per pass. The model is natively omnimodal (text, image,
video, audio via a 681M-param MiMo ViT and audio encoders) and supports up to
1M tokens of context.

Weights are stored as mxfp4 and computed as FP8 (block-wise e4m3, 128x128),
so the checkpoint is 566 GB on disk. Loading this mixed storage format
requires vLLM with the MiMo V2 mxfp4/bf16-router support
([vllm-project/vllm#57784](https://github.com/vllm-project/vllm/pull/57784)),
first shipped in the v0.31.0 stable release.

### Prerequisites

- Hardware: 8x H200 (TP8), 4x MI355X (TP4), or equivalent aggregate VRAM (>= 680 GB)

#### Pull the vLLM docker image

vLLM 0.31.0 is the first stable release that can load the mxfp4-stored
weights ([vllm-project/vllm#57784](https://github.com/vllm-project/vllm/pull/57784));
earlier stable releases cannot:

```bash
docker pull vllm/vllm-openai:v0.31.0
```

A vLLM nightly wheel built after 2026-09-20 also works (`uv pip install -U
vllm --extra-index-url https://wheels.vllm.ai/nightly/cu130`).

The Hugging Face model cards point at `vllm/vllm-openai:mimov25-cu129`, the
V2.5-era image; that tag predates the mxfp4-stored-weight support linked above,
so use the v0.31.0 image recommended above.

This release also registers the `mimo` tool-call and reasoning parsers with
the Rust frontend
([vllm-project/vllm#57933](https://github.com/vllm-project/vllm/pull/57933)).

### Launch command

Single-node TP8 (H200). Selecting the **RL (original)** variant in the builder
swaps only the checkpoint id (`XiaomiMiMo/MiMo-V2.6-Pro-RL`); everything else
is identical:

```bash
vllm serve XiaomiMiMo/MiMo-V2.6-Pro-MOPD \
  --tensor-parallel-size 8 \
  --trust-remote-code \
  --gpu-memory-utilization 0.95 \
  --max-model-len auto \
  --reasoning-parser mimo \
  --tool-call-parser mimo \
  --enable-auto-tool-choice \
  --generation-config vllm
```

#### AMD (MI355X, TP4)

```bash
VLLM_ROCM_USE_AITER=1 \
VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4 \
vllm serve XiaomiMiMo/MiMo-V2.6-Pro-MOPD \
  --tensor-parallel-size 4 \
  --trust-remote-code \
  --gpu-memory-utilization 0.93 \
  --kv-cache-dtype fp8 \
  --max-model-len auto \
  --reasoning-parser mimo \
  --tool-call-parser mimo \
  --enable-auto-tool-choice \
  --generation-config vllm
```

MXFP4 MoE weights run natively on CDNA 4, so no load-time requantization is
needed. `VLLM_ROCM_USE_AITER=1` is off by default and is required to reach
the AITER MXFP4 MoE and FP8 block-scaled GEMM kernels. Attention resolves to
`TRITON_ATTN_DIFFKV`, the only backend covering this model's asymmetric head
dims (QK 192 / V 128) together with attention sinks and sliding-window
attention.

On MI355X, **Tensor Parallel** defaults to **TP4**: 4x288 GiB clears the 680 GiB
requirement, so TP4 halves the GPUs per server and doubles the servers per node.
For TP8, pass `--tensor-parallel-size 8`.

The checkpoint pre-shards its fused QKV into `num_key_value_heads` (8) chunks,
each carrying its own FP8 scales. TP8 consumes them directly; TP4 merges two
chunks per rank, which vLLM handles by dequantizing, reordering and
re-quantizing at load time. TP4 costs no accuracy.

With DFlash speculative decoding (drafter ships inside the checkpoint).
vLLM does not resolve a `<repo>/dflash` Hub subfolder as the draft model —
`model` must be a concrete local path to the `dflash/` directory inside the
downloaded snapshot. Resolve it per checkpoint (run `hf download
<checkpoint>` first if it is not cached yet):

```bash
# MOPD variant
DFLASH_DIR=$(ls -d ~/.cache/huggingface/hub/models--XiaomiMiMo--MiMo-V2.6-Pro-MOPD/snapshots/*/dflash)
# RL (original) variant
# DFLASH_DIR=$(ls -d ~/.cache/huggingface/hub/models--XiaomiMiMo--MiMo-V2.6-Pro-RL/snapshots/*/dflash)
echo "$DFLASH_DIR"
```

```bash
vllm serve XiaomiMiMo/MiMo-V2.6-Pro-MOPD \
  --tensor-parallel-size 8 \
  --trust-remote-code \
  --gpu-memory-utilization 0.95 \
  --max-model-len auto \
  --speculative-config "{\"method\":\"dflash\",\"model\":\"$DFLASH_DIR\",\"num_speculative_tokens\":7}" \
  --compilation-config '{"cudagraph_mode":"FULL_DECODE_ONLY"}' \
  --reasoning-parser mimo \
  --tool-call-parser mimo \
  --enable-auto-tool-choice \
  --generation-config vllm
```

The drafter inherits the target's tensor-parallel size by default. To pin it
explicitly, add `"draft_tensor_parallel_size": 8` to the speculative config
(useful if you want the draft on fewer GPUs than the target, or to silence
mis-detection on asymmetric topologies).

### Client Usage

```bash
curl -X POST http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "XiaomiMiMo/MiMo-V2.6-Pro-MOPD",
    "messages": [{"role": "user", "content": "Hello MiMo!"}],
    "temperature": 1.0,
    "top_p": 0.95,
    "chat_template_kwargs": {"enable_thinking": true}
  }'
```

Recommended sampling: `temperature=1.0`, `top_p=0.95`.
Set `"enable_thinking": false` (or omit the kwargs) to disable thinking mode.

### References

- [MiMo-V2.6-Pro-MOPD on Hugging Face](https://huggingface.co/XiaomiMiMo/MiMo-V2.6-Pro-MOPD)
- [MiMo-V2.6-Pro-RL on Hugging Face](https://huggingface.co/XiaomiMiMo/MiMo-V2.6-Pro-RL)
- [MiMo-V2.6 technical report](https://huggingface.co/XiaomiMiMo/MiMo-V2.6-Pro-RL/blob/main/MiMo_V2_6_technical_report.pdf) (§5.6 covers MOPD2)
- [Diagnosing and Mitigating Tool-Call Repetition in MiMo-V2.6](https://mimo.xiaomi.com/blog/mimo-v2-6-tool-call-repetition)
- [MiMo-V2.6 blog post](https://mimo.xiaomi.com/mimo-v2-6)
- [vLLM PR #57784 — mxfp4 MoE / bf16 router / DFlash for MiMo V2](https://github.com/vllm-project/vllm/pull/57784)
- [vLLM PR #57933 — MiMo tool-call/reasoning parsers for the Rust frontend](https://github.com/vllm-project/vllm/pull/57933)
- [SGLang MiMo cookbook](https://docs.sglang.io/cookbook/autoregressive/Xiaomi/MiMo-V2.5)
- [vLLM ROCm GPU install docs](https://docs.vllm.ai/en/latest/getting_started/installation/gpu/?device=rocm)