# mistralai/Mistral-Large-4-1T-A52B — vLLM serving recipe

> Mistral Large 4: a 1T-total / 49B-active multimodal MoE with sparse MLA attention, a built-in DSpark drafter, tool calling and reasoning, in FP8, NVFP4 and AMD MXFP4 checkpoints.

Configuration: hardware `gb300` (4× GB300 NVL4) · strategy `single_node_tp` (Tensor Parallel) · variant `default` (NVFP4) · features `tool_calling`, `reasoning`, `spec_decoding`

- Interactive page: https://recipes.vllm.ai/mistralai/Mistral-Large-4-1T-A52B
- This configuration as JSON: https://recipes.vllm.ai/mistralai/Mistral-Large-4-1T-A52B.json?hardware=gb300

The commands below were rendered for exactly this configuration and take precedence over any example in the guide.

## Other configurations

Change the query string (the same parameters as the page URL) and fetch this URL again. Unknown or incompatible values fall back to the recipe default, so check the Configuration line.

- `hardware`: `b200`, `b300`, `gb300`
- `variant`: `default`, `fp8`, `mxfp4`
- `strategy`: `single_node_tp`, `multi_node_tp`, `multi_node_dep`, `multi_node_tp_pp`, `pd_cluster` (multi-node strategies also need `nodes=<n>`)
- `features`: comma-separated list of `tool_calling`, `reasoning`, `spec_decoding`, `text_only` (default: `tool_calling`, `reasoning`, `spec_decoding`)
- `preset`: a tuned, evidence-backed configuration (`baseline` opts out of a default one). Each sets its own hardware and strategy:
  - `gb300_dcp8` — TP8 DCP8 (long context) (gb300 · Multi-node TP · Long context, KV-capacity-bound · opt-in): `hardware=gb300&strategy=multi_node_tp&preset=gb300_dcp8`
  - `gb300_pcp8_dcp8` — TP1×PCP8×DCP8 + EP (lowest TTFT) (gb300 · Multi-node TP · Long prompts, low load · opt-in): `hardware=gb300&strategy=multi_node_tp&preset=gb300_pcp8_dcp8`
  - `h200_dcp8` — TP16 DCP8 (long context) (h200 · Multi-node TP · Long context, KV-capacity-bound · opt-in): `hardware=h200&strategy=multi_node_tp&preset=h200_dcp8`
  - `gb300_pd_dep_tp` — P DP4/EP → D TP4 (gb300 · PD · Throughput at 32k/60k input · default): `hardware=gb300&strategy=pd_cluster&preset=gb300_pd_dep_tp`
  - `gb300_pd_pcp_tp` — P PCP4×DCP4+EP → D TP4 (gb300 · PD · Lowest TTFT, ≥100 tok/s/user at 32k · opt-in): `hardware=gb300&strategy=pd_cluster&preset=gb300_pd_pcp_tp`
  - `gb300_pd_dep_dep` — P DP4/EP → D DP4/EP (gb300 · PD · Short inputs, high concurrency · opt-in): `hardware=gb300&strategy=pd_cluster&preset=gb300_pd_dep_dep`
  - `gb300_pd_agentic` — Agentic: P PCP4×DCP4+EP → D TP4 DCP4 (gb300 · PD · Agentic, KV-capacity-bound (AgentX traces) · opt-in): `hardware=gb300&strategy=pd_cluster&preset=gb300_pd_agentic`
  - `b200_pd_dep_tp` — P DP8/EP → D TP8 (b200 · PD · Throughput at 32k/60k input · default): `hardware=b200&strategy=pd_cluster&preset=b200_pd_dep_tp`
  - `mi350x_tp4` — TP4 (two replicas per node) (mi350x · TP · Interactive and mid concurrency · opt-in): `hardware=mi350x&variant=fp8&strategy=single_node_tp&preset=mi350x_tp4`
  - `mi350x_tp4_specialized` — TP4 specialized decode (lowest latency) (mi350x · TP · 1–4 concurrent requests · opt-in): `hardware=mi350x&variant=fp8&strategy=single_node_tp&preset=mi350x_tp4_specialized`
  - `mi350x_mxfp4_tp4` — TP4 (two replicas per node) (mi350x · TP · Interactive and mid concurrency · opt-in): `hardware=mi350x&variant=mxfp4&strategy=single_node_tp&preset=mi350x_mxfp4_tp4`
  - `mi350x_mxfp4_tp4_specialized` — TP4 specialized decode (lowest latency) (mi350x · TP · 1–4 concurrent requests · opt-in): `hardware=mi350x&variant=mxfp4&strategy=single_node_tp&preset=mi350x_mxfp4_tp4_specialized`
  - `mi355x_tp4` — TP4 (two replicas per node) (mi355x · TP · Interactive and mid concurrency · opt-in): `hardware=mi355x&variant=fp8&strategy=single_node_tp&preset=mi355x_tp4`
  - `mi355x_tp4_specialized` — TP4 specialized decode (lowest latency) (mi355x · TP · 1–4 concurrent requests · opt-in): `hardware=mi355x&variant=fp8&strategy=single_node_tp&preset=mi355x_tp4_specialized`
  - `mi355x_mxfp4_tp4` — TP4 (two replicas per node) (mi355x · TP · Interactive and mid concurrency · opt-in): `hardware=mi355x&variant=mxfp4&strategy=single_node_tp&preset=mi355x_mxfp4_tp4`
  - `mi355x_mxfp4_tp4_specialized` — TP4 specialized decode (lowest latency) (mi355x · TP · 1–4 concurrent requests · opt-in): `hardware=mi355x&variant=mxfp4&strategy=single_node_tp&preset=mi355x_mxfp4_tp4_specialized`

## Install

Requires vLLM >= 0.34.0.

Docker image: `vllm/vllm-openai:mistral-large-4`

Extra dependencies:

- Mistral tokenizer and image processing for Mistral Large 4

```bash
uv pip install -U "mistral_common[image]>=1.13.0"
```

- Only for the MegaMoE kernel (flashinfer_moe_ep_cutedsl) used by the P/D presets with expert parallelism (optional)

```bash
uv pip install "nvshmem4py-cu13==0.4.1"
```

## Serve

```bash
export VLLM_ENGINE_READY_TIMEOUT_S=6000
vllm serve mistralai/Mistral-Large-4-1T-A52B-NVFP4 \
  --tokenizer-mode mistral \
  --config-format mistral \
  --load-format mistral \
  --max-model-len 262144 \
  --max-num-batched-tokens 8192 \
  --max-num-seqs 128 \
  --gpu-memory-utilization 0.85 \
  --tensor-parallel-size 4 \
  --enable-auto-tool-choice \
  --tool-call-parser mistral \
  --reasoning-parser mistral \
  --speculative-config '{"method":"dspark","num_speculative_tokens":3}'
```

Docker equivalent:

```bash
docker run --gpus all \
  --privileged --ipc=host -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -e VLLM_ENGINE_READY_TIMEOUT_S=6000 \
  vllm/vllm-openai:mistral-large-4 mistralai/Mistral-Large-4-1T-A52B-NVFP4 \
  --tokenizer-mode mistral \
  --config-format mistral \
  --load-format mistral \
  --max-model-len 262144 \
  --max-num-batched-tokens 8192 \
  --max-num-seqs 128 \
  --gpu-memory-utilization 0.85 \
  --tensor-parallel-size 4 \
  --enable-auto-tool-choice \
  --tool-call-parser mistral \
  --reasoning-parser mistral \
  --speculative-config '{"method":"dspark","num_speculative_tokens":3}'
```

## Notes for this configuration

### Probabilistic drafting

For sampled serving at T ≥ 1 without top-p, pick **DSpark (probabilistic)**:
at T=1.5 it raises AL from 2.04 to 2.36–2.53 (+12–22% throughput). At
T=1.0 with top-p 0.95 both modes give the same acceptance (AL 2.81 vs 2.80
at c8), and greedy is ~1% faster per step. Both are lossless.

### HiSparse: set the TRTLLM all-reduce backend

With HiSparse KV offload (`--kv-transfer-config '{"kv_connector":"HiSparseConnector",...}'`),
also set `VLLM_FLASHINFER_ALLREDUCE_BACKEND=trtllm`. The default single-node
FlashInfer all-reduce (MNNVL) crashes the engine with `CUDA error: unspecified
launch failure` once HiSparse spills KV to host
([vllm#64327](https://github.com/vllm-project/vllm/issues/64327); fix in
[flashinfer#7782](https://github.com/flashinfer-ai/flashinfer/pull/7782)).
Not needed without HiSparse.

## Known issues

### ModuleNotFoundError: nvshmem

The MegaMoE backend (`--kernel-config '{"moe_backend":"flashinfer_moe_ep_cutedsl"}'`)
needs `nvshmem4py-cu13`. Install `nvshmem4py-cu13==0.4.1` (the optional
dependency above), or drop the `--kernel-config` to use the default MoE backend.

### SIGBUS in FlashInfer JIT or autotune

Seen on shared filesystems. Give each job a private
`FLASHINFER_WORKSPACE_BASE`, or add
`--kernel-config '{"enable_flashinfer_autotune": false}'`.

## Guide

> Background reference. Commands in it are examples and may target another configuration — use the Serve section above.

### Overview

Mistral Large 4 is Mistral AI's frontier mixture-of-experts multimodal model: 1T
parameters in total, 49B active per token (52B including embeddings and output
layers). It has 72 layers, 128 routed experts (top-4) plus one shared expert, MLA
attention with a DSA sparse indexer (top-2048 keys), cross-layer KV sharing, a
Pixtral vision encoder and a built-in DSpark drafter. It supports tool calling and
reasoning (`reasoning_effort`).

| Variant | Checkpoint | Weights | Validated on |
|---|---|---|---|
| NVFP4 (default) | [`mistralai/Mistral-Large-4-1T-A52B-NVFP4`](https://huggingface.co/mistralai/Mistral-Large-4-1T-A52B-NVFP4) | ModelOpt NVFP4 experts, FP8 attention / shared experts, ~610 GB | GB300, B200 |
| FP8 (default on AMD) | [`mistralai/Mistral-Large-4-1T-A52B`](https://huggingface.co/mistralai/Mistral-Large-4-1T-A52B) | FP8 block-scaled, ~1.06 TB (on AMD: MXFP4 experts at load) | GB300, H200, MI350X |
| MXFP4 (Quark) | [`amd/Mistral-Large-4-1T-A52B-Quark-MXFP4`](https://huggingface.co/amd/Mistral-Large-4-1T-A52B-Quark-MXFP4) | AMD Quark OCP MXFP4 routed experts, rest unchanged, ~578 GB | MI350 Series |

Both checkpoints ship the DSpark drafter. The speculative depth is fixed at 3 by
the checkpoint.

vLLM serves the model on model runner V2 (the default) with breakable CUDA graphs,
FlashMLA sparse attention and an FP8 latent KV cache (`fp8_ds_mla`) by default, so
no attention or KV-cache flags are needed. The checkpoint advertises 1,048,576
positions; we validated up to 256k, which is the `--max-model-len` set here.

### Prerequisites

- **vLLM**: the `vllm/vllm-openai:mistral-large-4` image. No vLLM release or
  nightly wheel includes Mistral Large 4 support yet.
- **AMD**: the ROCm image `vllm/vllm-openai-rocm:mistral-large-4`.
- **mistral_common**: `mistral_common[image] >= 1.13.0` (included in both images).
- **MegaMoE** (P/D presets only): the image does not include `nvshmem4py-cu13`;
  install it in the container, or drop the `--kernel-config` flag.
- **Hardware**: validated on NVIDIA GB300 (one 4-GPU tray with NVFP4, or 8 GPUs as
  two trays with NVFP4 and FP8), 8x B300 (NVFP4 and FP8), 8x B200 (NVFP4) and
  2x 8 H200 (FP8, TP16 or PP2 x TP8). FP8 needs ~128 GiB of weights per GPU on
  8 GPUs, NVFP4 ~76 GiB. AMD Instinct MI350X (8 GPUs, TP8 or two TP4 replicas)
  with the ROCm image.

### Choosing a configuration

All numbers are NVFP4 on 8x GB300 with DSpark unless stated; the presets carry
the full tables. On H200 use TP16 (or TP16 DCP8 for long context), with PP2 x TP8
as the alternative; we took no performance numbers there.

| Goal | Configuration | Why |
|---|---|---|
| Default, best interactivity | TP8 + DSpark | c1 TPOT 3.13 ms (NVFP4) / 3.56 ms (FP8), AL ~2.9; 1.9–2.2x output throughput vs no spec |
| Throughput, short/medium inputs | DP8 + EP + DSpark | 209.8 output tok/s/GPU at 8k (c32, 55 tok/s/user), 50.6 at 32k |
| Long context, KV-bound | TP8 DCP8 (+ DSpark) | 28.76M KV tokens vs 3.60M on TP8 (~27 concurrent 1M-token requests by KV size) |
| Lowest TTFT on long prompts | TP1×PCP8×DCP8 + EP (+ DSpark) | c1 TTFT 0.16 / 0.54 / 1.08 s at 8k / 32k / 60k, ~3x faster than TP8; text only |
| Disaggregated throughput | P DP4/EP → D TP4 | 241 output tok/s/GPU at 32k (≥40 tok/s/user), 129 at 60k |
| Disaggregated, lowest TTFT | P PCP4×DCP4+EP → D TP4 | c1 TTFT 0.35 / 0.64 / 1.21 s at 8k / 32k / 60k |
| Disaggregated, short inputs | P DP4/EP → D DP4/EP | 766 output tok/s/GPU at 8k |
| Agentic, KV-bound, no spec | P PCP4×DCP4+EP → D TP4 DCP4 | at 32 sessions TTFT p50 0.72 s vs 14.95 s with D TP4 |

Colocated TP8 + DSpark at ≥100 tok/s/user: 110.2 output tok/s/GPU at 8k (c8),
30.5 at 32k (c2), 21.4 at 60k (c2). DCP8 doesn't pay off colocated until KV
capacity binds. Disaggregated 4+4 beats colocated 8-GPU serving by 1.8–2.2x at 8k
and 4–5x at 32k/60k, because colocated decode shares steps with prefill chunks.

Tuning knobs used by the presets:

- `--kernel-config '{"moe_backend":"flashinfer_moe_ep_cutedsl"}'` (MegaMoE) with
  expert parallelism, +14–17% prefill req/s at 32k. Needs `nvshmem4py-cu13`. On EP
  decode nodes only with `--max-num-batched-tokens` below 512 (256 here).
- `--cp-kv-cache-interleave-size 64` on the side that runs DCP in P/D.
- Prefix caching (on by default) for agentic workloads, with session-sticky routing
  to the DP ranks of DP sides.

NVFP4 vs FP8 at TP8: 1.09–1.18x faster per token without spec and 1.14–1.22x with
DSpark, and ~1.5x the KV capacity.

### AMD (ROCm)

Mistral Large 4 runs on AMD Instinct MI350 Series GPUs (gfx950) with the ROCm build
of vLLM (image `vllm/vllm-openai-rocm:mistral-large-4`). The routed experts run as
OCP MXFP4 on AITER's fused MoE, either from the
AMD Quark MXFP4 checkpoint or from the FP8 checkpoint re-quantized at load
(`VLLM_FP8_MOE_REQUANT=mxfp4`, layer by layer, so peak load memory stays low). With
MXFP4 experts the model fits four GPUs, so one 8-GPU node serves two TP4 replicas.
The sparse MLA runs on AITER's MLA decode kernels (`ROCM_AITER_MLA_SPARSE`), the
DSA indexer on AITER's gfx950 indexer kernels, and the latent KV cache is plain FP8
(E4M3) by default, 1.86x the capacity of bf16. The builder adds the AITER
environment on AMD.

| Goal | Configuration | Why |
|---|---|---|
| Default | TP8 | one replica per node; DSpark at TP8 not yet validated on ROCm |
| Interactive, more replicas | TP4 + DSpark (preset) | AL 3.1; TPOT 2.7x / 1.9x / 1.8x lower than no spec at c1 / c8 / c32 |
| Lowest latency, 1–4 users | TP4 specialized decode (preset) | one fused kernel per MoE layer: TPOT 1.5x lower at c1; 2.9x with DSpark |

MXFP4 vs FP8 experts at TP8: +22–71% output tok/s at 1–64 concurrent requests.

### Accuracy

Validated against Mistral's reference implementation on 2x GB300 TP8: per-layer
parity and 256k long-context perplexity equal to the reference. GSM8K (1319
questions, 5-shot) scores 0.909 on the FP8 checkpoint and 0.918 with DSpark, which
sits at the base server's own noise floor. All P/D and colocated configurations
scored 0.965–0.980 on GSM8K-200 (zero-shot chat).

### Client usage

```bash
curl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{
  "model": "mistralai/Mistral-Large-4-1T-A52B",
  "messages": [{"role": "user", "content": "Explain sparse attention in two sentences."}],
  "reasoning_effort": "high"
}'
```

Image input (allow more images per prompt with `--limit-mm-per-prompt '{"image": N}'`):

```python
from openai import OpenAI

client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1")
response = client.chat.completions.create(
    model="mistralai/Mistral-Large-4-1T-A52B",
    messages=[{
        "role": "user",
        "content": [
            {"type": "text", "text": "What does this chart show?"},
            {"type": "image_url", "image_url": {"url": "https://example.com/chart.png"}},
        ],
    }],
)
print(response.choices[0].message.reasoning, response.choices[0].message.content)
```

### Limitations

- Prefill context parallelism is text-only (`--language-model-only`).
- DSpark doesn't run on a DCP decode node in P/D yet.
- DSpark wasn't validated with pipeline parallelism.
- AMD: NVFP4 checkpoints, DCP / PCP, NIXL P/D and HiSparse aren't supported on
  ROCm yet; TP8 with DSpark and MI300 Series (gfx942) are not validated.

### References

- [Mistral-Large-4-1T-A52B (FP8)](https://huggingface.co/mistralai/Mistral-Large-4-1T-A52B)
- [Mistral-Large-4-1T-A52B-NVFP4](https://huggingface.co/mistralai/Mistral-Large-4-1T-A52B-NVFP4)
- [Mistral-Large-4-1T-A52B-Quark-MXFP4](https://huggingface.co/amd/Mistral-Large-4-1T-A52B-Quark-MXFP4)
- [Cross-layer MLA KV sharing (vllm#64095)](https://github.com/vllm-project/vllm/pull/64095)
- [DSpark sliding-window MLA drafts (vllm#63795)](https://github.com/vllm-project/vllm/pull/63795) and [DSpark under DCP (vllm#63796)](https://github.com/vllm-project/vllm/pull/63796)
- [NIXL connector](https://docs.vllm.ai/en/latest/features/nixl_connector_usage.html)