vLLM/Recipes
Search
Mistral AI

mistralai/Mistral-Large-4-1T-A52B

Mistral Large 4: a 1T-total / 49B-active multimodal MoE with sparse MLA attention, a built-in DSpark drafter, tool calling and reasoning, in FP8, NVFP4 and AMD MXFP4 checkpoints.

8x GB300, NVFP4 + DSpark: TP8 c1 TPOT 3.1 ms; 1P1D (4+4) P DP4/EP → D TP4 reaches 241 output tok/s/GPU at 32k input

View on HuggingFace
moe1T / 49B1,048,576 ctxvLLM 0.34.0+multimodal

Overview

Mistral Large 4 is Mistral AI's frontier mixture-of-experts multimodal model: 1T parameters in total, 49B active per token (52B including embeddings and output layers). It has 72 layers, 128 routed experts (top-4) plus one shared expert, MLA attention with a DSA sparse indexer (top-2048 keys), cross-layer KV sharing, a Pixtral vision encoder and a built-in DSpark drafter. It supports tool calling and reasoning (reasoning_effort).

VariantCheckpointWeightsValidated on
NVFP4 (default)mistralai/Mistral-Large-4-1T-A52B-NVFP4ModelOpt NVFP4 experts, FP8 attention / shared experts, ~610 GBGB300, B200
FP8 (default on AMD)mistralai/Mistral-Large-4-1T-A52BFP8 block-scaled, ~1.06 TB (on AMD: MXFP4 experts at load)GB300, H200, MI350X
MXFP4 (Quark)amd/Mistral-Large-4-1T-A52B-Quark-MXFP4AMD Quark OCP MXFP4 routed experts, rest unchanged, ~578 GBMI350 Series

Both checkpoints ship the DSpark drafter. The speculative depth is fixed at 3 by the checkpoint.

vLLM serves the model on model runner V2 (the default) with breakable CUDA graphs, FlashMLA sparse attention and an FP8 latent KV cache (fp8_ds_mla) by default, so no attention or KV-cache flags are needed. The checkpoint advertises 1,048,576 positions; we validated up to 256k, which is the --max-model-len set here.

Prerequisites

  • vLLM: the vllm/vllm-openai:mistral-large-4 image. No vLLM release or nightly wheel includes Mistral Large 4 support yet.
  • AMD: the ROCm image vllm/vllm-openai-rocm:mistral-4-large.
  • mistral_common: mistral_common[image] >= 1.13.0 (included in both images).
  • MegaMoE (P/D presets only): the image does not include nvshmem4py-cu13; install it in the container, or drop the --kernel-config flag.
  • Hardware: validated on NVIDIA GB300 (one 4-GPU tray with NVFP4, or 8 GPUs as two trays with NVFP4 and FP8), 8x B300 (NVFP4 and FP8), 8x B200 (NVFP4) and 2x 8 H200 (FP8, TP16 or PP2 x TP8). FP8 needs ~128 GiB of weights per GPU on 8 GPUs, NVFP4 ~76 GiB. AMD Instinct MI350X (8 GPUs, TP8 or two TP4 replicas) with the ROCm image.

Choosing a configuration

All numbers are NVFP4 on 8x GB300 with DSpark unless stated; the presets carry the full tables. On H200 use TP16 (or TP16 DCP8 for long context), with PP2 x TP8 as the alternative; we took no performance numbers there.

GoalConfigurationWhy
Default, best interactivityTP8 + DSparkc1 TPOT 3.13 ms (NVFP4) / 3.56 ms (FP8), AL ~2.9; 1.9–2.2x output throughput vs no spec
Throughput, short/medium inputsDP8 + EP + DSpark209.8 output tok/s/GPU at 8k (c32, 55 tok/s/user), 50.6 at 32k
Long context, KV-boundTP8 DCP8 (+ DSpark)28.76M KV tokens vs 3.60M on TP8 (~27 concurrent 1M-token requests by KV size)
Lowest TTFT on long promptsTP1×PCP8×DCP8 + EP (+ DSpark)c1 TTFT 0.16 / 0.54 / 1.08 s at 8k / 32k / 60k, ~3x faster than TP8; text only
Disaggregated throughputP DP4/EP → D TP4241 output tok/s/GPU at 32k (≥40 tok/s/user), 129 at 60k
Disaggregated, lowest TTFTP PCP4×DCP4+EP → D TP4c1 TTFT 0.35 / 0.64 / 1.21 s at 8k / 32k / 60k
Disaggregated, short inputsP DP4/EP → D DP4/EP766 output tok/s/GPU at 8k
Agentic, KV-bound, no specP PCP4×DCP4+EP → D TP4 DCP4at 32 sessions TTFT p50 0.72 s vs 14.95 s with D TP4

Colocated TP8 + DSpark at ≥100 tok/s/user: 110.2 output tok/s/GPU at 8k (c8), 30.5 at 32k (c2), 21.4 at 60k (c2). DCP8 doesn't pay off colocated until KV capacity binds. Disaggregated 4+4 beats colocated 8-GPU serving by 1.8–2.2x at 8k and 4–5x at 32k/60k, because colocated decode shares steps with prefill chunks.

Tuning knobs used by the presets:

  • --kernel-config '{"moe_backend":"flashinfer_moe_ep_cutedsl"}' (MegaMoE) with expert parallelism, +14–17% prefill req/s at 32k. Needs nvshmem4py-cu13. On EP decode nodes only with --max-num-batched-tokens below 512 (256 here).
  • --cp-kv-cache-interleave-size 64 on the side that runs DCP in P/D.
  • Prefix caching (on by default) for agentic workloads, with session-sticky routing to the DP ranks of DP sides.

NVFP4 vs FP8 at TP8: 1.09–1.18x faster per token without spec and 1.14–1.22x with DSpark, and ~1.5x the KV capacity.

AMD (ROCm)

Mistral Large 4 runs on AMD Instinct MI350 Series GPUs (gfx950) with the ROCm build of vLLM (image vllm/vllm-openai-rocm:mistral-4-large). The routed experts run as OCP MXFP4 on AITER's fused MoE, either from the AMD Quark MXFP4 checkpoint or from the FP8 checkpoint re-quantized at load (VLLM_FP8_MOE_REQUANT=mxfp4, layer by layer, so peak load memory stays low). With MXFP4 experts the model fits four GPUs, so one 8-GPU node serves two TP4 replicas. The sparse MLA runs on AITER's MLA decode kernels (ROCM_AITER_MLA_SPARSE), the DSA indexer on AITER's gfx950 indexer kernels, and the latent KV cache is plain FP8 (E4M3) by default, 1.86x the capacity of bf16. The builder adds the AITER environment on AMD.

GoalConfigurationWhy
DefaultTP8one replica per node; DSpark at TP8 not yet validated on ROCm
Interactive, more replicasTP4 + DSpark (preset)AL 3.1; TPOT 2.7x / 1.9x / 1.8x lower than no spec at c1 / c8 / c32
Lowest latency, 1–4 usersTP4 specialized decode (preset)one fused kernel per MoE layer: TPOT 1.5x lower at c1; 2.9x with DSpark

MXFP4 vs FP8 experts at TP8: +22–71% output tok/s at 1–64 concurrent requests.

Accuracy

Validated against Mistral's reference implementation on 2x GB300 TP8: per-layer parity and 256k long-context perplexity equal to the reference. GSM8K (1319 questions, 5-shot) scores 0.909 on the FP8 checkpoint and 0.918 with DSpark, which sits at the base server's own noise floor. All P/D and colocated configurations scored 0.965–0.980 on GSM8K-200 (zero-shot chat).

Client usage

curl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{
  "model": "mistralai/Mistral-Large-4-1T-A52B",
  "messages": [{"role": "user", "content": "Explain sparse attention in two sentences."}],
  "reasoning_effort": "high"
}'

Image input (allow more images per prompt with --limit-mm-per-prompt '{"image": N}'):

from openai import OpenAI

client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1")
response = client.chat.completions.create(
    model="mistralai/Mistral-Large-4-1T-A52B",
    messages=[{
        "role": "user",
        "content": [
            {"type": "text", "text": "What does this chart show?"},
            {"type": "image_url", "image_url": {"url": "https://example.com/chart.png"}},
        ],
    }],
)
print(response.choices[0].message.reasoning, response.choices[0].message.content)

Limitations

  • Prefill context parallelism is text-only (--language-model-only).
  • DSpark doesn't run on a DCP decode node in P/D yet.
  • DSpark wasn't validated with pipeline parallelism.
  • AMD: NVFP4 checkpoints, DCP / PCP, NIXL P/D and HiSparse aren't supported on ROCm yet; TP8 with DSpark and MI300 Series (gfx942) are not validated.

References