# mistralai/Mistral-Large-3-675B-Instruct-2512 — vLLM serving recipe

> Mistral Large 3 (675B) with FP8 and NVFP4 weights for 8xH200 / 4xB200 deployments

Configuration: hardware `h200` (8× H200) · strategy `single_node_tp` (Tensor Parallel) · variant `default` (FP8) · features `tool_calling`, `encoder_parallel`

- Interactive page: https://recipes.vllm.ai/mistralai/Mistral-Large-3-675B-Instruct-2512
- This configuration as JSON: https://recipes.vllm.ai/mistralai/Mistral-Large-3-675B-Instruct-2512.json?hardware=h200

The commands below were rendered for exactly this configuration and take precedence over any example in the guide.

## Other configurations

Change the query string (the same parameters as the page URL) and fetch this URL again. Unknown or incompatible values fall back to the recipe default, so check the Configuration line.

- `hardware`: `h100`, `h200`, `b200`, `gb200`, `b300`, `gb300`, `mi300x`, `mi325x`, `mi350x`, `mi355x`
- `variant`: `default`, `nvfp4`, `fp8`, `redhat_nvfp4`
- `strategy`: `single_node_tp`, `multi_node_tp`, `multi_node_tp_pp`, `multi_node_tep`, `multi_node_dep`, `pd_cluster` (multi-node strategies also need `nodes=<n>`)
- `features`: comma-separated list of `tool_calling`, `text_only`, `encoder_parallel` (default: `tool_calling`, `encoder_parallel`)

## Install

Requires vLLM >= 0.11.0.

```bash
uv venv
source .venv/bin/activate
uv pip install -U vllm --torch-backend auto
```

Docker image: `vllm/vllm-openai:latest`

## Serve

```bash
vllm serve mistralai/Mistral-Large-3-675B-Instruct-2512 \
  --tokenizer_mode mistral \
  --config_format mistral \
  --load_format mistral \
  --tensor-parallel-size 8 \
  --enable-auto-tool-choice \
  --tool-call-parser mistral \
  --mm-encoder-tp-mode data
```

Docker equivalent:

```bash
docker run --gpus all \
  --privileged --ipc=host -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai:latest mistralai/Mistral-Large-3-675B-Instruct-2512 \
  --tokenizer_mode mistral \
  --config_format mistral \
  --load_format mistral \
  --tensor-parallel-size 8 \
  --enable-auto-tool-choice \
  --tool-call-parser mistral \
  --mm-encoder-tp-mode data
```

## Guide

> Background reference. Commands in it are examples and may target another configuration — use the Serve section above.

### Overview

Mistral-Large-3-675B-Instruct-2512 is available in FP8 and NVFP4 formats:

- **FP8** (`mistralai/Mistral-Large-3-675B-Instruct-2512`): up to 256K context, 8xH200
- **NVFP4** (`mistralai/Mistral-Large-3-675B-Instruct-2512-NVFP4`): best for <64K context, 4xB200

NVFP4 gives significant speed-up on B200 (native FP4 support). On older GPUs (A100/H100),
vLLM falls back to Marlin FP4, which matches FP8 speed while saving memory.

For large contexts (>64K) we observed a performance regression on NVFP4 — use FP8 in
those cases. A minor regression on vision datasets is expected with NVFP4 (calibration
was mainly on text).

### Prerequisites

- Hardware: 8xH200 for FP8, 4xB200 for NVFP4
- vLLM >= 0.11.0

#### Install vLLM

```bash
uv venv
source .venv/bin/activate
uv pip install -U vllm --torch-backend auto
```

### Launch commands

FP8 on 8xH200:

```bash
vllm serve mistralai/Mistral-Large-3-675B-Instruct-2512 \
  --tensor-parallel-size 8 \
  --tokenizer_mode mistral --config_format mistral --load_format mistral \
  --enable-auto-tool-choice --tool-call-parser mistral
```

NVFP4 on 4xB200:

```bash
vllm serve mistralai/Mistral-Large-3-675B-Instruct-2512-NVFP4 \
  --tensor-parallel-size 4 \
  --tokenizer_mode mistral --config_format mistral --load_format mistral \
  --enable-auto-tool-choice --tool-call-parser mistral
```

Additional flags:
- `--max-model-len`: default 262144; reduce to save memory
- `--max-num-batched-tokens`: balance throughput vs. latency
- `--limit-mm-per-prompt.image 0`: skip vision encoder for text-only tasks

### Client Usage

```python
from openai import OpenAI

client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1")
response = client.chat.completions.create(
    model="mistralai/Mistral-Large-3-675B-Instruct-2512",
    messages=[{"role": "user", "content": "Write a sentence..."}],
    temperature=0.15, max_tokens=262144,
)
print(response.choices[0].message.content)
```

### Troubleshooting

- Accuracy regression with NVFP4 at long context: switch to FP8 variant.
- OOM: reduce `--max-model-len` or adjust TP size.

### References

- [Mistral-Large-3 FP8](https://huggingface.co/mistralai/Mistral-Large-3-675B-Instruct-2512)
- [Mistral-Large-3 NVFP4](https://huggingface.co/mistralai/Mistral-Large-3-675B-Instruct-2512-NVFP4)