{ "hf_id": "mistralai/Mistral-Large-3-675B-Instruct-2512", "meta": { "title": "Mistral-Large-3-675B-Instruct", "slug": "mistral-large-3-675b-instruct", "provider": "Mistral AI", "description": "Mistral Large 3 (675B) with FP8 and NVFP4 weights for 8xH200 / 4xB200 deployments", "date_added": "2025-12-02", "date_updated": "2026-04-17", "tasks": [ "multimodal" ], "related_recipes": [ "mistralai/Ministral-3-14B-Instruct-2512" ], "hardware": { "h100": "verified", "b200": "verified" } }, "recommended_command": { "hardware": "h200", "strategy": "single_node_tp", "variant": "default", "node_count": 1, "deploy_type": "single_node", "env": {}, "docker_image": "vllm/vllm-openai:latest", "command": "vllm serve mistralai/Mistral-Large-3-675B-Instruct-2512 \\\n --tokenizer_mode mistral \\\n --config_format mistral \\\n --load_format mistral \\\n --tensor-parallel-size 8 \\\n --enable-auto-tool-choice \\\n --tool-call-parser mistral \\\n --mm-encoder-tp-mode data", "argv": [ "vllm", "serve", "mistralai/Mistral-Large-3-675B-Instruct-2512", "--tokenizer_mode", "mistral", "--config_format", "mistral", "--load_format", "mistral", "--tensor-parallel-size", "8", "--enable-auto-tool-choice", "--tool-call-parser", "mistral", "--mm-encoder-tp-mode", "data" ], "docker_command": "docker run --gpus all \\\n --privileged --ipc=host -p 8000:8000 \\\n -v ~/.cache/huggingface:/root/.cache/huggingface \\\n vllm/vllm-openai:latest mistralai/Mistral-Large-3-675B-Instruct-2512 \\\n --tokenizer_mode mistral \\\n --config_format mistral \\\n --load_format mistral \\\n --tensor-parallel-size 8 \\\n --enable-auto-tool-choice \\\n --tool-call-parser mistral \\\n --mm-encoder-tp-mode data", "docker_argv": [ "docker", "run", "--gpus", "all", "--privileged", "--ipc=host", "-p", "8000:8000", "-v", "~/.cache/huggingface:/root/.cache/huggingface", "vllm/vllm-openai:latest", "mistralai/Mistral-Large-3-675B-Instruct-2512", "--tokenizer_mode", "mistral", "--config_format", "mistral", "--load_format", "mistral", "--tensor-parallel-size", "8", "--enable-auto-tool-choice", "--tool-call-parser", "mistral", "--mm-encoder-tp-mode", "data" ], "strategy_spec": { "name": "single_node_tp", "deploy_type": "single_node", "display_name": "Tensor Parallel", "short_label": "TP", "orientation": "latency", "description": "Single-node tensor parallel. Splits the model across all local GPUs. TP size is set to the GPU count at deploy time. The simplest multi-GPU strategy — works for all model architectures.\n", "hardware_match": { "min_gpus": 1, "max_gpus": 8, "multi_node": false }, "vllm_args": [], "parallel_flag": "--tensor-parallel-size" }, "hardware_profile": { "brand": "NVIDIA", "generation": "hopper", "compute_arch": "sm90", "display_name": "H200", "description": "NVIDIA H200 SXM 141 GB HBM3e · 8-GPU HGX node", "gpu_count": 8, "vram_gb": 1128, "multi_node": false }, "alternatives": { "multi_node_tp": "/mistralai/Mistral-Large-3-675B-Instruct-2512/strategies/multi_node_tp.json", "multi_node_tp_pp": "/mistralai/Mistral-Large-3-675B-Instruct-2512/strategies/multi_node_tp_pp.json", "multi_node_tep": "/mistralai/Mistral-Large-3-675B-Instruct-2512/strategies/multi_node_tep.json", "multi_node_dep": "/mistralai/Mistral-Large-3-675B-Instruct-2512/strategies/multi_node_dep.json", "pd_cluster": "/mistralai/Mistral-Large-3-675B-Instruct-2512/strategies/pd_cluster.json", "kv_store_centralized_mooncake": "/mistralai/Mistral-Large-3-675B-Instruct-2512/strategies/kv_store_centralized_mooncake.json", "kv_store_distributed_mooncake": "/mistralai/Mistral-Large-3-675B-Instruct-2512/strategies/kv_store_distributed_mooncake.json" }, "by_hardware": { "h200": "/mistralai/Mistral-Large-3-675B-Instruct-2512/hw/h200.json", "h100": "/mistralai/Mistral-Large-3-675B-Instruct-2512/hw/h100.json", "b200": "/mistralai/Mistral-Large-3-675B-Instruct-2512/hw/b200.json", "gb200": "/mistralai/Mistral-Large-3-675B-Instruct-2512/hw/gb200.json", "b300": "/mistralai/Mistral-Large-3-675B-Instruct-2512/hw/b300.json", "gb300": "/mistralai/Mistral-Large-3-675B-Instruct-2512/hw/gb300.json", "mi300x": "/mistralai/Mistral-Large-3-675B-Instruct-2512/hw/mi300x.json", "mi325x": "/mistralai/Mistral-Large-3-675B-Instruct-2512/hw/mi325x.json", "mi350x": "/mistralai/Mistral-Large-3-675B-Instruct-2512/hw/mi350x.json", "mi355x": "/mistralai/Mistral-Large-3-675B-Instruct-2512/hw/mi355x.json" } }, "model": { "model_id": "mistralai/Mistral-Large-3-675B-Instruct-2512", "min_vllm_version": "0.11.0", "architecture": "moe", "parameter_count": "675B", "active_parameters": "22B", "context_length": 294912, "base_args": [ "--tokenizer_mode", "mistral", "--config_format", "mistral", "--load_format", "mistral" ], "base_env": {}, "install": { "pip": { "command": "uv venv\nsource .venv/bin/activate\nuv pip install -U vllm --torch-backend auto" }, "docker": { "command": "docker pull vllm/vllm-openai:latest" } } }, "features": { "tool_calling": { "description": "Mistral tool-call parser with automatic tool choice", "args": [ "--enable-auto-tool-choice", "--tool-call-parser", "mistral" ] }, "text_only": { "description": "Skip loading the vision encoder for text-only workloads — frees VRAM for KV cache. Mutually exclusive with encoder_parallel.", "args": [ "--language-model-only" ] }, "encoder_parallel": { "description": "Run the vision encoder in data-parallel mode — avoids TP comm overhead on the small encoder. Mutually exclusive with text_only.", "args": [ "--mm-encoder-tp-mode", "data" ] } }, "opt_in_features": [ "text_only" ], "variants": { "default": { "precision": "fp8", "vram_minimum_gb": 810, "description": "FP8 weights on 8xH200 (recommended for fine-tuning; up to 256K context)", "extra_args": [ "--tensor-parallel-size", "8" ] }, "nvfp4": { "model_id": "mistralai/Mistral-Large-3-675B-Instruct-2512-NVFP4", "precision": "nvfp4", "vram_minimum_gb": 405, "description": "NVFP4 weights on 4xB200 (use for <64K context; B200-native, Marlin fallback on A100/H100)", "extra_args": [ "--tensor-parallel-size", "4" ], "json": "/mistralai/Mistral-Large-3-675B-Instruct-2512-NVFP4.json" }, "fp8": { "model_id": "mistralai/Mistral-Large-3-675B-Instruct-2512-FP8", "precision": "fp8", "vram_minimum_gb": 810, "description": "FP8 quantized weights for Hopper/Blackwell GPUs", "extra_args": [ "--kv-cache-dtype", "fp8" ], "json": "/mistralai/Mistral-Large-3-675B-Instruct-2512-FP8.json" }, "redhat_nvfp4": { "model_id": "RedHatAI/Mistral-Large-3-675B-Instruct-2512-NVFP4", "precision": "nvfp4", "label": "NVFP4 (RedHat)", "vram_minimum_gb": 405, "description": "RedHatAI NVFP4 quantization", "json": "/RedHatAI/Mistral-Large-3-675B-Instruct-2512-NVFP4.json" } }, "hardware_overrides": { "amd": { "extra_args": [ "--no-enable-prefix-caching" ], "extra_env": { "VLLM_ROCM_USE_AITER": "1", "SAFETENSORS_FAST_GPU": "1" } } }, "guide": "## Overview\n\nMistral-Large-3-675B-Instruct-2512 is available in FP8 and NVFP4 formats:\n\n- **FP8** (`mistralai/Mistral-Large-3-675B-Instruct-2512`): up to 256K context, 8xH200\n- **NVFP4** (`mistralai/Mistral-Large-3-675B-Instruct-2512-NVFP4`): best for <64K context, 4xB200\n\nNVFP4 gives significant speed-up on B200 (native FP4 support). On older GPUs (A100/H100),\nvLLM falls back to Marlin FP4, which matches FP8 speed while saving memory.\n\nFor large contexts (>64K) we observed a performance regression on NVFP4 — use FP8 in\nthose cases. A minor regression on vision datasets is expected with NVFP4 (calibration\nwas mainly on text).\n\n## Prerequisites\n\n- Hardware: 8xH200 for FP8, 4xB200 for NVFP4\n- vLLM >= 0.11.0\n\n### Install vLLM\n\n```bash\nuv venv\nsource .venv/bin/activate\nuv pip install -U vllm --torch-backend auto\n```\n\n## Launch commands\n\nFP8 on 8xH200:\n\n```bash\nvllm serve mistralai/Mistral-Large-3-675B-Instruct-2512 \\\n --tensor-parallel-size 8 \\\n --tokenizer_mode mistral --config_format mistral --load_format mistral \\\n --enable-auto-tool-choice --tool-call-parser mistral\n```\n\nNVFP4 on 4xB200:\n\n```bash\nvllm serve mistralai/Mistral-Large-3-675B-Instruct-2512-NVFP4 \\\n --tensor-parallel-size 4 \\\n --tokenizer_mode mistral --config_format mistral --load_format mistral \\\n --enable-auto-tool-choice --tool-call-parser mistral\n```\n\nAdditional flags:\n- `--max-model-len`: default 262144; reduce to save memory\n- `--max-num-batched-tokens`: balance throughput vs. latency\n- `--limit-mm-per-prompt.image 0`: skip vision encoder for text-only tasks\n\n## Client Usage\n\n```python\nfrom openai import OpenAI\n\nclient = OpenAI(api_key=\"EMPTY\", base_url=\"http://localhost:8000/v1\")\nresponse = client.chat.completions.create(\n model=\"mistralai/Mistral-Large-3-675B-Instruct-2512\",\n messages=[{\"role\": \"user\", \"content\": \"Write a sentence...\"}],\n temperature=0.15, max_tokens=262144,\n)\nprint(response.choices[0].message.content)\n```\n\n## Troubleshooting\n\n- Accuracy regression with NVFP4 at long context: switch to FP8 variant.\n- OOM: reduce `--max-model-len` or adjust TP size.\n\n## References\n\n- [Mistral-Large-3 FP8](https://huggingface.co/mistralai/Mistral-Large-3-675B-Instruct-2512)\n- [Mistral-Large-3 NVFP4](https://huggingface.co/mistralai/Mistral-Large-3-675B-Instruct-2512-NVFP4)\n", "source": { "source": "vllm", "repo": "vllm-project/vllm", "ref": "main", "sha": "564ac0de1b228568cafcbfbe8ccaccc82d6023eb", "path": "recipes/models/mistralai/Mistral-Large-3-675B-Instruct-2512.yaml", "edit_url": "https://github.com/vllm-project/vllm/edit/main/recipes/models/mistralai/Mistral-Large-3-675B-Instruct-2512.yaml", "overlays": [] } }