{ "hf_id": "mistralai/Mistral-Small-4-119B-2603", "meta": { "title": "Mistral-Small-4-119B", "slug": "mistral-small-4-119b", "provider": "Mistral AI", "description": "Mistral Small 4 (119B MoE, 6.5B active) — multimodal hybrid instruct + reasoning model with native FP8 weights and 256K context", "date_added": "2026-05-13", "date_updated": "2026-05-13", "tasks": [ "multimodal" ], "related_recipes": [ "mistralai/Mistral-Medium-3.5-128B", "mistralai/Mistral-Large-3-675B-Instruct-2512" ] }, "recommended_command": { "hardware": "h200", "strategy": "single_node_tp", "variant": "default", "node_count": 1, "deploy_type": "single_node", "env": {}, "docker_image": "vllm/vllm-openai:latest", "command": "vllm serve mistralai/Mistral-Small-4-119B-2603 \\\n --max-model-len 262144 \\\n --tensor-parallel-size 2 \\\n --enable-auto-tool-choice \\\n --tool-call-parser mistral \\\n --reasoning-parser mistral", "argv": [ "vllm", "serve", "mistralai/Mistral-Small-4-119B-2603", "--max-model-len", "262144", "--tensor-parallel-size", "2", "--enable-auto-tool-choice", "--tool-call-parser", "mistral", "--reasoning-parser", "mistral" ], "docker_command": "docker run --gpus all \\\n --privileged --ipc=host -p 8000:8000 \\\n -v ~/.cache/huggingface:/root/.cache/huggingface \\\n vllm/vllm-openai:latest mistralai/Mistral-Small-4-119B-2603 \\\n --max-model-len 262144 \\\n --tensor-parallel-size 2 \\\n --enable-auto-tool-choice \\\n --tool-call-parser mistral \\\n --reasoning-parser mistral", "docker_argv": [ "docker", "run", "--gpus", "all", "--privileged", "--ipc=host", "-p", "8000:8000", "-v", "~/.cache/huggingface:/root/.cache/huggingface", "vllm/vllm-openai:latest", "mistralai/Mistral-Small-4-119B-2603", "--max-model-len", "262144", "--tensor-parallel-size", "2", "--enable-auto-tool-choice", "--tool-call-parser", "mistral", "--reasoning-parser", "mistral" ], "strategy_spec": { "name": "single_node_tp", "deploy_type": "single_node", "display_name": "Tensor Parallel", "short_label": "TP", "orientation": "latency", "description": "Single-node tensor parallel. Splits the model across all local GPUs. TP size is set to the GPU count at deploy time. The simplest multi-GPU strategy — works for all model architectures.\n", "hardware_match": { "min_gpus": 1, "max_gpus": 8, "multi_node": false }, "vllm_args": [], "parallel_flag": "--tensor-parallel-size" }, "hardware_profile": { "brand": "NVIDIA", "generation": "hopper", "compute_arch": "sm90", "display_name": "H200", "description": "NVIDIA H200 SXM 141 GB HBM3e · 8-GPU HGX node", "gpu_count": 8, "vram_gb": 1128, "multi_node": false }, "alternatives": { "multi_node_tp": "/mistralai/Mistral-Small-4-119B-2603/strategies/multi_node_tp.json", "pd_cluster": "/mistralai/Mistral-Small-4-119B-2603/strategies/pd_cluster.json", "kv_store_centralized_mooncake": "/mistralai/Mistral-Small-4-119B-2603/strategies/kv_store_centralized_mooncake.json", "kv_store_distributed_mooncake": "/mistralai/Mistral-Small-4-119B-2603/strategies/kv_store_distributed_mooncake.json" }, "by_hardware": { "h200": "/mistralai/Mistral-Small-4-119B-2603/hw/h200.json", "h100": "/mistralai/Mistral-Small-4-119B-2603/hw/h100.json", "b200": "/mistralai/Mistral-Small-4-119B-2603/hw/b200.json", "gb200": "/mistralai/Mistral-Small-4-119B-2603/hw/gb200.json", "b300": "/mistralai/Mistral-Small-4-119B-2603/hw/b300.json", "gb300": "/mistralai/Mistral-Small-4-119B-2603/hw/gb300.json", "mi300x": "/mistralai/Mistral-Small-4-119B-2603/hw/mi300x.json", "mi325x": "/mistralai/Mistral-Small-4-119B-2603/hw/mi325x.json", "mi350x": "/mistralai/Mistral-Small-4-119B-2603/hw/mi350x.json", "mi355x": "/mistralai/Mistral-Small-4-119B-2603/hw/mi355x.json" } }, "model": { "model_id": "mistralai/Mistral-Small-4-119B-2603", "min_vllm_version": "0.20.0", "architecture": "moe", "parameter_count": "119B", "active_parameters": "6.5B", "context_length": 262144, "base_args": [ "--max-model-len", "262144" ], "base_env": {}, "install": { "pip": { "command": "uv venv\nsource .venv/bin/activate\nuv pip install -U vllm --torch-backend auto" }, "docker": { "command": "docker pull vllm/vllm-openai:latest" }, "extras": [ { "note": "Mistral tokenizer / chat-template runtime — Mistral Small 4 needs >= 1.11.0 (vLLM 0.20.1+ bundles it, pin explicitly if you hit an older cached wheel)", "command": "uv pip install -U \"mistral_common>=1.11.0\"" }, { "note": "Transformers v5 silences YaRN warnings and is required for the latest Mistral 4 chat template", "command": "uv pip install -U transformers" } ] } }, "features": { "tool_calling": { "description": "Mistral tool-call parser with automatic tool choice — emits [TOOL_CALLS] / [ARGS] from the chat template", "args": [ "--enable-auto-tool-choice", "--tool-call-parser", "mistral" ] }, "reasoning": { "description": "Mistral reasoning parser extracts [THINK]...[/THINK] into message.reasoning (emitted when reasoning_effort='high')", "args": [ "--reasoning-parser", "mistral" ] }, "spec_decoding": { "description": "EAGLE speculative decoding via the mistralai/Mistral-Small-4-119B-2603-eagle 2-layer draft head", "default_mode": "eagle", "modes": { "eagle": { "label": "EAGLE", "description": "EAGLE speculative decoding.", "args": [ "--speculative-config", "{\"model\":\"mistralai/Mistral-Small-4-119B-2603-eagle\",\"num_speculative_tokens\":3,\"method\":\"eagle\",\"max_model_len\":\"65536\"}" ] }, "dspark": { "label": "DSpark", "description": "DSpark drafting with an externally trained draft model; the served checkpoint is unchanged.", "variants": [ "default" ], "args": [ "--speculative-config", "{\"model\": \"RedHatAI/Mistral-Small-4-119B-2603-speculator.dspark\", \"num_speculative_tokens\": 7, \"method\": \"dspark\"}" ] }, "dflash": { "label": "DFlash", "description": "DFlash speculative decoding with an externally trained draft model; the served checkpoint is unchanged.", "variants": [ "default" ], "args": [ "--speculative-config", "{\"model\": \"RedHatAI/Mistral-Small-4-119B-2603.dflash\", \"num_speculative_tokens\": 7, \"method\": \"dflash\"}" ] } } } }, "opt_in_features": [ "spec_decoding" ], "variants": { "default": { "precision": "fp8", "vram_minimum_gb": 143, "description": "Native FP8 E4M3 weights (vision tower / projector / lm_head kept in BF16); recommended on 2xB200/H200 or MI300X" }, "nvfp4": { "model_id": "mistralai/Mistral-Small-4-119B-2603-NVFP4", "precision": "nvfp4", "vram_minimum_gb": 72, "description": "NVFP4 4-bit weights for B200-class GPUs (Marlin fallback on Hopper)", "json": "/mistralai/Mistral-Small-4-119B-2603-NVFP4.json" }, "redhat_nvfp4": { "model_id": "RedHatAI/Mistral-Small-4-119B-2603-NVFP4", "precision": "nvfp4", "label": "NVFP4 (RedHat)", "vram_minimum_gb": 72, "description": "RedHatAI NVFP4 quantization", "json": "/RedHatAI/Mistral-Small-4-119B-2603-NVFP4.json" } }, "hardware_overrides": { "amd": { "extra_args": [ "--no-enable-prefix-caching" ], "extra_env": { "VLLM_ROCM_USE_AITER": "1", "SAFETENSORS_FAST_GPU": "1" } } }, "guide": "## Overview\n\nMistral Small 4 119B is a hybrid Mixture-of-Experts model from Mistral AI:\n**128 experts, 4 active per token** (plus 1 shared expert), 119B total\nparameters with **6.5B activated per token**. It unifies the capabilities of\nthree earlier Mistral families — **Instruct**, **Reasoning** (formerly\nMagistral), and **Devstral** — into a single checkpoint, with per-request\ntoggling between fast instant-reply and step-by-step reasoning via\n`reasoning_effort`.\n\nThe weights ship pre-quantized to FP8 E4M3 (the vision tower, multimodal\nprojector, and `lm_head` are kept in BF16). Multimodal input accepts text\nand image (Pixtral-style encoder, 1540x1540, patch size 14). Context length\nis **256K** via YaRN scaling (factor 128x over the 8K base) — the config\nadvertises 1M positions but Mistral recommends serving at 256K.\n\nMistral Small 4 also ships two companion checkpoints:\n\n- **NVFP4** ([`mistralai/Mistral-Small-4-119B-2603-NVFP4`](https://huggingface.co/mistralai/Mistral-Small-4-119B-2603-NVFP4)) —\n 4-bit `compressed-tensors` weights (~72 GB raw).\n- **EAGLE draft head** ([`mistralai/Mistral-Small-4-119B-2603-eagle`](https://huggingface.co/mistralai/Mistral-Small-4-119B-2603-eagle)) —\n a 2-layer Mistral-style draft trained on the 119B target; enable via the\n `spec_decoding` feature.\n\n## Prerequisites\n\n- Hardware: 2xB200 / 2xH200 / 2xMI300X (FP8 default) or 1xB200 with reduced\n context (NVFP4).\n- vLLM **>= 0.20.0** — earlier releases load the model but trip on the\n Mistral tool-call / reasoning parsers fixed in PR #39217 (in 0.19.1+)\n and the grammar factory landed in 0.20.0.\n\n### Install vLLM\n\n```bash\nuv venv && source .venv/bin/activate\nuv pip install -U vllm --torch-backend=auto\nuv pip install -U \"mistral_common>=1.11.0\" transformers\n```\n\nVerify with `python -c \"import mistral_common; print(mistral_common.__version__)\"`.\n\n## Launch command\n\nFP8 default (2xB200 / 2xH200):\n\n```bash\nvllm serve mistralai/Mistral-Small-4-119B-2603 \\\n --max-model-len 262144 \\\n --tensor-parallel-size 2 \\\n --tool-call-parser mistral --enable-auto-tool-choice \\\n --reasoning-parser mistral \\\n --max-num-batched-tokens 16384 --max-num-seqs 128 \\\n --gpu-memory-utilization 0.8\n```\n\nNVFP4 variant (single B200, or 2xB200 for full 256K context):\n\n```bash\nvllm serve mistralai/Mistral-Small-4-119B-2603-NVFP4 \\\n --max-model-len 262144 \\\n --tensor-parallel-size 2 \\\n --tool-call-parser mistral --enable-auto-tool-choice \\\n --reasoning-parser mistral \\\n --max-num-batched-tokens 16384 --max-num-seqs 128 \\\n --gpu-memory-utilization 0.8\n```\n\nMistral publishes a custom Docker image\n[`mistralllm/vllm-ms4:latest`](https://hub.docker.com/r/mistralllm/vllm-ms4)\nwith patched tool-call / reasoning parsing — use it if you're pinned to a\nvLLM version below 0.20.0 and hit either of those issues.\n\n## EAGLE speculative decoding\n\nThe EAGLE draft head is **not** included in the default config — toggle the\n`spec_decoding` feature (or pass `--speculative-config` directly):\n\n```bash\nvllm serve mistralai/Mistral-Small-4-119B-2603 \\\n --max-model-len 262144 \\\n --tensor-parallel-size 2 \\\n --tool-call-parser mistral --enable-auto-tool-choice \\\n --reasoning-parser mistral \\\n --max-num-batched-tokens 16384 --max-num-seqs 128 \\\n --gpu-memory-utilization 0.8 \\\n --speculative-config '{\n \"model\": \"mistralai/Mistral-Small-4-119B-2603-eagle\",\n \"num_speculative_tokens\": 3,\n \"method\": \"eagle\",\n \"max_model_len\": \"65536\"\n }'\n```\n\n## Client usage\n\nReasoning is opt-in per request via `reasoning_effort`. Only `\"none\"` and\n`\"high\"` are accepted — anything else raises an exception in the chat\ntemplate. Recommended sampling:\n\n- `reasoning_effort=\"none\"`: `temperature` 0.0–0.7 depending on task.\n- `reasoning_effort=\"high\"`: `temperature=0.7`.\n\n```python\nfrom openai import OpenAI\n\nclient = OpenAI(api_key=\"EMPTY\", base_url=\"http://localhost:8000/v1\")\nresp = client.chat.completions.create(\n model=\"mistralai/Mistral-Small-4-119B-2603\",\n messages=[{\"role\": \"user\", \"content\": \"Plan a 3-day Paris trip.\"}],\n extra_body={\"reasoning_effort\": \"high\"},\n temperature=0.7, max_tokens=4096,\n)\nmsg = resp.choices[0].message\nprint(\"reasoning:\", msg.reasoning)\nprint(\"answer:\", msg.content)\n```\n\nImage input (vision):\n\n```python\nresp = client.chat.completions.create(\n model=\"mistralai/Mistral-Small-4-119B-2603\",\n messages=[{\n \"role\": \"user\",\n \"content\": [\n {\"type\": \"image_url\", \"image_url\": {\"url\": \"https://...\"}},\n {\"type\": \"text\", \"text\": \"Describe this image.\"},\n ],\n }],\n max_tokens=512,\n)\n```\n\nTool calling follows the standard OpenAI schema — the chat template emits\n`[AVAILABLE_TOOLS]` / `[TOOL_CALLS]` tokens which the `mistral` tool-call\nparser surfaces as `message.tool_calls`.\n\n## Troubleshooting\n\n- **OOM at full 256K context** on 2xH200: drop `--max-model-len` to 131072\n or 65536, or move to NVFP4.\n- **`reasoning_effort` rejected**: the chat template only accepts `\"none\"`\n and `\"high\"`.\n- **NVFP4 weight-loader errors** on older wheels: try Mistral's\n `mistralllm/vllm-ms4:latest` Docker image, which carries the parser /\n weight-loader fixes Mistral ships ahead of the upstream merge.\n\n## References\n\n- [Model card](https://huggingface.co/mistralai/Mistral-Small-4-119B-2603)\n- [NVFP4 variant](https://huggingface.co/mistralai/Mistral-Small-4-119B-2603-NVFP4)\n- [EAGLE draft head](https://huggingface.co/mistralai/Mistral-Small-4-119B-2603-eagle)\n", "source": { "source": "vllm", "repo": "vllm-project/vllm", "ref": "main", "sha": "564ac0de1b228568cafcbfbe8ccaccc82d6023eb", "path": "recipes/models/mistralai/Mistral-Small-4-119B-2603.yaml", "edit_url": "https://github.com/vllm-project/vllm/edit/main/recipes/models/mistralai/Mistral-Small-4-119B-2603.yaml", "overlays": [] } }