{ "hf_id": "nvidia/NVIDIA-Nemotron-Nano-9B-v2", "meta": { "title": "NVIDIA Nemotron-Nano-9B-v2", "slug": "nemotron-nano-9b-v2", "provider": "NVIDIA", "description": "NVIDIA Nemotron-Nano 9B (Mamba-hybrid dense) reasoning + tool-use model with FP8 / NVFP4 / Japanese variants", "date_added": "2026-04-28", "date_updated": "2026-04-28", "tasks": [ "text" ], "related_recipes": [ "nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16", "nvidia/NVIDIA-Nemotron-Nano-12B-v2-VL-BF16" ], "hardware": { "h100": "verified", "h200": "verified", "b200": "verified" } }, "recommended_command": { "hardware": "h200", "strategy": "single_node_tp", "variant": "default", "node_count": 1, "deploy_type": "single_node", "env": {}, "docker_image": "vllm/vllm-openai:latest", "command": "vllm serve nvidia/NVIDIA-Nemotron-Nano-9B-v2 \\\n --trust-remote-code \\\n --tensor-parallel-size 1 \\\n --enable-auto-tool-choice \\\n --tool-parser-plugin nemotron_toolcall_parser_no_streaming.py \\\n --tool-call-parser nemotron_json", "argv": [ "vllm", "serve", "nvidia/NVIDIA-Nemotron-Nano-9B-v2", "--trust-remote-code", "--tensor-parallel-size", "1", "--enable-auto-tool-choice", "--tool-parser-plugin", "nemotron_toolcall_parser_no_streaming.py", "--tool-call-parser", "nemotron_json" ], "docker_command": "docker run --gpus all \\\n --privileged --ipc=host -p 8000:8000 \\\n -v ~/.cache/huggingface:/root/.cache/huggingface \\\n vllm/vllm-openai:latest nvidia/NVIDIA-Nemotron-Nano-9B-v2 \\\n --trust-remote-code \\\n --tensor-parallel-size 1 \\\n --enable-auto-tool-choice \\\n --tool-parser-plugin nemotron_toolcall_parser_no_streaming.py \\\n --tool-call-parser nemotron_json", "docker_argv": [ "docker", "run", "--gpus", "all", "--privileged", "--ipc=host", "-p", "8000:8000", "-v", "~/.cache/huggingface:/root/.cache/huggingface", "vllm/vllm-openai:latest", "nvidia/NVIDIA-Nemotron-Nano-9B-v2", "--trust-remote-code", "--tensor-parallel-size", "1", "--enable-auto-tool-choice", "--tool-parser-plugin", "nemotron_toolcall_parser_no_streaming.py", "--tool-call-parser", "nemotron_json" ], "strategy_spec": { "name": "single_node_tp", "deploy_type": "single_node", "display_name": "Tensor Parallel", "short_label": "TP", "orientation": "latency", "description": "Single-node tensor parallel. Splits the model across all local GPUs. TP size is set to the GPU count at deploy time. The simplest multi-GPU strategy — works for all model architectures.\n", "hardware_match": { "min_gpus": 1, "max_gpus": 8, "multi_node": false }, "vllm_args": [], "parallel_flag": "--tensor-parallel-size" }, "hardware_profile": { "brand": "NVIDIA", "generation": "hopper", "compute_arch": "sm90", "display_name": "H200", "description": "NVIDIA H200 SXM 141 GB HBM3e · 8-GPU HGX node", "gpu_count": 8, "vram_gb": 1128, "multi_node": false }, "alternatives": { "multi_node_tp": "/nvidia/NVIDIA-Nemotron-Nano-9B-v2/strategies/multi_node_tp.json", "multi_node_tp_pp": "/nvidia/NVIDIA-Nemotron-Nano-9B-v2/strategies/multi_node_tp_pp.json", "kv_store_centralized_mooncake": "/nvidia/NVIDIA-Nemotron-Nano-9B-v2/strategies/kv_store_centralized_mooncake.json", "kv_store_distributed_mooncake": "/nvidia/NVIDIA-Nemotron-Nano-9B-v2/strategies/kv_store_distributed_mooncake.json" }, "by_hardware": { "h200": "/nvidia/NVIDIA-Nemotron-Nano-9B-v2/hw/h200.json", "h100": "/nvidia/NVIDIA-Nemotron-Nano-9B-v2/hw/h100.json", "b200": "/nvidia/NVIDIA-Nemotron-Nano-9B-v2/hw/b200.json", "gb200": "/nvidia/NVIDIA-Nemotron-Nano-9B-v2/hw/gb200.json", "b300": "/nvidia/NVIDIA-Nemotron-Nano-9B-v2/hw/b300.json", "gb300": "/nvidia/NVIDIA-Nemotron-Nano-9B-v2/hw/gb300.json", "mi300x": "/nvidia/NVIDIA-Nemotron-Nano-9B-v2/hw/mi300x.json", "mi325x": "/nvidia/NVIDIA-Nemotron-Nano-9B-v2/hw/mi325x.json", "mi350x": "/nvidia/NVIDIA-Nemotron-Nano-9B-v2/hw/mi350x.json", "mi355x": "/nvidia/NVIDIA-Nemotron-Nano-9B-v2/hw/mi355x.json" } }, "model": { "model_id": "nvidia/NVIDIA-Nemotron-Nano-9B-v2", "min_vllm_version": "0.10.1", "architecture": "dense", "parameter_count": "9B", "active_parameters": "9B", "context_length": 131072, "base_args": [ "--trust-remote-code" ], "base_env": {}, "install": { "pip": { "command": "uv venv\nsource .venv/bin/activate\nuv pip install -U vllm --torch-backend auto" }, "docker": { "command": "docker pull vllm/vllm-openai:latest" } } }, "features": { "tool_calling": { "description": "Custom Nemotron tool-call parser plugin (download: nemotron_toolcall_parser_no_streaming.py)", "args": [ "--enable-auto-tool-choice", "--tool-parser-plugin", "nemotron_toolcall_parser_no_streaming.py", "--tool-call-parser", "nemotron_json" ] } }, "opt_in_features": [], "variants": { "default": { "precision": "bf16", "vram_minimum_gb": 22, "description": "BF16 weights on 1 GPU" }, "fp8": { "model_id": "nvidia/NVIDIA-Nemotron-Nano-9B-v2-FP8", "precision": "fp8", "vram_minimum_gb": 11, "description": "FP8 weights", "json": "/nvidia/NVIDIA-Nemotron-Nano-9B-v2-FP8.json" }, "nvfp4": { "model_id": "nvidia/NVIDIA-Nemotron-Nano-9B-v2-NVFP4", "precision": "nvfp4", "vram_minimum_gb": 6, "description": "NVFP4 weights for Blackwell", "json": "/nvidia/NVIDIA-Nemotron-Nano-9B-v2-NVFP4.json" }, "base": { "model_id": "nvidia/NVIDIA-Nemotron-Nano-9B-v2-Base", "precision": "bf16", "vram_minimum_gb": 22, "description": "Pre-RL base checkpoint (BF16)", "json": "/nvidia/NVIDIA-Nemotron-Nano-9B-v2-Base.json" }, "japanese": { "model_id": "nvidia/NVIDIA-Nemotron-Nano-9B-v2-Japanese", "precision": "bf16", "vram_minimum_gb": 22, "description": "Japanese-specialized fine-tune (BF16)", "json": "/nvidia/NVIDIA-Nemotron-Nano-9B-v2-Japanese.json" }, "redhat_fp8": { "model_id": "RedHatAI/NVIDIA-Nemotron-Nano-9B-v2-FP8-dynamic", "precision": "fp8", "label": "FP8 Dynamic (RedHat)", "vram_minimum_gb": 11, "description": "RedHatAI FP8 (E4M3) with dynamic per-token activation quantization", "extra_args": [ "--kv-cache-dtype", "fp8" ], "json": "/RedHatAI/NVIDIA-Nemotron-Nano-9B-v2-FP8-dynamic.json" } }, "kv_offload_support": { "offloading_cpu": "verified", "offloading_fs": "verified" }, "hardware_overrides": {}, "guide": "## Overview\n\nNemotron-Nano-9B-v2 is a 9B Mamba-hybrid dense reasoning model that runs on a single\nH100/H200/B200. Variants ship in BF16, FP8, and NVFP4; a Japanese-specialized\nfine-tune and a pre-RL Base checkpoint are also available.\n\n## Prerequisites\n\n- Hardware: 1x H100/H200/B200 (or comparable)\n- vLLM >= 0.10.1 (`pip install -U vllm`)\n- Docker with NVIDIA Container Toolkit (recommended)\n\n## Launch command\n\n```bash\nwget https://huggingface.co/nvidia/NVIDIA-Nemotron-Nano-9B-v2/resolve/main/nemotron_toolcall_parser_no_streaming.py\n\nvllm serve nvidia/NVIDIA-Nemotron-Nano-9B-v2 \\\n --trust-remote-code \\\n --max-model-len 131072 \\\n --tensor-parallel-size 1 \\\n --enable-auto-tool-choice \\\n --tool-parser-plugin nemotron_toolcall_parser_no_streaming.py \\\n --tool-call-parser nemotron_json\n```\n\nFP8:\n\n```bash\nvllm serve nvidia/NVIDIA-Nemotron-Nano-9B-v2-FP8 \\\n --trust-remote-code \\\n --tensor-parallel-size 1\n```\n\nNVFP4 (Blackwell only):\n\n```bash\nvllm serve nvidia/NVIDIA-Nemotron-Nano-9B-v2-NVFP4 \\\n --trust-remote-code \\\n --tensor-parallel-size 1\n```\n\n## Benchmarking\n\n```bash\nvllm bench serve \\\n --model nvidia/NVIDIA-Nemotron-Nano-9B-v2 \\\n --trust-remote-code \\\n --dataset-name random \\\n --random-input-len 1024 --random-output-len 1024 \\\n --num-warmups 20 \\\n --ignore-eos \\\n --max-concurrency 256 \\\n --num-prompts 1024\n```\n\n## References\n\n- [Nemotron-Nano-9B-v2](https://huggingface.co/nvidia/NVIDIA-Nemotron-Nano-9B-v2)\n- [Nemotron-Nano-9B-v2-FP8](https://huggingface.co/nvidia/NVIDIA-Nemotron-Nano-9B-v2-FP8)\n- [Nemotron-Nano-9B-v2-NVFP4](https://huggingface.co/nvidia/NVIDIA-Nemotron-Nano-9B-v2-NVFP4)\n- [Nemotron-Nano-9B-v2-Base](https://huggingface.co/nvidia/NVIDIA-Nemotron-Nano-9B-v2-Base)\n- [Nemotron-Nano-9B-v2-Japanese](https://huggingface.co/nvidia/NVIDIA-Nemotron-Nano-9B-v2-Japanese)\n", "source": { "source": "vllm", "repo": "vllm-project/vllm", "ref": "main", "sha": "564ac0de1b228568cafcbfbe8ccaccc82d6023eb", "path": "recipes/models/nvidia/NVIDIA-Nemotron-Nano-9B-v2.yaml", "edit_url": "https://github.com/vllm-project/vllm/edit/main/recipes/models/nvidia/NVIDIA-Nemotron-Nano-9B-v2.yaml", "overlays": [] } }