# nvidia/NVIDIA-Nemotron-Nano-9B-v2 — vLLM serving recipe

> NVIDIA Nemotron-Nano 9B (Mamba-hybrid dense) reasoning + tool-use model with FP8 / NVFP4 / Japanese variants

Configuration: hardware `h200` (8× H200) · strategy `single_node_tp` (Tensor Parallel) · variant `default` (BF16) · features `tool_calling`

- Interactive page: https://recipes.vllm.ai/nvidia/NVIDIA-Nemotron-Nano-9B-v2
- This configuration as JSON: https://recipes.vllm.ai/nvidia/NVIDIA-Nemotron-Nano-9B-v2.json?hardware=h200

The commands below were rendered for exactly this configuration and take precedence over any example in the guide.

## Other configurations

Change the query string (the same parameters as the page URL) and fetch this URL again. Unknown or incompatible values fall back to the recipe default, so check the Configuration line.

- `hardware`: `h100`, `h200`, `b200`, `gb200`, `b300`, `gb300`, `mi300x`, `mi325x`, `mi350x`, `mi355x`
- `variant`: `default`, `fp8`, `nvfp4`, `base`, `japanese`, `redhat_fp8`
- `strategy`: `single_node_tp`, `multi_node_tp`, `multi_node_tp_pp` (multi-node strategies also need `nodes=<n>`)
- `features`: comma-separated list of `tool_calling` (default: `tool_calling`)

## Install

Requires vLLM >= 0.10.1.

```bash
uv venv
source .venv/bin/activate
uv pip install -U vllm --torch-backend auto
```

Docker image: `vllm/vllm-openai:latest`

## Serve

```bash
vllm serve nvidia/NVIDIA-Nemotron-Nano-9B-v2 \
  --trust-remote-code \
  --tensor-parallel-size 1 \
  --enable-auto-tool-choice \
  --tool-parser-plugin nemotron_toolcall_parser_no_streaming.py \
  --tool-call-parser nemotron_json
```

Docker equivalent:

```bash
docker run --gpus all \
  --privileged --ipc=host -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai:latest nvidia/NVIDIA-Nemotron-Nano-9B-v2 \
  --trust-remote-code \
  --tensor-parallel-size 1 \
  --enable-auto-tool-choice \
  --tool-parser-plugin nemotron_toolcall_parser_no_streaming.py \
  --tool-call-parser nemotron_json
```

## Guide

> Background reference. Commands in it are examples and may target another configuration — use the Serve section above.

### Overview

Nemotron-Nano-9B-v2 is a 9B Mamba-hybrid dense reasoning model that runs on a single
H100/H200/B200. Variants ship in BF16, FP8, and NVFP4; a Japanese-specialized
fine-tune and a pre-RL Base checkpoint are also available.

### Prerequisites

- Hardware: 1x H100/H200/B200 (or comparable)
- vLLM >= 0.10.1 (`pip install -U vllm`)
- Docker with NVIDIA Container Toolkit (recommended)

### Launch command

```bash
wget https://huggingface.co/nvidia/NVIDIA-Nemotron-Nano-9B-v2/resolve/main/nemotron_toolcall_parser_no_streaming.py

vllm serve nvidia/NVIDIA-Nemotron-Nano-9B-v2 \
  --trust-remote-code \
  --max-model-len 131072 \
  --tensor-parallel-size 1 \
  --enable-auto-tool-choice \
  --tool-parser-plugin nemotron_toolcall_parser_no_streaming.py \
  --tool-call-parser nemotron_json
```

FP8:

```bash
vllm serve nvidia/NVIDIA-Nemotron-Nano-9B-v2-FP8 \
  --trust-remote-code \
  --tensor-parallel-size 1
```

NVFP4 (Blackwell only):

```bash
vllm serve nvidia/NVIDIA-Nemotron-Nano-9B-v2-NVFP4 \
  --trust-remote-code \
  --tensor-parallel-size 1
```

### Benchmarking

```bash
vllm bench serve \
  --model nvidia/NVIDIA-Nemotron-Nano-9B-v2 \
  --trust-remote-code \
  --dataset-name random \
  --random-input-len 1024 --random-output-len 1024 \
  --num-warmups 20 \
  --ignore-eos \
  --max-concurrency 256 \
  --num-prompts 1024
```

### References

- [Nemotron-Nano-9B-v2](https://huggingface.co/nvidia/NVIDIA-Nemotron-Nano-9B-v2)
- [Nemotron-Nano-9B-v2-FP8](https://huggingface.co/nvidia/NVIDIA-Nemotron-Nano-9B-v2-FP8)
- [Nemotron-Nano-9B-v2-NVFP4](https://huggingface.co/nvidia/NVIDIA-Nemotron-Nano-9B-v2-NVFP4)
- [Nemotron-Nano-9B-v2-Base](https://huggingface.co/nvidia/NVIDIA-Nemotron-Nano-9B-v2-Base)
- [Nemotron-Nano-9B-v2-Japanese](https://huggingface.co/nvidia/NVIDIA-Nemotron-Nano-9B-v2-Japanese)