# nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16 — vLLM serving recipe > NVIDIA Nemotron-3-Nano Mamba-hybrid MoE (30B total / ~3B active) with BF16 and FP8 variants Configuration: hardware `h200` (8× H200) · strategy `single_node_tp` (Tensor Parallel) · variant `default` (BF16) · features `tool_calling`, `reasoning` - Interactive page: https://recipes.vllm.ai/nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16 - This configuration as JSON: https://recipes.vllm.ai/nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16.json?hardware=h200 The commands below were rendered for exactly this configuration and take precedence over any example in the guide. ## Other configurations Change the query string (the same parameters as the page URL) and fetch this URL again. Unknown or incompatible values fall back to the recipe default, so check the Configuration line. - `hardware`: `h100`, `h200`, `b200`, `gb200`, `b300`, `gb300`, `mi300x`, `mi325x`, `mi350x`, `mi355x`, `arc_pro_b60`, `arc_pro_b70` - `variant`: `default`, `fp8` - `strategy`: `single_node_tp`, `multi_node_tp`, `multi_node_tp_pp`, `multi_node_tep`, `multi_node_dep` (multi-node strategies also need `nodes=`) - `features`: comma-separated list of `tool_calling`, `reasoning`, `long_context` (default: `tool_calling`, `reasoning`) ## Install Requires vLLM >= 0.17.0. ```bash uv venv source .venv/bin/activate uv pip install -U vllm --torch-backend auto ``` Docker image: `vllm/vllm-openai:latest` ## Serve ```bash vllm serve nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16 \ --trust-remote-code \ --async-scheduling \ --kv-cache-dtype auto \ --tensor-parallel-size 1 \ --enable-auto-tool-choice \ --tool-call-parser qwen3_coder \ --reasoning-parser-plugin nano_v3_reasoning_parser.py \ --reasoning-parser nano_v3 ``` Docker equivalent: ```bash docker run --gpus all \ --privileged --ipc=host -p 8000:8000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ vllm/vllm-openai:latest nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16 \ --trust-remote-code \ --async-scheduling \ --kv-cache-dtype auto \ --tensor-parallel-size 1 \ --enable-auto-tool-choice \ --tool-call-parser qwen3_coder \ --reasoning-parser-plugin nano_v3_reasoning_parser.py \ --reasoning-parser nano_v3 ``` ## Guide > Background reference. Commands in it are examples and may target another configuration — use the Serve section above. ### Overview NVIDIA Nemotron-3-Nano-30B-A3B is a hybrid-Mamba MoE model (30B total, ~3B active) with FP8 and BF16 variants. It supports DGX Spark and Jetson Thor in addition to standard Hopper/Blackwell servers. ### Prerequisites - Hardware: 1x H100/H200 or comparable; DGX Spark, Jetson Thor, and Intel Arc Pro B60/B70 supported - vLLM >= 0.17.0 (0.28.0 recommended for full support) - Docker with NVIDIA Container Toolkit (recommended) #### Pull Docker Image ```bash docker pull --platform linux/amd64 vllm/vllm-openai:v0.28.0 docker tag vllm/vllm-openai:v0.28.0 vllm/vllm-openai:deploy ``` DGX Spark users can build from source (see README) or use the NGC image: ```bash docker pull nvcr.io/nvidia/vllm:25.12.post1-py3 ``` Jetson Thor: ```bash docker pull ghcr.io/nvidia-ai-iot/vllm:latest-jetson-thor ``` Intel XPU (B60 / B70): ```bash docker pull vllm/vllm-openai-xpu:latest ``` ### Launch commands FP8 with FlashInfer MoE backend (Blackwell/Hopper): ```bash vllm serve nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-FP8 \ --trust-remote-code \ --async-scheduling \ --kv-cache-dtype fp8 \ --tensor-parallel-size 1 \ --moe-backend flashinfer_cutlass ``` BF16 (with reasoning + tool parsers — typical for Spark/Thor): ```bash wget https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16/resolve/main/nano_v3_reasoning_parser.py vllm serve nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16 \ --max-num-seqs 8 \ --tensor-parallel-size 1 \ --max-model-len 262144 \ --trust-remote-code \ --enable-auto-tool-choice \ --tool-call-parser qwen3_coder \ --reasoning-parser-plugin nano_v3_reasoning_parser.py \ --reasoning-parser nano_v3 ``` #### Intel XPU (B60, B70) Validated on 4× Intel Arc Pro B60 / B70 (B60 24 GB, B70 32 GB per card) with the official vLLM XPU image `vllm/vllm-openai-xpu:latest`. FP8: ```bash docker run --device /dev/dri \ -v /dev/dri/by-path:/dev/dri/by-path --shm-size=16g \ --privileged --ipc=host -p 8000:8000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ vllm/vllm-openai-xpu:latest \ nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-FP8 \ --trust-remote-code \ --async-scheduling \ --kv-cache-dtype fp8 \ --tensor-parallel-size 4 \ --enforce-eager \ --max-model-len 8192 ``` BF16: ```bash docker run --device /dev/dri \ -v /dev/dri/by-path:/dev/dri/by-path --shm-size=16g \ --privileged --ipc=host -p 8000:8000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ vllm/vllm-openai-xpu:latest \ nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16 \ --trust-remote-code \ --async-scheduling \ --kv-cache-dtype auto \ --tensor-parallel-size 4 \ --enforce-eager \ --enable-auto-tool-choice \ --tool-call-parser qwen3_coder \ --reasoning-parser-plugin nano_v3_reasoning_parser.py \ --reasoning-parser nano_v3 \ --max-model-len 8192 ``` Key flags: - `kv-cache-dtype fp8` for FP8 variant, `auto` for BF16 - `async-scheduling` reduces host overhead between decode steps - `mamba-ssm-cache-dtype float32` for best accuracy, `float16` for speed - `max-num-seqs` cap to match client concurrency for lower per-user latency ### Benchmarking ```bash vllm bench serve \ --model nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-FP8 \ --trust-remote-code \ --dataset-name random \ --random-input-len 1024 --random-output-len 1024 \ --num-warmups 20 \ --ignore-eos \ --max-concurrency 1024 \ --num-prompts 2048 ``` ### Troubleshooting - Use `--kv-cache-dtype fp8` only with the FP8 checkpoint. - Balance TP and `--max-num-seqs` for throughput vs. per-user latency. ### References - [Nemotron-3-Nano-30B-A3B-BF16](https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16) - [Nemotron-3-Nano-30B-A3B-FP8](https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-FP8)