# nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16 — vLLM serving recipe

> Mamba2-Transformer hybrid MoE omnimodal model (31B total / 3B active) with unified video, audio, image, and text understanding; reasoning + tool calling; BF16, FP8, and NVFP4 variants

Configuration: hardware `h200` (8× H200) · strategy `single_node_tp` (Tensor Parallel) · variant `default` (BF16) · features `tool_calling`, `reasoning`

- Interactive page: https://recipes.vllm.ai/nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16
- This configuration as JSON: https://recipes.vllm.ai/nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16.json?hardware=h200

The commands below were rendered for exactly this configuration and take precedence over any example in the guide.

## Other configurations

Change the query string (the same parameters as the page URL) and fetch this URL again. Unknown or incompatible values fall back to the recipe default, so check the Configuration line.

- `hardware`: `h100`, `h200`, `b200`, `gb200`, `b300`, `gb300`, `mi300x`, `mi325x`, `mi350x`, `mi355x`
- `variant`: `default`, `fp8`, `nvfp4`
- `strategy`: `single_node_tp`, `multi_node_tp` (multi-node strategies also need `nodes=<n>`)
- `features`: comma-separated list of `tool_calling`, `reasoning` (default: `tool_calling`, `reasoning`)

## Install

Requires vLLM >= 0.20.0.

Pinned to 0.28.0 with the audio extra (required for audio + use_audio_in_video).

```bash
uv pip install "vllm[audio]==0.28.0"
```

Docker image: `vllm/vllm-openai:latest`

## Serve

```bash
vllm serve nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16 \
  --trust-remote-code \
  --max-model-len 131072 \
  --media-io-kwargs '{"video": {"num_frames": 512, "fps": 1}}' \
  --video-pruning-rate 0.5 \
  --tensor-parallel-size 1 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --reasoning-parser nemotron_v3
```

Docker equivalent:

```bash
docker run --gpus all \
  --privileged --ipc=host -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai:latest nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16 \
  --trust-remote-code \
  --max-model-len 131072 \
  --media-io-kwargs '{"video": {"num_frames": 512, "fps": 1}}' \
  --video-pruning-rate 0.5 \
  --tensor-parallel-size 1 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --reasoning-parser nemotron_v3
```

## Guide

> Background reference. Commands in it are examples and may target another configuration — use the Serve section above.

### Overview

NVIDIA Nemotron-3-Nano-Omni-30B-A3B-Reasoning is a Mamba2-Transformer hybrid
MoE omnimodal model (31B total / 3B active) that unifies video, audio, image,
and text understanding. It is built on the Nemotron-3-Nano-30B-A3B LLM
backbone with a CRADIO v4-H vision encoder and a Parakeet speech encoder, and
ships in BF16, ModelOpt FP8, and ModelOpt NVFP4 variants.

Capabilities:
- Video (mp4, up to 2 minutes, sampled at 1–2 FPS / 128–256 frames)
- Audio (wav, mp3, up to 1 hour, ≥8 kHz)
- Image (jpeg, png)
- Text (English, up to 256K context)
- Reasoning with chain-of-thought (`<think>` tags)
- Tool calling
- Word-level timestamps for transcription

### Prerequisites

- vLLM **0.28.0** (pinned: `pip install vllm[audio]==0.28.0`,
  or pull `vllm/vllm-openai:v0.28.0`)
- Hardware: 1× B200 / H200 / H100 (single-GPU TP1 is the documented profile)
- The `audio` extra is required for any audio input, including
  `use_audio_in_video=true`

#### Pull the Docker image

```bash
# CUDA 13:
docker pull vllm/vllm-openai:v0.28.0
# CUDA 12.9:
docker pull vllm/vllm-openai:v0.28.0-cu129
```

### Launch command

General single-GPU invocation (B200 / H200 / H100):

```bash
vllm serve nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16 \
  --served-model-name nemotron \
  --host 0.0.0.0 \
  --port 5000 \
  --tensor-parallel-size 1 \
  --max-model-len 131072 \
  --trust-remote-code \
  --video-pruning-rate 0.5 \
  --media-io-kwargs '{"video": {"num_frames": 512, "fps": 1}}' \
  --reasoning-parser nemotron_v3 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder
```

Swap the model id for the FP8 or NVFP4 checkpoint and add `--kv-cache-dtype fp8`:

```bash
vllm serve nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-NVFP4 \
  --kv-cache-dtype fp8 \
  ...
```

#### Platform-specific notes

- **RTX Pro:** append `--moe-backend triton` (FlashInfer + RTX Pro bug).
- **NVFP4 + TP>1:** append `--moe-backend flashinfer_cutlass` (TRTLLM_GEN MoE
  kernel bug at TP>1 on NVFP4).
- **DGX Spark (aarch64):** unified LPDDR5X memory; lower
  `--gpu-memory-utilization` to 0.70 and reduce `--max-model-len` (e.g. 32768)
  if you hit OOM. Use `--max-num-seqs 8`.

### Recommended sampling

| Mode | temperature | top_p | top_k | max_tokens | reasoning_budget |
|------|------------:|------:|------:|-----------:|------------------:|
| Thinking | 0.6 | 0.95 | — | 20480 | 16384 |
| Instruct | 0.2 | — | 1 | 1024 | — |

Toggle thinking mode via `chat_template_kwargs={"enable_thinking": true}`
(default) or `false` to disable.

### Client usage

```python
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="")
resp = client.chat.completions.create(
    model="nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-NVFP4",
    messages=[{"role": "user", "content": "Hello!"}],
    extra_body={"chat_template_kwargs": {"enable_thinking": True}},
)
```

### Benchmarking

```bash
vllm bench serve \
  --model nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-FP8 \
  --trust-remote-code \
  --dataset-name random \
  --random-input-len 1024 --random-output-len 1024 \
  --num-warmups 20 --ignore-eos \
  --max-concurrency 1024 --num-prompts 2048
```

### References

- [BF16 model card](https://huggingface.co/nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16)
- [FP8 model card](https://huggingface.co/nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-FP8)
- [NVFP4 model card](https://huggingface.co/nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-NVFP4)
- [Nemotron 3 Nano LLM backbone](https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16)