# nvidia/NVIDIA-Nemotron-Nano-12B-v2-VL-BF16 — vLLM serving recipe

> NVIDIA Nemotron-Nano 12B vision-language model with video support and Efficient Video Sampling (EVS)

Configuration: hardware `h200` (8× H200) · strategy `single_node_tp` (Tensor Parallel) · variant `default` (BF16) · features `video_compression`

- Interactive page: https://recipes.vllm.ai/nvidia/NVIDIA-Nemotron-Nano-12B-v2-VL-BF16
- This configuration as JSON: https://recipes.vllm.ai/nvidia/NVIDIA-Nemotron-Nano-12B-v2-VL-BF16.json?hardware=h200

The commands below were rendered for exactly this configuration and take precedence over any example in the guide.

## Other configurations

Change the query string (the same parameters as the page URL) and fetch this URL again. Unknown or incompatible values fall back to the recipe default, so check the Configuration line.

- `hardware`: `h100`, `h200`, `b200`, `gb200`, `b300`, `gb300`, `mi300x`, `mi325x`, `mi350x`, `mi355x`
- `variant`: `default`, `fp8`, `nvfp4`
- `strategy`: `single_node_tp`, `multi_node_tp`, `multi_node_tp_pp` (multi-node strategies also need `nodes=<n>`)
- `features`: comma-separated list of `video_compression`, `text_only`, `encoder_parallel` (default: `video_compression`)

## Install

Requires vLLM >= 0.11.1.

```bash
uv venv
source .venv/bin/activate
uv pip install -U vllm --torch-backend auto
```

Docker image: `vllm/vllm-openai:latest`

## Serve

```bash
export VLLM_VIDEO_LOADER_BACKEND=opencv
vllm serve nvidia/NVIDIA-Nemotron-Nano-12B-v2-VL-BF16 \
  --trust-remote-code \
  --tensor-parallel-size 1 \
  --video-pruning-rate 0.75
```

Docker equivalent:

```bash
docker run --gpus all \
  --privileged --ipc=host -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -e VLLM_VIDEO_LOADER_BACKEND=opencv \
  vllm/vllm-openai:latest nvidia/NVIDIA-Nemotron-Nano-12B-v2-VL-BF16 \
  --trust-remote-code \
  --tensor-parallel-size 1 \
  --video-pruning-rate 0.75
```

## Guide

> Background reference. Commands in it are examples and may target another configuration — use the Serve section above.

### Overview

Nemotron-Nano-12B-v2-VL is a vision-language model with image and video support. It
includes Efficient Video Sampling (EVS) to prune video tokens and reduce compute. The
model is available in BF16, FP8, and NVFP4 (QAD) precisions.

### Prerequisites

- Hardware: 1x GPU (A100/H100/B200, etc.)
- vLLM: 0.11.0 does NOT include this model; use latest nightly or install from source
- DGX Spark: use `nvcr.io/nvidia/vllm:25.12.post1-py3`

#### Install vLLM

```bash
docker pull vllm/vllm-openai:nightly-8bff831f0aa239006f34b721e63e1340e3472067
# or for DGX Spark:
docker pull nvcr.io/nvidia/vllm:25.12.post1-py3
```

### Launch command

```bash
export VLLM_VIDEO_LOADER_BACKEND=opencv
export CHECKPOINT_PATH="nvidia/NVIDIA-Nemotron-Nano-12B-v2-VL-BF16"
export CUDA_VISIBLE_DEVICES=0

python3 -m vllm.entrypoints.openai.api_server \
  --model ${CHECKPOINT_PATH} \
  --trust-remote-code \
  --media-io-kwargs '{"video": {"fps": 2, "num_frames": 128}}' \
  --max-model-len 131072 \
  --data-parallel-size 1 \
  --port 5566 \
  --allowed-local-media-path / \
  --video-pruning-rate 0.75 \
  --served-model-name "nvidia/NVIDIA-Nemotron-Nano-12B-v2-VL-BF16"
```

Flags:
- `--max-model-len`: reduce for shorter contexts to save memory
- `--allowed-local-media-path <root>`: limit local-file access
- `--video-pruning-rate <0..1>`: EVS compression; higher prunes more video tokens

### Client Usage

Describe a video:

```python
from openai import OpenAI

client = OpenAI(base_url="http://localhost:5566/v1", api_key="<ignored>")
completion = client.chat.completions.create(
    model="nvidia/NVIDIA-Nemotron-Nano-12B-v2-VL-BF16",
    messages=[{
        "role": "user",
        "content": [
            {"type": "text", "text": "Describe the video."},
            {"type": "video_url", "video_url": {"url": "file:///path/to/video.mp4"}},
        ],
    }],
)
print(completion.choices[0].message.content)
```

### Offline / LLM API

```python
from vllm import LLM, SamplingParams

llm = LLM(
    "nvidia/NVIDIA-Nemotron-Nano-12B-v2-VL-BF16",
    trust_remote_code=True,
    max_model_len=2**17,
    allowed_local_media_path="/",
    video_pruning_rate=0.75,
    media_io_kwargs=dict(video=dict(fps=2, num_frames=128)),
)
```

### Troubleshooting

- Set `VLLM_VIDEO_LOADER_BACKEND=opencv` (required for video inputs).
- OOM: lower `--max-model-len` or increase `--video-pruning-rate`.

### References

- [Nemotron-Nano-12B-v2-VL-BF16](https://huggingface.co/nvidia/NVIDIA-Nemotron-Nano-12B-v2-VL-BF16)
- [Nemotron-Nano-12B-v2-VL-FP8](https://huggingface.co/nvidia/NVIDIA-Nemotron-Nano-12B-v2-VL-FP8)
- [Nemotron-Nano-12B-v2-VL-NVFP4-QAD](https://huggingface.co/nvidia/NVIDIA-Nemotron-Nano-12B-v2-VL-NVFP4-QAD)
- [EVS paper](https://arxiv.org/abs/2510.14624)