{
  "hf_id": "nvidia/NVIDIA-Nemotron-Nano-12B-v2-VL-BF16",
  "meta": {
    "title": "NVIDIA Nemotron-Nano-12B-v2-VL",
    "slug": "nemotron-nano-12b-v2-vl",
    "provider": "NVIDIA",
    "description": "NVIDIA Nemotron-Nano 12B vision-language model with video support and Efficient Video Sampling (EVS)",
    "date_added": "2025-10-31",
    "date_updated": "2026-04-17",
    "tasks": [
      "multimodal"
    ],
    "related_recipes": [
      "nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16"
    ],
    "hardware": {
      "h100": "verified",
      "b200": "verified"
    }
  },
  "recommended_command": {
    "hardware": "h200",
    "strategy": "single_node_tp",
    "variant": "default",
    "node_count": 1,
    "deploy_type": "single_node",
    "env": {
      "VLLM_VIDEO_LOADER_BACKEND": "opencv"
    },
    "docker_image": "vllm/vllm-openai:latest",
    "command": "vllm serve nvidia/NVIDIA-Nemotron-Nano-12B-v2-VL-BF16 \\\n  --trust-remote-code \\\n  --tensor-parallel-size 1 \\\n  --video-pruning-rate 0.75",
    "argv": [
      "vllm",
      "serve",
      "nvidia/NVIDIA-Nemotron-Nano-12B-v2-VL-BF16",
      "--trust-remote-code",
      "--tensor-parallel-size",
      "1",
      "--video-pruning-rate",
      "0.75"
    ],
    "docker_command": "docker run --gpus all \\\n  --privileged --ipc=host -p 8000:8000 \\\n  -v ~/.cache/huggingface:/root/.cache/huggingface \\\n  -e VLLM_VIDEO_LOADER_BACKEND=opencv \\\n  vllm/vllm-openai:latest nvidia/NVIDIA-Nemotron-Nano-12B-v2-VL-BF16 \\\n  --trust-remote-code \\\n  --tensor-parallel-size 1 \\\n  --video-pruning-rate 0.75",
    "docker_argv": [
      "docker",
      "run",
      "--gpus",
      "all",
      "--privileged",
      "--ipc=host",
      "-p",
      "8000:8000",
      "-v",
      "~/.cache/huggingface:/root/.cache/huggingface",
      "-e",
      "VLLM_VIDEO_LOADER_BACKEND=opencv",
      "vllm/vllm-openai:latest",
      "nvidia/NVIDIA-Nemotron-Nano-12B-v2-VL-BF16",
      "--trust-remote-code",
      "--tensor-parallel-size",
      "1",
      "--video-pruning-rate",
      "0.75"
    ],
    "strategy_spec": {
      "name": "single_node_tp",
      "deploy_type": "single_node",
      "display_name": "Tensor Parallel",
      "short_label": "TP",
      "orientation": "latency",
      "description": "Single-node tensor parallel. Splits the model across all local GPUs. TP size is set to the GPU count at deploy time. The simplest multi-GPU strategy — works for all model architectures.\n",
      "hardware_match": {
        "min_gpus": 1,
        "max_gpus": 8,
        "multi_node": false
      },
      "vllm_args": [],
      "parallel_flag": "--tensor-parallel-size"
    },
    "hardware_profile": {
      "brand": "NVIDIA",
      "generation": "hopper",
      "compute_arch": "sm90",
      "display_name": "H200",
      "description": "NVIDIA H200 SXM 141 GB HBM3e · 8-GPU HGX node",
      "gpu_count": 8,
      "vram_gb": 1128,
      "multi_node": false
    },
    "alternatives": {
      "multi_node_tp": "/nvidia/NVIDIA-Nemotron-Nano-12B-v2-VL-BF16/strategies/multi_node_tp.json",
      "multi_node_tp_pp": "/nvidia/NVIDIA-Nemotron-Nano-12B-v2-VL-BF16/strategies/multi_node_tp_pp.json",
      "kv_store_centralized_mooncake": "/nvidia/NVIDIA-Nemotron-Nano-12B-v2-VL-BF16/strategies/kv_store_centralized_mooncake.json",
      "kv_store_distributed_mooncake": "/nvidia/NVIDIA-Nemotron-Nano-12B-v2-VL-BF16/strategies/kv_store_distributed_mooncake.json"
    },
    "by_hardware": {
      "h200": "/nvidia/NVIDIA-Nemotron-Nano-12B-v2-VL-BF16/hw/h200.json",
      "h100": "/nvidia/NVIDIA-Nemotron-Nano-12B-v2-VL-BF16/hw/h100.json",
      "b200": "/nvidia/NVIDIA-Nemotron-Nano-12B-v2-VL-BF16/hw/b200.json",
      "gb200": "/nvidia/NVIDIA-Nemotron-Nano-12B-v2-VL-BF16/hw/gb200.json",
      "b300": "/nvidia/NVIDIA-Nemotron-Nano-12B-v2-VL-BF16/hw/b300.json",
      "gb300": "/nvidia/NVIDIA-Nemotron-Nano-12B-v2-VL-BF16/hw/gb300.json",
      "mi300x": "/nvidia/NVIDIA-Nemotron-Nano-12B-v2-VL-BF16/hw/mi300x.json",
      "mi325x": "/nvidia/NVIDIA-Nemotron-Nano-12B-v2-VL-BF16/hw/mi325x.json",
      "mi350x": "/nvidia/NVIDIA-Nemotron-Nano-12B-v2-VL-BF16/hw/mi350x.json",
      "mi355x": "/nvidia/NVIDIA-Nemotron-Nano-12B-v2-VL-BF16/hw/mi355x.json"
    }
  },
  "model": {
    "model_id": "nvidia/NVIDIA-Nemotron-Nano-12B-v2-VL-BF16",
    "min_vllm_version": "0.11.1",
    "architecture": "dense",
    "parameter_count": "12B",
    "active_parameters": "12B",
    "context_length": 131072,
    "base_args": [
      "--trust-remote-code"
    ],
    "base_env": {
      "VLLM_VIDEO_LOADER_BACKEND": "opencv"
    },
    "install": {
      "pip": {
        "command": "uv venv\nsource .venv/bin/activate\nuv pip install -U vllm --torch-backend auto"
      },
      "docker": {
        "command": "docker pull vllm/vllm-openai:latest"
      }
    }
  },
  "features": {
    "video_compression": {
      "description": "Efficient Video Sampling (EVS) prunes video tokens; 0.75 means 75% pruning",
      "args": [
        "--video-pruning-rate",
        "0.75"
      ]
    },
    "text_only": {
      "description": "Skip loading the vision encoder for text-only workloads — frees VRAM for KV cache. Mutually exclusive with encoder_parallel.",
      "args": [
        "--language-model-only"
      ]
    },
    "encoder_parallel": {
      "description": "Run the vision encoder in data-parallel mode — avoids TP comm overhead on the small encoder. Mutually exclusive with text_only.",
      "args": [
        "--mm-encoder-tp-mode",
        "data"
      ]
    }
  },
  "opt_in_features": [
    "text_only",
    "encoder_parallel"
  ],
  "variants": {
    "default": {
      "precision": "bf16",
      "vram_minimum_gb": 29,
      "description": "BF16 weights on 1 GPU"
    },
    "fp8": {
      "model_id": "nvidia/NVIDIA-Nemotron-Nano-12B-v2-VL-FP8",
      "precision": "fp8",
      "vram_minimum_gb": 14,
      "description": "FP8 weights on 1 GPU",
      "json": "/nvidia/NVIDIA-Nemotron-Nano-12B-v2-VL-FP8.json"
    },
    "nvfp4": {
      "model_id": "nvidia/NVIDIA-Nemotron-Nano-12B-v2-VL-NVFP4-QAD",
      "precision": "nvfp4",
      "vram_minimum_gb": 8,
      "description": "NVFP4 (QAD) weights for Blackwell",
      "json": "/nvidia/NVIDIA-Nemotron-Nano-12B-v2-VL-NVFP4-QAD.json"
    }
  },
  "kv_offload_support": {
    "offloading_cpu": "verified",
    "offloading_fs": "verified"
  },
  "hardware_overrides": {},
  "guide": "## Overview\n\nNemotron-Nano-12B-v2-VL is a vision-language model with image and video support. It\nincludes Efficient Video Sampling (EVS) to prune video tokens and reduce compute. The\nmodel is available in BF16, FP8, and NVFP4 (QAD) precisions.\n\n## Prerequisites\n\n- Hardware: 1x GPU (A100/H100/B200, etc.)\n- vLLM: 0.11.0 does NOT include this model; use latest nightly or install from source\n- DGX Spark: use `nvcr.io/nvidia/vllm:25.12.post1-py3`\n\n### Install vLLM\n\n```bash\ndocker pull vllm/vllm-openai:nightly-8bff831f0aa239006f34b721e63e1340e3472067\n# or for DGX Spark:\ndocker pull nvcr.io/nvidia/vllm:25.12.post1-py3\n```\n\n## Launch command\n\n```bash\nexport VLLM_VIDEO_LOADER_BACKEND=opencv\nexport CHECKPOINT_PATH=\"nvidia/NVIDIA-Nemotron-Nano-12B-v2-VL-BF16\"\nexport CUDA_VISIBLE_DEVICES=0\n\npython3 -m vllm.entrypoints.openai.api_server \\\n  --model ${CHECKPOINT_PATH} \\\n  --trust-remote-code \\\n  --media-io-kwargs '{\"video\": {\"fps\": 2, \"num_frames\": 128}}' \\\n  --max-model-len 131072 \\\n  --data-parallel-size 1 \\\n  --port 5566 \\\n  --allowed-local-media-path / \\\n  --video-pruning-rate 0.75 \\\n  --served-model-name \"nvidia/NVIDIA-Nemotron-Nano-12B-v2-VL-BF16\"\n```\n\nFlags:\n- `--max-model-len`: reduce for shorter contexts to save memory\n- `--allowed-local-media-path <root>`: limit local-file access\n- `--video-pruning-rate <0..1>`: EVS compression; higher prunes more video tokens\n\n## Client Usage\n\nDescribe a video:\n\n```python\nfrom openai import OpenAI\n\nclient = OpenAI(base_url=\"http://localhost:5566/v1\", api_key=\"<ignored>\")\ncompletion = client.chat.completions.create(\n    model=\"nvidia/NVIDIA-Nemotron-Nano-12B-v2-VL-BF16\",\n    messages=[{\n        \"role\": \"user\",\n        \"content\": [\n            {\"type\": \"text\", \"text\": \"Describe the video.\"},\n            {\"type\": \"video_url\", \"video_url\": {\"url\": \"file:///path/to/video.mp4\"}},\n        ],\n    }],\n)\nprint(completion.choices[0].message.content)\n```\n\n## Offline / LLM API\n\n```python\nfrom vllm import LLM, SamplingParams\n\nllm = LLM(\n    \"nvidia/NVIDIA-Nemotron-Nano-12B-v2-VL-BF16\",\n    trust_remote_code=True,\n    max_model_len=2**17,\n    allowed_local_media_path=\"/\",\n    video_pruning_rate=0.75,\n    media_io_kwargs=dict(video=dict(fps=2, num_frames=128)),\n)\n```\n\n## Troubleshooting\n\n- Set `VLLM_VIDEO_LOADER_BACKEND=opencv` (required for video inputs).\n- OOM: lower `--max-model-len` or increase `--video-pruning-rate`.\n\n## References\n\n- [Nemotron-Nano-12B-v2-VL-BF16](https://huggingface.co/nvidia/NVIDIA-Nemotron-Nano-12B-v2-VL-BF16)\n- [Nemotron-Nano-12B-v2-VL-FP8](https://huggingface.co/nvidia/NVIDIA-Nemotron-Nano-12B-v2-VL-FP8)\n- [Nemotron-Nano-12B-v2-VL-NVFP4-QAD](https://huggingface.co/nvidia/NVIDIA-Nemotron-Nano-12B-v2-VL-NVFP4-QAD)\n- [EVS paper](https://arxiv.org/abs/2510.14624)\n",
  "source": {
    "source": "vllm",
    "repo": "vllm-project/vllm",
    "ref": "main",
    "sha": "564ac0de1b228568cafcbfbe8ccaccc82d6023eb",
    "path": "recipes/models/nvidia/NVIDIA-Nemotron-Nano-12B-v2-VL-BF16.yaml",
    "edit_url": "https://github.com/vllm-project/vllm/edit/main/recipes/models/nvidia/NVIDIA-Nemotron-Nano-12B-v2-VL-BF16.yaml",
    "overlays": []
  }
}