# nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16 — vLLM serving recipe > NVIDIA Nemotron-3-Super Mamba-hybrid latent-MoE (~120B total / ~12B active) with BF16, FP8, and NVFP4 variants Configuration: hardware `h200` (8× H200) · strategy `single_node_tp` (Tensor Parallel) · variant `default` (BF16) · features `tool_calling`, `reasoning` - Interactive page: https://recipes.vllm.ai/nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16 - This configuration as JSON: https://recipes.vllm.ai/nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16.json?hardware=h200 The commands below were rendered for exactly this configuration and take precedence over any example in the guide. ## Other configurations Change the query string (the same parameters as the page URL) and fetch this URL again. Unknown or incompatible values fall back to the recipe default, so check the Configuration line. - `hardware`: `h100`, `h200`, `b200`, `gb200`, `b300`, `gb300`, `dgx_station_gb300`, `dgx_spark_gb10`, `rtx_pro_6000`, `mi300x`, `mi325x`, `mi350x`, `mi355x` - `variant`: `default`, `fp8`, `nvfp4`, `base_bf16` - `strategy`: `single_node_tp`, `multi_node_tp`, `multi_node_tp_pp`, `multi_node_tep`, `multi_node_dep` (multi-node strategies also need `nodes=`) - `features`: comma-separated list of `tool_calling`, `reasoning`, `spec_decoding`, `long_context` (default: `tool_calling`, `reasoning`) ## Install Requires vLLM >= 0.17.1. ```bash uv venv source .venv/bin/activate uv pip install -U vllm --torch-backend auto ``` Docker image: `vllm/vllm-openai:latest` ## Serve ```bash vllm serve nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16 \ --trust-remote-code \ --kv-cache-dtype fp8 \ --tensor-parallel-size 8 \ --enable-auto-tool-choice \ --tool-call-parser qwen3_coder \ --reasoning-parser nemotron_v3 ``` Docker equivalent: ```bash docker run --gpus all \ --privileged --ipc=host -p 8000:8000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ vllm/vllm-openai:latest nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16 \ --trust-remote-code \ --kv-cache-dtype fp8 \ --tensor-parallel-size 8 \ --enable-auto-tool-choice \ --tool-call-parser qwen3_coder \ --reasoning-parser nemotron_v3 ``` ## Guide > Background reference. Commands in it are examples and may target another configuration — use the Serve section above. ### Overview NVIDIA Nemotron-3-Super-120B-A12B is a hybrid-Mamba latent-MoE model (~120B total, ~12B active per token) trained for general reasoning, tool use, and agentic workflows. It supports a 1M-token context window and Multi-Token Prediction (MTP). Variants ship in BF16, FP8, and NVFP4 (Blackwell). A pre-RL Base BF16 checkpoint is also available for downstream fine-tuning. ### Prerequisites - Hardware: 4-8x H100/H200/B200/RTX Pro 6000, DGX Station, or DGX Spark - vLLM >= 0.17.1 - Docker with NVIDIA Container Toolkit (recommended) ### Launch commands Reference command from the [vLLM blog](https://vllm.ai/blog/nemotron-3-super) (BF16, 4x H100, FP8 KV cache): ```bash vllm serve nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16 \ --kv-cache-dtype fp8 \ --tensor-parallel-size 4 \ --trust-remote-code \ --served-model-name nemotron \ --enable-auto-tool-choice \ --tool-call-parser qwen3_coder \ --reasoning-parser nemotron_v3 ``` FP8 weights: ```bash vllm serve nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-FP8 \ --kv-cache-dtype fp8 \ --tensor-parallel-size 4 \ --trust-remote-code \ --enable-auto-tool-choice \ --tool-call-parser qwen3_coder \ --reasoning-parser nemotron_v3 ``` NVFP4 (Blackwell only): ```bash vllm serve nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4 \ --tensor-parallel-size 2 \ --trust-remote-code \ --enable-auto-tool-choice \ --tool-call-parser qwen3_coder \ --reasoning-parser nemotron_v3 ``` #### DGX Spark (GB10) The NVFP4 variant runs on a single DGX Spark (GB10) at TP=1 inside the `vllm/vllm-openai:v0.28.0-ubuntu2404` container. The command below is the optimal config tuned end-to-end on DGX Spark: the full 262144-token context fits with an FP8 KV cache and a float32 Mamba SSM cache (`VLLM_ALLOW_LONG_MAX_MODEL_LEN=1` allows the long context, `VLLM_FLOAT32_MATMUL_PRECISION=high` keeps the Mamba scan numerically stable), `--gpu-memory-utilization 0.8` leaves headroom in the 128 GB unified memory, batching is sized for a single GB10 (`--max-num-seqs 8`, `--max-num-batched-tokens 16384`), `--load-format fastsafetensors` speeds up weight loading, and MTP speculative decoding plus prefix caching are enabled for lower latency: ```bash # Use container: vllm/vllm-openai:v0.28.0-ubuntu2404 VLLM_FLOAT32_MATMUL_PRECISION="high" \ VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 \ vllm serve nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4 \ --host 0.0.0.0 \ --port 8000 \ --max-model-len 262144 \ --gpu-memory-utilization 0.8 \ --max-num-seqs 8 \ --load-format fastsafetensors \ --attention-backend flashinfer \ --max-num-batched-tokens 16384 \ --kv-cache-dtype fp8 \ --mamba-cache-mode align \ --mamba-ssm-cache-dtype float32 \ --speculative-config '{"method":"mtp","num_speculative_tokens":3}' \ --enable-prefix-caching \ --reasoning-parser nemotron_v3 \ --enable-auto-tool-choice \ --tool-call-parser qwen3_coder ``` #### RTX Pro 6000 NVFP4 The NVFP4 variant runs on a single RTX Pro 6000 at TP=1. Requires vLLM v0.24.0 or later. A few notes: - **Set `--gpu-memory-utilization 0.92`** to avoid an OOM at startup. If you encounter an OOM, try adjusting the value. - **Set `VLLM_FLOAT32_MATMUL_PRECISION="high"`** to use high precision for the matmul operations. - **Set `VLLM_ALLOW_LONG_MAX_MODEL_LEN=1`** to allow the model to use the full context length. ```bash # Use container: vllm/vllm-openai:v0.28.0-ubuntu2404 VLLM_FLOAT32_MATMUL_PRECISION="high" \ VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 \ vllm serve nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4 \ --max-model-len 262144 \ --gpu-memory-utilization 0.92 \ --max-num-seqs 8 \ --max-num-batched-tokens 16384 \ --kv-cache-dtype fp8 \ --mamba-cache-mode align \ --mamba-ssm-cache-dtype float32 \ --speculative-config '{"method":"mtp","num_speculative_tokens":3}' \ --enable-prefix-caching \ --reasoning-parser nemotron_v3 \ --enable-auto-tool-choice \ --tool-call-parser qwen3_coder ``` ### Benchmarking ```bash vllm bench serve \ --model nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-FP8 \ --trust-remote-code \ --dataset-name random \ --random-input-len 1024 --random-output-len 1024 \ --num-warmups 20 \ --ignore-eos \ --max-concurrency 1024 \ --num-prompts 2048 ``` ### References - [vLLM blog: Nemotron-3-Super](https://vllm.ai/blog/nemotron-3-super) - [vLLM blog: vLLM on DGX Spark](https://vllm.ai/blog/2026-06-01-vllm-dgx-spark) - [Nemotron-3-Super-120B-A12B-BF16](https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16) - [Nemotron-3-Super-120B-A12B-FP8](https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-FP8) - [Nemotron-3-Super-120B-A12B-NVFP4](https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4) - [Nemotron-3-Super-120B-A12B-Base-BF16](https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-Base-BF16)