{ "hf_id": "nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16", "meta": { "title": "NVIDIA Nemotron 3.5 Lightning", "slug": "nemotron-3-5-lightning", "provider": "NVIDIA", "description": "NVIDIA Nemotron 3.5 Lightning hybrid Mamba-MoE (30B total / 3B active) with NVFP4 and BF16 checkpoints, 1M context, and MTP / DSpark / DFlash speculative decoding", "date_added": "2026-08-03", "date_updated": "2026-10-08", "tasks": [ "text" ], "related_recipes": [ "nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16" ], "hardware": { "h100": "verified", "dgx_spark_gb10": "verified", "rtx_pro_6000": "verified", "dgx_station_gb300": "verified" } }, "recommended_command": { "hardware": "h200", "strategy": "single_node_tp", "variant": "default", "node_count": 1, "deploy_type": "single_node", "env": {}, "docker_image": "vllm/vllm-openai:v0.28.0", "command": "vllm serve nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 \\\n --mamba-backend flashinfer \\\n --mamba-cache-mode align \\\n --enable-prefix-caching \\\n --kv-cache-dtype fp8 \\\n --tensor-parallel-size 1 \\\n --quantization modelopt_fp4 \\\n --moe-backend humming \\\n --linear-backend humming \\\n --mamba-ssu-algorithm horizontal \\\n --async-scheduling \\\n --max-num-seqs 256 \\\n --max-num-batched-tokens 32768 \\\n --reasoning-parser nemotron_v3 \\\n --enable-auto-tool-choice \\\n --tool-call-parser qwen3_coder \\\n --mamba-ssm-cache-dtype float16 \\\n --enable-mamba-cache-stochastic-rounding \\\n --mamba-cache-philox-rounds 5", "argv": [ "vllm", "serve", "nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4", "--mamba-backend", "flashinfer", "--mamba-cache-mode", "align", "--enable-prefix-caching", "--kv-cache-dtype", "fp8", "--tensor-parallel-size", "1", "--quantization", "modelopt_fp4", "--moe-backend", "humming", "--linear-backend", "humming", "--mamba-ssu-algorithm", "horizontal", "--async-scheduling", "--max-num-seqs", "256", "--max-num-batched-tokens", "32768", "--reasoning-parser", "nemotron_v3", "--enable-auto-tool-choice", "--tool-call-parser", "qwen3_coder", "--mamba-ssm-cache-dtype", "float16", "--enable-mamba-cache-stochastic-rounding", "--mamba-cache-philox-rounds", "5" ], "docker_command": "docker run --gpus all \\\n --privileged --ipc=host -p 8000:8000 \\\n -v ~/.cache/huggingface:/root/.cache/huggingface \\\n vllm/vllm-openai:v0.28.0 nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 \\\n --mamba-backend flashinfer \\\n --mamba-cache-mode align \\\n --enable-prefix-caching \\\n --kv-cache-dtype fp8 \\\n --tensor-parallel-size 1 \\\n --quantization modelopt_fp4 \\\n --moe-backend humming \\\n --linear-backend humming \\\n --mamba-ssu-algorithm horizontal \\\n --async-scheduling \\\n --max-num-seqs 256 \\\n --max-num-batched-tokens 32768 \\\n --reasoning-parser nemotron_v3 \\\n --enable-auto-tool-choice \\\n --tool-call-parser qwen3_coder \\\n --mamba-ssm-cache-dtype float16 \\\n --enable-mamba-cache-stochastic-rounding \\\n --mamba-cache-philox-rounds 5", "docker_argv": [ "docker", "run", "--gpus", "all", "--privileged", "--ipc=host", "-p", "8000:8000", "-v", "~/.cache/huggingface:/root/.cache/huggingface", "vllm/vllm-openai:v0.28.0", "nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4", "--mamba-backend", "flashinfer", "--mamba-cache-mode", "align", "--enable-prefix-caching", "--kv-cache-dtype", "fp8", "--tensor-parallel-size", "1", "--quantization", "modelopt_fp4", "--moe-backend", "humming", "--linear-backend", "humming", "--mamba-ssu-algorithm", "horizontal", "--async-scheduling", "--max-num-seqs", "256", "--max-num-batched-tokens", "32768", "--reasoning-parser", "nemotron_v3", "--enable-auto-tool-choice", "--tool-call-parser", "qwen3_coder", "--mamba-ssm-cache-dtype", "float16", "--enable-mamba-cache-stochastic-rounding", "--mamba-cache-philox-rounds", "5" ], "strategy_spec": { "name": "single_node_tp", "deploy_type": "single_node", "display_name": "Tensor Parallel", "short_label": "TP", "orientation": "latency", "description": "Single-node tensor parallel. Splits the model across all local GPUs. TP size is set to the GPU count at deploy time. The simplest multi-GPU strategy — works for all model architectures.\n", "hardware_match": { "min_gpus": 1, "max_gpus": 8, "multi_node": false }, "vllm_args": [], "parallel_flag": "--tensor-parallel-size" }, "hardware_profile": { "brand": "NVIDIA", "generation": "hopper", "compute_arch": "sm90", "display_name": "H200", "description": "NVIDIA H200 SXM 141 GB HBM3e · 8-GPU HGX node", "gpu_count": 8, "vram_gb": 1128, "multi_node": false }, "alternatives": { "single_node_tep": "/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16/strategies/single_node_tep.json", "kv_store_centralized_mooncake": "/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16/strategies/kv_store_centralized_mooncake.json", "kv_store_distributed_mooncake": "/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16/strategies/kv_store_distributed_mooncake.json" }, "by_hardware": { "h200": "/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16/hw/h200.json", "h100": "/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16/hw/h100.json", "b200": "/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16/hw/b200.json", "gb200": "/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16/hw/gb200.json", "b300": "/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16/hw/b300.json", "gb300": "/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16/hw/gb300.json", "dgx_station_gb300": "/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16/hw/dgx_station_gb300.json", "dgx_spark_gb10": "/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16/hw/dgx_spark_gb10.json", "rtx_pro_6000": "/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16/hw/rtx_pro_6000.json" } }, "model": { "model_id": "nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16", "min_vllm_version": "0.27.1", "docker_image": "vllm/vllm-openai:v0.28.0", "architecture": "moe", "parameter_count": "30B", "active_parameters": "3B", "context_length": 262144, "base_args": [ "--mamba-backend", "flashinfer", "--mamba-cache-mode", "align", "--enable-prefix-caching", "--max-num-batched-tokens", "16384" ], "base_env": {}, "install": { "pip": { "command": "uv venv\nsource .venv/bin/activate\nuv pip install -U vllm --torch-backend auto" }, "docker": { "command": "docker pull vllm/vllm-openai:v0.28.0" }, "extras": [ { "command": "uv pip install \"fastokens>=0.2.0\"", "note": "Required by the BF16 variant, which sets VLLM_USE_FASTOKENS=1; vLLM does not bundle it yet (vllm-project/vllm#47859)" } ] } }, "features": { "reasoning": { "description": "Nemotron v3 reasoning parser — per-request reasoning toggle with a configurable token budget", "args": [ "--reasoning-parser", "nemotron_v3" ] }, "tool_calling": { "description": "Qwen3 Coder tool-call parser with automatic tool choice", "args": [ "--enable-auto-tool-choice", "--tool-call-parser", "qwen3_coder" ] }, "mamba_fast_ssm_cache": { "label": "Fast SSM Cache", "description": "FP16 Mamba SSM cache with stochastic rounding — faster decode than the FP32 default; the Philox rounds control the rounding stream", "args": [ "--mamba-ssm-cache-dtype", "float16", "--enable-mamba-cache-stochastic-rounding", "--mamba-cache-philox-rounds", "5" ] }, "spec_decoding": { "description": "Speculative decoding — built-in MTP heads, the DSpark hybrid speculator, or the DFlash diffusion drafter", "default_mode": "dspark", "modes": { "dspark": { "label": "DSpark", "description": "Hybrid autoregressive + diffusion speculator with a W4A16 draft head. Lowest latency on H100 and DGX Spark; needs the separate DSpark draft checkpoint.", "variants": [ "default" ], "args": [ "--speculative-config", "{\"method\":\"dspark\",\"model\":\"nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4-DSpark\",\"num_speculative_tokens\":3}" ], "hardware_overrides": { "dgx_spark_gb10": { "args": [ "--speculative-config", "{\"method\":\"dspark\",\"model\":\"nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4-DSpark\",\"num_speculative_tokens\":7}" ] } } }, "mtp": { "label": "MTP", "description": "Built-in Multi-Token Prediction heads — no extra checkpoint to download.", "args": [ "--speculative-config", "{\"method\":\"mtp\",\"num_speculative_tokens\":3}" ], "hardware_overrides": { "blackwell": { "args": [ "--speculative-config", "{\"method\":\"mtp\",\"num_speculative_tokens\":3,\"moe_backend\":\"triton\"}" ] } } }, "dflash": { "label": "DFlash", "description": "Diffusion drafter that proposes a whole block of tokens in parallel; needs the separate DFlash draft checkpoint.", "variants": [ "default" ], "args": [ "--speculative-config", "{\"method\":\"dflash\",\"model\":\"nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4-DFlash\",\"num_speculative_tokens\":3,\"attention_backend\":\"FLASHINFER\"}" ] } } }, "long_context": { "description": "Raise the context window from the default 256K to the model's full 1M tokens. Needs a large KV-cache budget — lower --max-model-len if you are memory-constrained.", "args": [ "--max-model-len", "1048576" ], "env": { "VLLM_ALLOW_LONG_MAX_MODEL_LEN": "1" } } }, "opt_in_features": [ "spec_decoding", "long_context" ], "variants": { "default": { "model_id": "nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4", "precision": "nvfp4", "precision_hardware": { "brand": "NVIDIA" }, "vram_minimum_gb": 18, "label": "NVFP4", "description": "ModelOpt NVFP4 (W4A16) checkpoint with FP8 KV cache — the default deployment target, and the only one the DSpark/DFlash drafts are published for", "extra_args": [ "--kv-cache-dtype", "fp8", "--moe-backend", "marlin" ], "hardware_overrides": { "hopper": { "extra_args": [ "--quantization", "modelopt_fp4", "--moe-backend", "humming", "--linear-backend", "humming", "--mamba-ssu-algorithm", "horizontal", "--async-scheduling", "--max-num-seqs", "256", "--max-num-batched-tokens", "32768" ] }, "dgx_spark_gb10": { "docker_image": "vllm/vllm-openai:nightly", "extra_args": [ "--gpu-memory-utilization", "0.7", "--load-format", "fastsafetensors" ] }, "rtx_pro_6000": { "extra_args": [ "--attention-backend", "FLASHINFER", "--linear-backend", "marlin", "--attention-config", "{\"use_trtllm_attention\":true}", "--enable-flashinfer-autotune", "--mamba-cache-mode", "none", "--async-scheduling", "--enable-chunked-prefill", "--gpu-memory-utilization", "0.8", "--load-format", "instanttensor", "--safetensors-load-strategy", "prefetch" ] }, "dgx_station_gb300": { "extra_args": [ "--attention-backend", "flashinfer", "--enable-chunked-prefill", "--async-scheduling" ] } }, "json": "/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4.json" }, "bf16": { "model_id": "nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16", "precision": "bf16", "vram_minimum_gb": 72, "description": "Full-precision BF16 reference checkpoint — MTP only (no DSpark/DFlash draft is published for BF16)", "extra_args": [ "--moe-backend", "flashinfer_cutlass" ], "extra_env": { "VLLM_USE_FASTOKENS": "1" } } }, "hardware_overrides": {}, "guide": "## Overview\n\nNemotron 3.5 Lightning is a hybrid Mamba-MoE model — 30B total parameters, 3B active,\nup to 1M context, distilled from Nemotron 3 Ultra and trained for popular agent\nharnesses. It ships as an NVFP4 (W4A16) ModelOpt checkpoint, the default deployment\ntarget, and a BF16 reference checkpoint.\n\n## Prerequisites\n\n- **vLLM:** 0.27.1 or newer — `vllm/vllm-openai:v0.28.0`\n- **Hardware:** 1x H100, 1x DGX Spark (GB10), or 1x DGX Station (GB300)\n- **Speculative decoding:** **MTP** (built into the checkpoint), **DSpark** (hybrid\n autoregressive + diffusion) and **DFlash** (diffusion) are all supported. DSpark\n gives the best latency on H100 and DGX Spark; for maximum throughput, serve without\n speculative decoding.\n\n## Client Usage\n\nOnce the vLLM server is running, consume it via the OpenAI-compatible API. Reasoning\nis returned separately from the answer, on `message.reasoning`:\n\n```python\nfrom openai import OpenAI\n\n# Set this to match the --served-model-name used when starting the server\nSERVED_MODEL_NAME = \"nemotron-3.5-lightning\"\nBASE_URL = \"http://127.0.0.1:8000/v1\"\n\nclient = OpenAI(base_url=BASE_URL, api_key=\"null\")\n\nresponse = client.chat.completions.create(\n model=SERVED_MODEL_NAME,\n messages=[\n {\"role\": \"system\", \"content\": \"You are a helpful assistant.\"},\n {\"role\": \"user\", \"content\": \"Briefly explain: what is vLLM and why is it useful for large model inference?\"},\n ],\n temperature=1.0,\n top_p=0.95,\n max_tokens=2048,\n)\n\nchoice = response.choices[0]\nprint(\"Reasoning:\", choice.message.reasoning)\nprint(\"Content:\", choice.message.content)\n```\n\nReasoning tokens count toward `max_tokens`. If `content` comes back empty or `None`,\nthe reasoning trace consumed the entire budget before the model produced an answer —\nraise `max_tokens`.\n\n## Notes\n\n- **Speculative decoding**: DSpark is preferred on DGX Spark. On DGX Station, prefer\n MTP — choose it in the Speculative Decoding panel above.\n- **DGX Spark (GB10)**: the selectable config is the one measured on 2026-08-31 with\n aiperf 0.12.0 on vLLM `0.28.1rc1.dev130+g44fe2a392` — DSpark drafting 7 tokens at\n `--gpu-memory-utilization 0.7`, `--mamba-cache-mode align` and fastsafetensors\n loading, served from `vllm/vllm-openai:nightly`. On SPEED-Bench at 8K input / 256\n output, concurrency 1, it reached 7,252 prefill tok/s and 93.3 decode tok/s at a\n 3.98 s mean end-to-end latency.\n", "source": { "source": "vllm", "repo": "vllm-project/vllm", "ref": "main", "sha": "564ac0de1b228568cafcbfbe8ccaccc82d6023eb", "path": "recipes/models/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16.yaml", "edit_url": "https://github.com/vllm-project/vllm/edit/main/recipes/models/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16.yaml", "overlays": [] } }