nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16

NVIDIA Nemotron 3.5 Lightning hybrid Mamba-MoE (30B total / 3B active) with NVFP4 and BF16 checkpoints, 1M context, and MTP / DSpark / DFlash speculative decoding

View on HuggingFace

moe30B / 3B262,144 ctxvLLM 0.27.1+text

Default configuration: 8× H200 · Tensor Parallel · NVFP4

vllm serve nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 \
  --mamba-backend flashinfer \
  --mamba-cache-mode align \
  --enable-prefix-caching \
  --kv-cache-dtype fp8 \
  --tensor-parallel-size 1 \
  --quantization modelopt_fp4 \
  --moe-backend humming \
  --linear-backend humming \
  --mamba-ssu-algorithm horizontal \
  --async-scheduling \
  --max-num-seqs 256 \
  --max-num-batched-tokens 32768 \
  --reasoning-parser nemotron_v3 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --mamba-ssm-cache-dtype float16 \
  --enable-mamba-cache-stochastic-rounding \
  --mamba-cache-philox-rounds 5

The interactive builder loads with JavaScript. Append .md to this page's URL, query string included, for any configuration's command as Markdown — e.g. /nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16.md.

Guide

Overview

Nemotron 3.5 Lightning is a hybrid Mamba-MoE model — 30B total parameters, 3B active, up to 1M context, distilled from Nemotron 3 Ultra and trained for popular agent harnesses. It ships as an NVFP4 (W4A16) ModelOpt checkpoint, the default deployment target, and a BF16 reference checkpoint.

Prerequisites

vLLM: 0.27.1 or newer — vllm/vllm-openai:v0.28.0
Hardware: 1x H100, 1x DGX Spark (GB10), or 1x DGX Station (GB300)
Speculative decoding: MTP (built into the checkpoint), DSpark (hybrid autoregressive + diffusion) and DFlash (diffusion) are all supported. DSpark gives the best latency on H100 and DGX Spark; for maximum throughput, serve without speculative decoding.

Client Usage

Once the vLLM server is running, consume it via the OpenAI-compatible API. Reasoning is returned separately from the answer, on message.reasoning:

from openai import OpenAI

# Set this to match the --served-model-name used when starting the server
SERVED_MODEL_NAME = "nemotron-3.5-lightning"
BASE_URL = "http://127.0.0.1:8000/v1"

client = OpenAI(base_url=BASE_URL, api_key="null")

response = client.chat.completions.create(
    model=SERVED_MODEL_NAME,
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Briefly explain: what is vLLM and why is it useful for large model inference?"},
    ],
    temperature=1.0,
    top_p=0.95,
    max_tokens=2048,
)

choice = response.choices[0]
print("Reasoning:", choice.message.reasoning)
print("Content:", choice.message.content)

Reasoning tokens count toward max_tokens. If content comes back empty or None, the reasoning trace consumed the entire budget before the model produced an answer — raise max_tokens.

Notes

Speculative decoding: DSpark is preferred on DGX Spark. On DGX Station, prefer MTP — choose it in the Speculative Decoding panel above.
DGX Spark (GB10): the selectable config is the one measured on 2026-08-31 with aiperf 0.12.0 on vLLM 0.28.1rc1.dev130+g44fe2a392 — DSpark drafting 7 tokens at --gpu-memory-utilization 0.7, --mamba-cache-mode align and fastsafetensors loading, served from vllm/vllm-openai:nightly. On SPEED-Bench at 8K input / 256 output, concurrency 1, it reached 7,252 prefill tok/s and 93.3 decode tok/s at a 3.98 s mean end-to-end latency.