nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16

Overview

Nemotron 3.5 Lightning is a hybrid Mamba-MoE model — 30B total parameters, 3B active, up to 1M context, distilled from Nemotron 3 Ultra and trained for popular agent harnesses. It ships as an NVFP4 (W4A16) ModelOpt checkpoint, the default deployment target, and a BF16 reference checkpoint.

Prerequisites

• vLLM: 0.27.1 or newer — vllm/vllm-openai:v0.27.1
• Hardware: 1x H100, 1x DGX Spark (GB10), or 1x DGX Station (GB300)
• Speculative decoding: MTP (built into the checkpoint), DSpark (hybrid autoregressive + diffusion) and DFlash (diffusion) are all supported. DSpark gives the best latency on H100 and DGX Spark; for maximum throughput, serve without speculative decoding.

Client Usage

Once the vLLM server is running, consume it via the OpenAI-compatible API. Reasoning is returned separately from the answer, on message.reasoning:

from openai import OpenAI

# Set this to match the --served-model-name used when starting the server
SERVED_MODEL_NAME = "nemotron-3.5-lightning"
BASE_URL = "http://127.0.0.1:8000/v1"

client = OpenAI(base_url=BASE_URL, api_key="null")

response = client.chat.completions.create(
    model=SERVED_MODEL_NAME,
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Briefly explain: what is vLLM and why is it useful for large model inference?"},
    ],
    temperature=1.0,
    top_p=0.95,
    max_tokens=2048,
)

choice = response.choices[0]
print("Reasoning:", choice.message.reasoning)
print("Content:", choice.message.content)

Reasoning tokens count toward max_tokens. If content comes back empty or None, the reasoning trace consumed the entire budget before the model produced an answer — raise max_tokens.

Notes

• Speculative decoding: DSpark is preferred on DGX Spark. On DGX Station, prefer MTP — choose it in the Speculative Decoding panel above.