# nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16 — vLLM serving recipe

> NVIDIA Nemotron 3.5 Lightning hybrid Mamba-MoE (30B total / 3B active) with NVFP4 and BF16 checkpoints, 1M context, and MTP / DSpark / DFlash speculative decoding

Configuration: hardware `h200` (8× H200) · strategy `single_node_tp` (Tensor Parallel) · variant `default` (NVFP4) · features `reasoning`, `tool_calling`, `mamba_fast_ssm_cache`

- Interactive page: https://recipes.vllm.ai/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16
- This configuration as JSON: https://recipes.vllm.ai/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16.json?hardware=h200

The commands below were rendered for exactly this configuration and take precedence over any example in the guide.

## Other configurations

Change the query string (the same parameters as the page URL) and fetch this URL again. Unknown or incompatible values fall back to the recipe default, so check the Configuration line.

- `hardware`: `h100`, `h200`, `b200`, `gb200`, `b300`, `gb300`, `dgx_station_gb300`, `dgx_spark_gb10`, `rtx_pro_6000`
- `variant`: `default`, `bf16`
- `strategy`: `single_node_tp`, `single_node_tep` (multi-node strategies also need `nodes=<n>`)
- `features`: comma-separated list of `reasoning`, `tool_calling`, `mamba_fast_ssm_cache`, `spec_decoding`, `long_context` (default: `reasoning`, `tool_calling`, `mamba_fast_ssm_cache`)

## Install

Requires vLLM >= 0.27.1.

```bash
uv venv
source .venv/bin/activate
uv pip install -U vllm --torch-backend auto
```

Docker image: `vllm/vllm-openai:v0.28.0`

Extra dependencies:

- Required by the BF16 variant, which sets VLLM_USE_FASTOKENS=1; vLLM does not bundle it yet (vllm-project/vllm#47859)

```bash
uv pip install "fastokens>=0.2.0"
```

## Serve

```bash
vllm serve nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 \
  --mamba-backend flashinfer \
  --mamba-cache-mode align \
  --enable-prefix-caching \
  --kv-cache-dtype fp8 \
  --tensor-parallel-size 1 \
  --quantization modelopt_fp4 \
  --moe-backend humming \
  --linear-backend humming \
  --mamba-ssu-algorithm horizontal \
  --async-scheduling \
  --max-num-seqs 256 \
  --max-num-batched-tokens 32768 \
  --reasoning-parser nemotron_v3 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --mamba-ssm-cache-dtype float16 \
  --enable-mamba-cache-stochastic-rounding \
  --mamba-cache-philox-rounds 5
```

Docker equivalent:

```bash
docker run --gpus all \
  --privileged --ipc=host -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai:v0.28.0 nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 \
  --mamba-backend flashinfer \
  --mamba-cache-mode align \
  --enable-prefix-caching \
  --kv-cache-dtype fp8 \
  --tensor-parallel-size 1 \
  --quantization modelopt_fp4 \
  --moe-backend humming \
  --linear-backend humming \
  --mamba-ssu-algorithm horizontal \
  --async-scheduling \
  --max-num-seqs 256 \
  --max-num-batched-tokens 32768 \
  --reasoning-parser nemotron_v3 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --mamba-ssm-cache-dtype float16 \
  --enable-mamba-cache-stochastic-rounding \
  --mamba-cache-philox-rounds 5
```

## Guide

> Background reference. Commands in it are examples and may target another configuration — use the Serve section above.

### Overview

Nemotron 3.5 Lightning is a hybrid Mamba-MoE model — 30B total parameters, 3B active,
up to 1M context, distilled from Nemotron 3 Ultra and trained for popular agent
harnesses. It ships as an NVFP4 (W4A16) ModelOpt checkpoint, the default deployment
target, and a BF16 reference checkpoint.

### Prerequisites

- **vLLM:** 0.27.1 or newer — `vllm/vllm-openai:v0.28.0`
- **Hardware:** 1x H100, 1x DGX Spark (GB10), or 1x DGX Station (GB300)
- **Speculative decoding:** **MTP** (built into the checkpoint), **DSpark** (hybrid
  autoregressive + diffusion) and **DFlash** (diffusion) are all supported. DSpark
  gives the best latency on H100 and DGX Spark; for maximum throughput, serve without
  speculative decoding.

### Client Usage

Once the vLLM server is running, consume it via the OpenAI-compatible API. Reasoning
is returned separately from the answer, on `message.reasoning`:

```python
from openai import OpenAI

# Set this to match the --served-model-name used when starting the server
SERVED_MODEL_NAME = "nemotron-3.5-lightning"
BASE_URL = "http://127.0.0.1:8000/v1"

client = OpenAI(base_url=BASE_URL, api_key="null")

response = client.chat.completions.create(
    model=SERVED_MODEL_NAME,
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Briefly explain: what is vLLM and why is it useful for large model inference?"},
    ],
    temperature=1.0,
    top_p=0.95,
    max_tokens=2048,
)

choice = response.choices[0]
print("Reasoning:", choice.message.reasoning)
print("Content:", choice.message.content)
```

Reasoning tokens count toward `max_tokens`. If `content` comes back empty or `None`,
the reasoning trace consumed the entire budget before the model produced an answer —
raise `max_tokens`.

### Notes

- **Speculative decoding**: DSpark is preferred on DGX Spark. On DGX Station, prefer
  MTP — choose it in the Speculative Decoding panel above.
- **DGX Spark (GB10)**: the selectable config is the one measured on 2026-08-31 with
  aiperf 0.12.0 on vLLM `0.28.1rc1.dev130+g44fe2a392` — DSpark drafting 7 tokens at
  `--gpu-memory-utilization 0.7`, `--mamba-cache-mode align` and fastsafetensors
  loading, served from `vllm/vllm-openai:nightly`. On SPEED-Bench at 8K input / 256
  output, concurrency 1, it reached 7,252 prefill tok/s and 93.3 decode tok/s at a
  3.98 s mean end-to-end latency.