vLLM/Recipes
Qwen

Qwen/Qwen3.6-35B-A3B

Smaller Qwen3.6 multimodal MoE model (35B total / 3B active) with BF16, FP8, and NVIDIA NVFP4 variants

Compact Qwen3.6 MoE with 3B active parameters — single-GPU FP8 or 2-4 GPU BF16 serving

View on HuggingFace
moe35B / 3B262,144 ctxvLLM 0.17.0+multimodaltext
Guide

Overview

Qwen3.6-35B-A3B is the smaller sibling of Qwen3.5, sharing the same gated-delta-networks MoE architecture but with 35B total parameters and 3B activated (256 experts, 8 routed + 1 shared). The recipe covers the BF16 base model, Qwen's official FP8 checkpoint, and NVIDIA's ModelOpt NVFP4 checkpoint.

Prerequisites

  • vLLM version: >= 0.17.0
  • NVFP4 vLLM version: >= 0.28.0. 0.24.0 loads the checkpoint, but FlashInfer only selects its XQA decode kernel on SM120 GPUs from 0.28.0.
  • Hardware (BF16): 1x H200 or 2x H100
  • Hardware (FP8): single H100/H200 or 1x MI300X/MI325X/MI355X
  • Hardware (NVFP4): NVIDIA Blackwell GPUs, including DGX Spark (GB10)

NVFP4 on Blackwell

All NVFP4 tuning lives in variants.nvfp4.hardware_overrides, keyed by GPU profile, so the exact flag set for a box is readable straight from the recipe YAML (or from by_hardware in the JSON API) without going through the command builder.

Four profiles are tuned and verified. DGX Spark (GB10) and DGX Station (GB300) run the checkpoint on the default loader with FlashInfer attention, --moe-backend marlin and a 3-token MTP draft on Triton; Spark shares unified memory with the host, so --gpu-memory-utilization is capped at 0.5. The SM120 parts (RTX PRO 6000, RTX 5090) load through --quantization modelopt_fp4 with FlashInfer's TRT-LLM attention kernels, which need --block-size 128 and VLLM_HAS_FLASHINFER_CUBIN=1, and sustain a deeper 4-token MTP draft on flashinfer_cutlass. The RTX 5090's 32 GB caps context at 64K.

Datacenter Blackwell (B200/GB200/B300/GB300) intentionally carries no overrides — it runs vLLM's own NVFP4 defaults, which select the native FP4 path rather than the Marlin W4A16 kernel the workstation parts need.

Processing Ultra-Long Texts

Qwen3.6-35B-A3B natively supports 262,144 tokens. For longer inputs, apply YaRN RoPE scaling via --hf-overrides and raise --max-model-len. Pick factor to match your real workload — 2.0 covers ~524K, 4.0 covers ~1M — since YaRN at higher factors degrades short-context quality.

VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 vllm serve Qwen/Qwen3.6-35B-A3B-FP8 \
  --tensor-parallel-size 2 \
  --max-model-len 1010000 \
  --reasoning-parser qwen3 \
  --hf-overrides '{"text_config": {"rope_parameters": {"mrope_interleaved": true, "mrope_section": [11, 11, 10], "rope_type": "yarn", "rope_theta": 10000000, "partial_rotary_factor": 0.25, "factor": 4.0, "original_max_position_embeddings": 262144}}}'

See the model card for the full parameter reference.

Client Usage

from openai import OpenAI

client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1")
resp = client.chat.completions.create(
    model="Qwen/Qwen3.6-35B-A3B",
    messages=[{"role": "user", "content": "Explain gated delta networks in one paragraph."}],
    max_tokens=512,
)
print(resp.choices[0].message.content)

Troubleshooting

  • CUDA graph / Mamba cache size error: reduce --max-cudagraph-capture-size (default 512). See vLLM PR #34571.
  • Reasoning disable: add --default-chat-template-kwargs '{"enable_thinking": false}'.
  • Prefix Caching (Mamba): currently experimental in "align" mode.

References