vLLM/Recipes
Qwen

Qwen/Qwen3.5-397B-A17B

Multimodal MoE model with gated delta networks architecture, 397B total / 17B active parameters, up to 262K context

Verified on 8x H200, 8x MI300X/MI355X, a single MI350X/MI355X (MXFP4), GB200 nodes, and Ascend 950DT

View on HuggingFace
moe397B / 17B262,144 ctxvLLM 0.17.0+multimodaltext
Guide

Overview

Qwen3.5 is a multimodal mixture-of-experts model featuring a gated delta networks architecture with 397B total parameters and 17B active parameters. This guide covers how to efficiently deploy and serve the model across different hardware configurations and workload profiles using vLLM.

Prerequisites

Pip Install

NVIDIA

uv venv
source .venv/bin/activate
uv pip install -U vllm --torch-backend=auto

AMD

Note: The vLLM wheel for ROCm requires Python 3.12, ROCm 7.0, and glibc >= 2.35. If your environment does not meet these requirements, please use the Docker-based setup. Supported GPUs: MI300X, MI325X, MI355X.

uv venv --python 3.12
source .venv/bin/activate
uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm

Docker

NVIDIA

docker run --gpus all \
  -p 8000:8000 \
  --ipc=host \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai Qwen/Qwen3.5-397B-A17B \
    --tensor-parallel-size 8 \
    --reasoning-parser qwen3 \
    --enable-prefix-caching

For Blackwell GPUs, use vllm/vllm-openai:cu130-nightly.

AMD

docker run --device=/dev/kfd --device=/dev/dri \
  --security-opt seccomp=unconfined \
  --group-add video \
  --ipc=host \
  -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai-rocm:latest \
  Qwen/Qwen3.5-397B-A17B-FP8 \
  --tensor-parallel-size 8 \
  --reasoning-parser qwen3 \
  --enable-prefix-caching

Deployment Configurations

The configurations below have been verified on 8x H200 GPUs and 8x MI300X/MI355X GPUs. We recommend using the official FP8 checkpoint Qwen/Qwen3.5-397B-A17B-FP8 for optimal serving efficiency.

Throughput-Focused (Text-Only)

For maximum text throughput under high concurrency, use --language-model-only to skip loading the vision encoder and free up memory for KV cache, and enable Expert Parallelism.

vllm serve Qwen/Qwen3.5-397B-A17B-FP8 \
  -dp 8 \
  --enable-expert-parallel \
  --language-model-only \
  --reasoning-parser qwen3 \
  --enable-prefix-caching

Throughput-Focused (Multimodal)

For multimodal workloads, use --mm-encoder-tp-mode data for data-parallel vision encoding and --mm-processor-cache-type shm for shared-memory caching of preprocessed multimodal inputs.

vllm serve Qwen/Qwen3.5-397B-A17B-FP8 \
  -dp 8 \
  --enable-expert-parallel \
  --mm-encoder-tp-mode data \
  --mm-processor-cache-type shm \
  --reasoning-parser qwen3 \
  --enable-prefix-caching

To enable tool calling, add --enable-auto-tool-choice --tool-call-parser qwen3_coder to the serve command.

Latency-Focused

For latency-sensitive workloads at low concurrency, enable MTP-1 speculative decoding and disable prefix caching. MTP-1 reduces time-per-output-token (TPOT) with a high acceptance rate, at the cost of lower throughput under load.

Note (AMD): MTP runs on ROCm with vLLM 0.31.0 (verified with the single-GPU MXFP4 setup below). On ROCm, pair it with --kv-cache-dtype fp8: with the default KV cache its speedup fades as the prompt grows.

vllm serve Qwen/Qwen3.5-397B-A17B-FP8 \
  --tensor-parallel-size 8 \
  --speculative-config '{"method": "mtp", "num_speculative_tokens": 1}' \
  --reasoning-parser qwen3

GB200 Deployment

We recommend using the NVFP4 checkpoint nvidia/Qwen3.5-397B-A17B-NVFP4 for optimal serving efficiency on GB200 nodes.

vllm serve nvidia/Qwen3.5-397B-A17B-NVFP4 \
  -dp 4 \
  --enable-expert-parallel \
  --language-model-only \
  --reasoning-parser qwen3 \
  --enable-prefix-caching

MI355X Deployment

vllm serve Qwen/Qwen3.5-397B-A17B-FP8 \
  -tp 2 \
  --enable-expert-parallel \
  --language-model-only \
  --reasoning-parser qwen3 \
  --enable-prefix-caching

Single MI350X / MI355X (MXFP4)

AMD's MXFP4 checkpoint amd/Qwen3.5-397B-A17B-MXFP4 (239 GB) fits one 288 GB MI350X or MI355X: pick the MXFP4 (W4A4) variant above. It sets three things vLLM does not pick on its own: VLLM_ROCM_USE_AITER=1 turns on the AITER kernels; --attention-backend ROCM_AITER_FA replaces the auto-selected ROCM_ATTN, whose decode slows down as the context grows on this hybrid model; and --max-num-seqs 64, because the default (1024) is more than the ~296 Mamba cache blocks a single GPU leaves, and the server would refuse to start.

VLLM_ROCM_USE_AITER=1 vllm serve amd/Qwen3.5-397B-A17B-MXFP4 \
  --tensor-parallel-size 1 \
  --attention-backend ROCM_AITER_FA \
  --max-num-seqs 64 \
  --trust-remote-code \
  --reasoning-parser qwen3

Single-stream decode in tokens/s at ~200 / ~2k / ~12k / ~30k prompt tokens (128 output tokens), one GPU, vllm/vllm-openai-rocm:v0.31.0:

~200~2k~12k~30k
MXFP4, this variant (MI355X)119119119117
+ MTP (3 tokens) + --kv-cache-dtype fp8 (MI355X)219201209188
+ MTP (3 tokens), default KV cache (MI350X)19816412580
auto-selected ROCM_ATTN instead of ROCM_AITER_FA (MI355X)116893919
GPTQ-Int4, AITER + ROCM_AITER_FA (MI350X)75747474

With MTP, add --kv-cache-dtype fp8: on ROCm, MTP's gain otherwise fades as the prompt grows. The MXFP4 checkpoint quantizes activations as well as weights (W4A4): on the same wikitext-2 tokens vLLM scored PPL 3.43 with it and 3.39 with GPTQ-Int4 (W4A16), so GPTQ-Int4 is the higher-fidelity option at about two thirds of the speed.

Processing Ultra-Long Texts

Qwen3.5-397B-A17B natively supports 262,144 tokens. To serve longer inputs, turn on the Long Context feature above: it applies the model card's YaRN RoPE configuration via --hf-overrides and raises --max-model-len to ~1,010,000 tokens. Static YaRN holds the scaling factor constant regardless of input length, so leave it off unless you actually serve long prompts; for a ~524K workload, halve the card's factor to 2.0.

See the model card for the full parameter reference.

Client Usage

import time
from openai import OpenAI

client = OpenAI(
    api_key="EMPTY",
    base_url="http://localhost:8000/v1",
    timeout=3600
)

messages = [
    {
        "role": "user",
        "content": [
            {
                "type": "image_url",
                "image_url": {
                    "url": "https://ofasys-multimodal-wlcb-3-toshanghai.oss-accelerate.aliyuncs.com/wpf272043/keepme/image/receipt.png"
                }
            },
            {
                "type": "text",
                "text": "Read all the text in the image."
            }
        ]
    }
]

start = time.time()
response = client.chat.completions.create(
    model="Qwen/Qwen3.5-397B-A17B",
    messages=messages,
    max_tokens=2048
)
print(f"Response costs: {time.time() - start:.2f}s")
print(f"Generated text: {response.choices[0].message.content}")

Troubleshooting

CUDA graph / Mamba cache size error

You may encounter:

assert num_cache_lines >= batch

This occurs because the CUDA graph capture size is larger than the Mamba cache size. Reduce --max-cudagraph-capture-size (default is 512). See https://github.com/vllm-project/vllm/pull/34571 for details.

Configuration tips

  • Disable Reasoning: Add --reasoning-parser qwen3 --default-chat-template-kwargs '{"enable_thinking": false}' to disable reasoning mode via command-line parameters.
  • Prefix Caching: Prefix caching for Mamba cache "align" mode is currently experimental.
  • Multi-token Prediction: MTP-1 reduces per-token latency but degrades throughput under high concurrency because speculative tokens consume KV cache capacity. Adjust num_speculative_tokens (1-5) based on your use case.
  • Encoder Data Parallelism: --mm-encoder-tp-mode data deploys the vision encoder in a data-parallel fashion. This consumes additional memory and may require adjustment of --gpu-memory-utilization.

References