# Qwen/Qwen3.5-122B-A10B — vLLM serving recipe

> Mid-size Qwen3.5 multimodal MoE (122B total / 10B active) with gated delta networks, 256 experts, and 262K context

Configuration: hardware `h200` (8× H200) · strategy `single_node_tp` (Tensor Parallel) · variant `default` (BF16) · features `tool_calling`, `reasoning`, `encoder_parallel` · frontend `rust`

- Interactive page: https://recipes.vllm.ai/Qwen/Qwen3.5-122B-A10B
- This configuration as JSON: https://recipes.vllm.ai/Qwen/Qwen3.5-122B-A10B.json?hardware=h200

The commands below were rendered for exactly this configuration and take precedence over any example in the guide.

## Other configurations

Change the query string (the same parameters as the page URL) and fetch this URL again. Unknown or incompatible values fall back to the recipe default, so check the Configuration line.

- `hardware`: `h100`, `h200`, `b200`, `gb200`, `b300`, `gb300`, `mi300x`, `mi325x`, `mi350x`, `mi355x`
- `variant`: `default`, `fp8`, `nvfp4`, `gptq_int4`, `redhat_fp8`
- `strategy`: `single_node_tp`, `single_node_tep`, `single_node_dep`, `multi_node_tp`, `multi_node_dep`, `multi_node_tep`, `pd_cluster` (multi-node strategies also need `nodes=<n>`)
- `features`: comma-separated list of `tool_calling`, `reasoning`, `spec_decoding`, `text_only`, `encoder_parallel`, `long_context` (default: `tool_calling`, `reasoning`, `encoder_parallel`)

## Install

Requires vLLM >= 0.17.0.

```bash
uv venv
source .venv/bin/activate
uv pip install -U vllm --torch-backend auto
```

Docker image: `vllm/vllm-openai:latest`

## Serve

```bash
export VLLM_USE_RUST_FRONTEND=1
vllm serve Qwen/Qwen3.5-122B-A10B \
  --trust-remote-code \
  --tensor-parallel-size 8 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --reasoning-parser qwen3 \
  --mm-encoder-tp-mode data
```

Docker equivalent:

```bash
docker run --gpus all \
  --privileged --ipc=host -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -e VLLM_USE_RUST_FRONTEND=1 \
  vllm/vllm-openai:latest Qwen/Qwen3.5-122B-A10B \
  --trust-remote-code \
  --tensor-parallel-size 8 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --reasoning-parser qwen3 \
  --mm-encoder-tp-mode data
```

## Guide

> Background reference. Commands in it are examples and may target another configuration — use the Serve section above.

### Overview

[Qwen3.5-122B-A10B](https://huggingface.co/Qwen/Qwen3.5-122B-A10B) is a mid-tier
member of the Qwen3.5 family, sharing the gated delta networks MoE architecture
with 122B total parameters and 10B activated per token (256 experts). It is
multimodal (vision + text) and natively supports 262K context.

### Prerequisites

- **vLLM version:** >= 0.17.0
- **Hardware (BF16):** 4x H200 or 8x H100
- **Hardware (FP8):** 2x H200 or 4x H100
- **Hardware (Int4):** single 80 GB GPU

#### Install vLLM

```bash
uv venv
source .venv/bin/activate
uv pip install -U vllm --torch-backend=auto
```

### Launching the Server

#### BF16 on 4xH200 (TP4)

```bash
vllm serve Qwen/Qwen3.5-122B-A10B \
  --tensor-parallel-size 4 \
  --max-model-len 262144 \
  --reasoning-parser qwen3
```

#### FP8 on 2xH200 (TP2)

```bash
vllm serve Qwen/Qwen3.5-122B-A10B-FP8 \
  --tensor-parallel-size 2 \
  --max-model-len 262144 \
  --reasoning-parser qwen3
```

#### Throughput-focused (text-only, EP)

```bash
vllm serve Qwen/Qwen3.5-122B-A10B-FP8 \
  -dp 4 --enable-expert-parallel \
  --language-model-only \
  --reasoning-parser qwen3 \
  --enable-prefix-caching
```

#### MTP speculative decoding

```bash
vllm serve Qwen/Qwen3.5-122B-A10B-FP8 \
  --tensor-parallel-size 2 \
  --speculative-config '{"method": "mtp", "num_speculative_tokens": 1}' \
  --reasoning-parser qwen3
```

#### AMD (MI325X / MI355X, single GPU, FP8)

The FP8 checkpoint fits one MI300X-class GPU. Three settings matter on ROCm and none is on by
default: `VLLM_ROCM_USE_AITER=1` switches the MoE to the AITER FP8 kernels,
`--attention-backend ROCM_AITER_FA` keeps decode flat across context (the auto-selected
`ROCM_ATTN` decays with prompt length on this hybrid GDN model), and `--kv-cache-dtype fp8`
keeps MTP flat across context too.

```bash
VLLM_ROCM_USE_AITER=1 vllm serve Qwen/Qwen3.5-122B-A10B-FP8 \
  --attention-backend ROCM_AITER_FA \
  --kv-cache-dtype fp8 --max-num-seqs 64 \
  --reasoning-parser qwen3
```

`--max-num-seqs 64` because the default exceeds the Mamba block budget on 192 GB. MTP
(`--speculative-config '{"method":"mtp","num_speculative_tokens":1}'`) works with the same
flags. Single-stream decode (tokens/s at ~200 / ~2k / ~12k / ~30k prompt tokens, 128 output
tokens, prefix caching off), measured 2026-10-07 on `vllm/vllm-openai-rocm:v0.31.0`:

| | MI325X | MI355X |
|---|---|---|
| AITER + `ROCM_AITER_FA` | 128 / 127 / 127 / 127 | 143 / 143 / 143 / 143 |
| + MTP | 184 / 169 / 172 / 164 | 204 / 193 / 195 / 190 |

This recipe used `ROCM_AITER_UNIFIED_ATTN` before v0.31.0. Avoid it on MI350X/MI355X with
v0.31.0: there it runs AITER's new gfx950 Gluon kernel, which decodes at 110 tokens/s and,
with MTP, falls from 200 to 72 tokens/s between ~200 and ~30k prompt tokens
([vllm-project/vllm#60408](https://github.com/vllm-project/vllm/issues/60408)). MI325X is
unaffected (126 tokens/s; 182 to 162 with MTP).

### Client Usage

```python
from openai import OpenAI

client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1")
resp = client.chat.completions.create(
    model="Qwen/Qwen3.5-122B-A10B",
    messages=[{"role": "user", "content": "Summarize the gated delta networks paper."}],
    max_tokens=512,
)
print(resp.choices[0].message.content)
```

### Troubleshooting

- **CUDA graph / Mamba cache size error:** reduce `--max-cudagraph-capture-size`
  (default 512). See [vLLM PR #34571](https://github.com/vllm-project/vllm/pull/34571).
- **Disable reasoning:** add `--default-chat-template-kwargs '{"enable_thinking": false}'`.
- **Prefix Caching (Mamba):** currently experimental in "align" mode.

### References

- [Model card](https://huggingface.co/Qwen/Qwen3.5-122B-A10B)
- [FP8 checkpoint](https://huggingface.co/Qwen/Qwen3.5-122B-A10B-FP8)
- [GPTQ-Int4 checkpoint](https://huggingface.co/Qwen/Qwen3.5-122B-A10B-GPTQ-Int4)
- [Qwen3.5-397B-A17B recipe](../Qwen3.5-397B-A17B)