# Qwen/Qwen3.5-35B-A3B — vLLM serving recipe

> Compact Qwen3.5 multimodal MoE (35B total / 3B active) with gated delta networks, 256 experts, and 262K context

Configuration: hardware `arc_pro_b60` (4× Arc Pro B60) · strategy `single_node_tp` (Tensor Parallel) · variant `default` (BF16) · features `tool_calling`, `reasoning`, `encoder_parallel` · frontend `rust`

- Interactive page: https://recipes.vllm.ai/Qwen/Qwen3.5-35B-A3B
- This configuration as JSON: https://recipes.vllm.ai/Qwen/Qwen3.5-35B-A3B.json?hardware=arc_pro_b60

The commands below were rendered for exactly this configuration and take precedence over any example in the guide.

## Other configurations

Change the query string (the same parameters as the page URL) and fetch this URL again. Unknown or incompatible values fall back to the recipe default, so check the Configuration line.

- `hardware`: `xeon6`, `arc_pro_b60`, `arc_pro_b70`
- `variant`: `default`, `fp8`, `gptq_int4`, `redhat_fp8`
- `strategy`: `single_node_tp`, `single_node_tep`, `single_node_dep`, `multi_node_tp`, `multi_node_dep`, `multi_node_tep` (multi-node strategies also need `nodes=<n>`)
- `features`: comma-separated list of `tool_calling`, `reasoning`, `spec_decoding`, `text_only`, `encoder_parallel`, `long_context` (default: `tool_calling`, `reasoning`, `encoder_parallel`)

## Install

Requires vLLM >= 0.17.0.

Docker image: `vllm/vllm-openai-xpu:latest`

## Serve

```bash
export VLLM_USE_RUST_FRONTEND=1
vllm serve Qwen/Qwen3.5-35B-A3B \
  --trust-remote-code \
  --tensor-parallel-size 4 \
  --enforce-eager \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --reasoning-parser qwen3 \
  --mm-encoder-tp-mode data
```

Docker equivalent:

```bash
docker run --device /dev/dri \
  -v /dev/dri/by-path:/dev/dri/by-path --shm-size=16g \
  --privileged --ipc=host -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -e VLLM_USE_RUST_FRONTEND=1 \
  vllm/vllm-openai-xpu:latest Qwen/Qwen3.5-35B-A3B \
  --trust-remote-code \
  --tensor-parallel-size 4 \
  --enforce-eager \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --reasoning-parser qwen3 \
  --mm-encoder-tp-mode data
```

## Guide

> Background reference. Commands in it are examples and may target another configuration — use the Serve section above.

### Overview

[Qwen3.5-35B-A3B](https://huggingface.co/Qwen/Qwen3.5-35B-A3B) is the smallest
MoE in the Qwen3.5 family, sharing the gated delta networks architecture with
35B total parameters and 3B activated per token (256 experts). With FP8 weights
it fits on a single 80 GB GPU and supports the full 262K context.

### Prerequisites

- **vLLM version:** >= 0.17.0
- **Hardware (BF16):** 1x H200, 2x H100, 2x Xeon 6 NUMA nodes, 4x Intel Arc Pro B60 / B70
- **Hardware (FP8):** single H100/H200
- **Hardware (Int4):** single 24 GB GPU

#### Pip Install

##### NVIDIA
```bash
uv venv
source .venv/bin/activate
uv pip install -U vllm --torch-backend=auto
```

##### CPU
For Intel and AMD x86 CPUs, follow the [CPU pre-built wheels](https://docs.vllm.ai/en/latest/getting_started/installation/cpu/#pre-built-wheels) installation instructions.
```bash
uv venv
source .venv/bin/activate
export VLLM_VERSION=$(curl -s https://api.github.com/repos/vllm-project/vllm/releases/latest | jq -r .tag_name | sed 's/^v//')
uv pip install https://github.com/vllm-project/vllm/releases/download/v${VLLM_VERSION}/vllm-${VLLM_VERSION}+cpu-cp38-abi3-manylinux_2_35_x86_64.whl --torch-backend cpu
```

#### Docker
```bash
docker pull vllm/vllm-openai-xpu:latest  # Intel XPU (B60 / B70)
```

### Launching the Server

#### Single-GPU FP8

```bash
vllm serve Qwen/Qwen3.5-35B-A3B-FP8 \
  --max-model-len 262144 \
  --reasoning-parser qwen3
```

#### BF16 on 2xH200 (TP2)

```bash
vllm serve Qwen/Qwen3.5-35B-A3B \
  --tensor-parallel-size 2 \
  --max-model-len 262144 \
  --reasoning-parser qwen3
```

#### Intel Xeon 6 Deployment via Docker

Launch the x86 CPU vLLM Docker container for `Qwen/Qwen3.5-35B-A3B`:

```bash
docker run \
  --privileged --ipc=host -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -e VLLM_ENGINE_READY_TIMEOUT_S=1800 \
  vllm/vllm-openai-cpu:latest-x86_64 Qwen/Qwen3.5-35B-A3B \
  --trust-remote-code \
  --tensor-parallel-size 1 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --reasoning-parser qwen3 \
  --mm-encoder-tp-mode data
```

#### MTP speculative decoding

```bash
vllm serve Qwen/Qwen3.5-35B-A3B-FP8 \
  --speculative-config '{"method": "mtp", "num_speculative_tokens": 1}' \
  --reasoning-parser qwen3
```

#### Docker (Intel XPU B60 / B70)

Validated on 4× Intel Arc Pro B60 / B70 (B60 24 GB, B70 32 GB per card) with the
official vLLM XPU image `vllm/vllm-openai-xpu:latest`.

```bash
docker run --device /dev/dri \
  -v /dev/dri/by-path:/dev/dri/by-path --shm-size=16g \
  --privileged --ipc=host -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai-xpu:latest \
  Qwen/Qwen3.5-35B-A3B \
  --reasoning-parser qwen3 \
  --tensor-parallel-size 4 \
  --max-model-len 8192 \
  --enforce-eager
```

### Client Usage

```python
from openai import OpenAI

client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1")
resp = client.chat.completions.create(
    model="Qwen/Qwen3.5-35B-A3B",
    messages=[{"role": "user", "content": "Explain gated delta networks in one paragraph."}],
    max_tokens=512,
)
print(resp.choices[0].message.content)
```

### Troubleshooting

- **CPU startup timeout:** a cold weight download followed by CPU compilation
  can exceed the default 600-second engine readiness timeout. The CPU recipe
  sets `VLLM_ENGINE_READY_TIMEOUT_S=1800`; set any external health-check
  deadline higher (for example, 2100 seconds). Increasing only the external
  deadline does not prevent the frontend from timing out.
- **CUDA graph / Mamba cache size error:** reduce `--max-cudagraph-capture-size`
  (default 512). See [vLLM PR #34571](https://github.com/vllm-project/vllm/pull/34571).
- **Disable reasoning:** add `--default-chat-template-kwargs '{"enable_thinking": false}'`.
- **Prefix Caching (Mamba):** currently experimental in "align" mode.

### References

- [Model card](https://huggingface.co/Qwen/Qwen3.5-35B-A3B)
- [FP8 checkpoint](https://huggingface.co/Qwen/Qwen3.5-35B-A3B-FP8)
- [GPTQ-Int4 checkpoint](https://huggingface.co/Qwen/Qwen3.5-35B-A3B-GPTQ-Int4)
- [Qwen3.5-397B-A17B recipe](../Qwen3.5-397B-A17B)