# Qwen/Qwen3.5-4B — vLLM serving recipe

> Qwen3.5 compact dense multimodal model (4B) — fits on 16 GB consumer GPUs with full 262K context or one Xeon 6 NUMA node or Intel Arc Pro B60/B70

Configuration: hardware `arc_pro_b60` (4× Arc Pro B60) · strategy `single_node_tp` (Tensor Parallel) · variant `default` (BF16) · features `tool_calling`, `reasoning` · frontend `rust`

- Interactive page: https://recipes.vllm.ai/Qwen/Qwen3.5-4B
- This configuration as JSON: https://recipes.vllm.ai/Qwen/Qwen3.5-4B.json?hardware=arc_pro_b60

The commands below were rendered for exactly this configuration and take precedence over any example in the guide.

## Other configurations

Change the query string (the same parameters as the page URL) and fetch this URL again. Unknown or incompatible values fall back to the recipe default, so check the Configuration line.

- `hardware`: `xeon6`, `arc_pro_b60`, `arc_pro_b70`
- `variant`: `default`, `fp8`
- `strategy`: `single_node_tp` (multi-node strategies also need `nodes=<n>`)
- `features`: comma-separated list of `tool_calling`, `reasoning`, `spec_decoding`, `text_only`, `long_context` (default: `tool_calling`, `reasoning`)

## Install

Requires vLLM >= 0.17.0.

Docker image: `vllm/vllm-openai-xpu:latest`

## Serve

```bash
export VLLM_USE_RUST_FRONTEND=1
vllm serve Qwen/Qwen3.5-4B \
  --trust-remote-code \
  --tensor-parallel-size 1 \
  --enforce-eager \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --reasoning-parser qwen3
```

Docker equivalent:

```bash
docker run --device /dev/dri \
  -v /dev/dri/by-path:/dev/dri/by-path --shm-size=16g \
  --privileged --ipc=host -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -e VLLM_USE_RUST_FRONTEND=1 \
  vllm/vllm-openai-xpu:latest Qwen/Qwen3.5-4B \
  --trust-remote-code \
  --tensor-parallel-size 1 \
  --enforce-eager \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --reasoning-parser qwen3
```

## Guide

> Background reference. Commands in it are examples and may target another configuration — use the Serve section above.

### Overview

[Qwen3.5-4B](https://huggingface.co/Qwen/Qwen3.5-4B) is the compact dense
entry in the Qwen3.5 family — same gated delta networks architecture, vision
encoder, 262K context, and MTP decoding as the larger siblings, sized for
16 GB consumer GPUs.

### Prerequisites

- **vLLM version:** >= 0.17.0
- **Hardware:** single 16 GB GPU (RTX 4080 / L4 / A10 / T4-24GB) or one Xeon 6 NUMA node or one Intel Arc Pro B60/B70

#### Install vLLM

```bash
uv venv
source .venv/bin/activate
uv pip install -U vllm --torch-backend=auto
```

#### pip (Intel Xeon 6 CPUs)
For Intel and AMD x86 CPUs, follow the [CPU pre-built wheels](https://docs.vllm.ai/en/latest/getting_started/installation/cpu/#pre-built-wheels) installation instructions.

#### Docker 
```bash
docker pull vllm/vllm-openai-xpu:latest  # Intel XPU (B60 / B70)
```

### Launching the Server

```bash
vllm serve Qwen/Qwen3.5-4B \
  --max-model-len 262144 \
  --reasoning-parser qwen3
```

#### Intel Xeon 6 Deployment via Docker

Launch the x86 CPU vLLM Docker container for `Qwen/Qwen3.5-4B`:

```bash
docker run \
  --privileged --ipc=host -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai-cpu:latest-x86_64 Qwen/Qwen3.5-4B \
  --trust-remote-code \
  --tensor-parallel-size 1 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --reasoning-parser qwen3
```

#### Docker (Intel XPU B60 / B70)

Validated on 1× Intel Arc Pro B60 / B70 (B60 24 GB, B70 32 GB per card) with the
official vLLM XPU image `vllm/vllm-openai-xpu:latest`.

```bash
docker run --device /dev/dri \
  -v /dev/dri/by-path:/dev/dri/by-path --shm-size=16g \
  --privileged --ipc=host -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai-xpu:latest \
  Qwen/Qwen3.5-4B \
  --reasoning-parser qwen3 \
  --enforce-eager
```


#### MTP speculative decoding

```bash
vllm serve Qwen/Qwen3.5-4B \
  --speculative-config '{"method": "mtp", "num_speculative_tokens": 1}' \
  --reasoning-parser qwen3
```

### Client Usage

```python
from openai import OpenAI

client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1")
resp = client.chat.completions.create(
    model="Qwen/Qwen3.5-4B",
    messages=[{"role": "user", "content": "Hello!"}],
    max_tokens=128,
)
print(resp.choices[0].message.content)
```

### Troubleshooting

- **CUDA graph / Mamba cache size error:** reduce `--max-cudagraph-capture-size`.
- **Disable reasoning:** add `--default-chat-template-kwargs '{"enable_thinking": false}'`.

### References

- [Model card](https://huggingface.co/Qwen/Qwen3.5-4B)
- [Base checkpoint](https://huggingface.co/Qwen/Qwen3.5-4B-Base)