{ "hf_id": "Qwen/Qwen3.5-4B", "meta": { "title": "Qwen3.5-4B", "slug": "qwen3.5-4b", "provider": "Qwen", "description": "Qwen3.5 compact dense multimodal model (4B) — fits on 16 GB consumer GPUs with full 262K context or one Xeon 6 NUMA node or Intel Arc Pro B60/B70", "date_added": "2026-04-22", "date_updated": "2026-09-14", "tasks": [ "multimodal", "text" ], "performance_headline": "Consumer-GPU-friendly Qwen3.5 dense with MTP support", "related_recipes": [ "Qwen/Qwen3.5-9B", "Qwen/Qwen3.5-2B" ], "hardware": { "xeon6": "verified", "arc_pro_b60": "verified", "arc_pro_b70": "verified" } }, "recommended_command": { "hardware": "arc_pro_b60", "strategy": "single_node_tp", "variant": "default", "node_count": 1, "deploy_type": "single_node", "env": { "VLLM_USE_RUST_FRONTEND": "1" }, "docker_image": "vllm/vllm-openai-xpu:latest", "command": "vllm serve Qwen/Qwen3.5-4B \\\n --trust-remote-code \\\n --tensor-parallel-size 1 \\\n --enforce-eager \\\n --enable-auto-tool-choice \\\n --tool-call-parser qwen3_coder \\\n --reasoning-parser qwen3", "argv": [ "vllm", "serve", "Qwen/Qwen3.5-4B", "--trust-remote-code", "--tensor-parallel-size", "1", "--enforce-eager", "--enable-auto-tool-choice", "--tool-call-parser", "qwen3_coder", "--reasoning-parser", "qwen3" ], "docker_command": "docker run --device /dev/dri \\\n -v /dev/dri/by-path:/dev/dri/by-path --shm-size=16g \\\n --privileged --ipc=host -p 8000:8000 \\\n -v ~/.cache/huggingface:/root/.cache/huggingface \\\n -e VLLM_USE_RUST_FRONTEND=1 \\\n vllm/vllm-openai-xpu:latest Qwen/Qwen3.5-4B \\\n --trust-remote-code \\\n --tensor-parallel-size 1 \\\n --enforce-eager \\\n --enable-auto-tool-choice \\\n --tool-call-parser qwen3_coder \\\n --reasoning-parser qwen3", "docker_argv": [ "docker", "run", "--device", "/dev/dri", "-v", "/dev/dri/by-path:/dev/dri/by-path", "--shm-size", "16g", "--privileged", "--ipc=host", "-p", "8000:8000", "-v", "~/.cache/huggingface:/root/.cache/huggingface", "-e", "VLLM_USE_RUST_FRONTEND=1", "vllm/vllm-openai-xpu:latest", "Qwen/Qwen3.5-4B", "--trust-remote-code", "--tensor-parallel-size", "1", "--enforce-eager", "--enable-auto-tool-choice", "--tool-call-parser", "qwen3_coder", "--reasoning-parser", "qwen3" ], "strategy_spec": { "name": "single_node_tp", "deploy_type": "single_node", "display_name": "Tensor Parallel", "short_label": "TP", "orientation": "latency", "description": "Single-node tensor parallel. Splits the model across all local GPUs. TP size is set to the GPU count at deploy time. The simplest multi-GPU strategy — works for all model architectures.\n", "hardware_match": { "min_gpus": 1, "max_gpus": 8, "multi_node": false }, "vllm_args": [], "parallel_flag": "--tensor-parallel-size" }, "hardware_profile": { "brand": "Intel", "generation": "xpu", "display_name": "Arc Pro B60", "description": "Intel Arc Pro B60 24 GB · 4-card node · vLLM XPU backend", "gpu_count": 4, "vram_gb": 96, "multi_node": false, "restricted": true }, "by_hardware": { "arc_pro_b60": "/Qwen/Qwen3.5-4B/hw/arc_pro_b60.json", "xeon6": "/Qwen/Qwen3.5-4B/hw/xeon6.json", "arc_pro_b70": "/Qwen/Qwen3.5-4B/hw/arc_pro_b70.json" } }, "model": { "model_id": "Qwen/Qwen3.5-4B", "min_vllm_version": "0.17.0", "default_frontend": "rust", "architecture": "dense", "parameter_count": "4B", "active_parameters": "4B", "context_length": 262144, "base_args": [ "--trust-remote-code" ], "base_env": {}, "install": { "pip": { "command": "uv venv\nsource .venv/bin/activate\nuv pip install -U vllm --torch-backend auto" }, "docker": { "command": "docker pull vllm/vllm-openai:latest" } } }, "features": { "tool_calling": { "description": "Enable automatic tool choice with Qwen3 Coder parser", "args": [ "--enable-auto-tool-choice", "--tool-call-parser", "qwen3_coder" ] }, "reasoning": { "description": "Enable chain-of-thought reasoning with Qwen3 parser", "args": [ "--reasoning-parser", "qwen3" ] }, "spec_decoding": { "description": "Multi-token prediction speculative decoding for lower latency", "args": [ "--speculative-config", "{\"method\":\"mtp\",\"num_speculative_tokens\":1}" ] }, "text_only": { "description": "Skip loading the vision encoder for text-only workloads.", "args": [ "--language-model-only" ] }, "long_context": { "description": "YaRN RoPE scaling past the native 262,144-token window to ~1,010,000 tokens. Static YaRN holds the scaling factor constant regardless of input length, so leave this off unless you actually serve long prompts; for ~524K, halve the factor to 2.0.", "args": [ "--hf-overrides", "{\"text_config\": {\"rope_parameters\": {\"mrope_interleaved\": true, \"mrope_section\": [11, 11, 10], \"rope_type\": \"yarn\", \"rope_theta\": 10000000, \"partial_rotary_factor\": 0.25, \"factor\": 4.0, \"original_max_position_embeddings\": 262144}}}", "--max-model-len", "1010000" ], "env": { "VLLM_ALLOW_LONG_MAX_MODEL_LEN": "1" } } }, "opt_in_features": [ "spec_decoding", "text_only", "long_context" ], "variants": { "default": { "precision": "bf16", "vram_minimum_gb": 10, "description": "Full precision BF16 — fits on a single 16 GB GPU or one Xeon 6 NUMA node or one Intel Arc Pro B60/B70", "supported_hardware": [ "xeon6", "arc_pro_b60", "arc_pro_b70" ] }, "fp8": { "model_id": "RedHatAI/Qwen3.5-4B-FP8-dynamic", "precision": "fp8", "vram_minimum_gb": 5, "description": "RedHatAI FP8 (E4M3) with dynamic per-token activation quantization", "supported_hardware": [ "arc_pro_b60", "arc_pro_b70" ], "extra_args": [ "--kv-cache-dtype", "fp8" ], "json": "/RedHatAI/Qwen3.5-4B-FP8-dynamic.json" } }, "hardware_overrides": { "xpu": { "extra_args": [ "--enforce-eager" ] } }, "guide": "## Overview\n\n[Qwen3.5-4B](https://huggingface.co/Qwen/Qwen3.5-4B) is the compact dense\nentry in the Qwen3.5 family — same gated delta networks architecture, vision\nencoder, 262K context, and MTP decoding as the larger siblings, sized for\n16 GB consumer GPUs.\n\n## Prerequisites\n\n- **vLLM version:** >= 0.17.0\n- **Hardware:** single 16 GB GPU (RTX 4080 / L4 / A10 / T4-24GB) or one Xeon 6 NUMA node or one Intel Arc Pro B60/B70\n\n### Install vLLM\n\n```bash\nuv venv\nsource .venv/bin/activate\nuv pip install -U vllm --torch-backend=auto\n```\n\n### pip (Intel Xeon 6 CPUs)\nFor Intel and AMD x86 CPUs, follow the [CPU pre-built wheels](https://docs.vllm.ai/en/latest/getting_started/installation/cpu/#pre-built-wheels) installation instructions.\n\n### Docker \n```bash\ndocker pull vllm/vllm-openai-xpu:latest # Intel XPU (B60 / B70)\n```\n\n## Launching the Server\n\n```bash\nvllm serve Qwen/Qwen3.5-4B \\\n --max-model-len 262144 \\\n --reasoning-parser qwen3\n```\n\n### Intel Xeon 6 Deployment via Docker\n\nLaunch the x86 CPU vLLM Docker container for `Qwen/Qwen3.5-4B`:\n\n```bash\ndocker run \\\n --privileged --ipc=host -p 8000:8000 \\\n -v ~/.cache/huggingface:/root/.cache/huggingface \\\n vllm/vllm-openai-cpu:latest-x86_64 Qwen/Qwen3.5-4B \\\n --trust-remote-code \\\n --tensor-parallel-size 1 \\\n --enable-auto-tool-choice \\\n --tool-call-parser qwen3_coder \\\n --reasoning-parser qwen3\n```\n\n### Docker (Intel XPU B60 / B70)\n\nValidated on 1× Intel Arc Pro B60 / B70 (B60 24 GB, B70 32 GB per card) with the\nofficial vLLM XPU image `vllm/vllm-openai-xpu:latest`.\n\n```bash\ndocker run --device /dev/dri \\\n -v /dev/dri/by-path:/dev/dri/by-path --shm-size=16g \\\n --privileged --ipc=host -p 8000:8000 \\\n -v ~/.cache/huggingface:/root/.cache/huggingface \\\n vllm/vllm-openai-xpu:latest \\\n Qwen/Qwen3.5-4B \\\n --reasoning-parser qwen3 \\\n --enforce-eager\n```\n\n\n### MTP speculative decoding\n\n```bash\nvllm serve Qwen/Qwen3.5-4B \\\n --speculative-config '{\"method\": \"mtp\", \"num_speculative_tokens\": 1}' \\\n --reasoning-parser qwen3\n```\n\n## Client Usage\n\n```python\nfrom openai import OpenAI\n\nclient = OpenAI(api_key=\"EMPTY\", base_url=\"http://localhost:8000/v1\")\nresp = client.chat.completions.create(\n model=\"Qwen/Qwen3.5-4B\",\n messages=[{\"role\": \"user\", \"content\": \"Hello!\"}],\n max_tokens=128,\n)\nprint(resp.choices[0].message.content)\n```\n\n## Troubleshooting\n\n- **CUDA graph / Mamba cache size error:** reduce `--max-cudagraph-capture-size`.\n- **Disable reasoning:** add `--default-chat-template-kwargs '{\"enable_thinking\": false}'`.\n\n## References\n\n- [Model card](https://huggingface.co/Qwen/Qwen3.5-4B)\n- [Base checkpoint](https://huggingface.co/Qwen/Qwen3.5-4B-Base)\n", "source": { "source": "vllm", "repo": "vllm-project/vllm", "ref": "main", "sha": "564ac0de1b228568cafcbfbe8ccaccc82d6023eb", "path": "recipes/models/Qwen/Qwen3.5-4B.yaml", "edit_url": "https://github.com/vllm-project/vllm/edit/main/recipes/models/Qwen/Qwen3.5-4B.yaml", "overlays": [] } }