{
  "hf_id": "Qwen/Qwen3.5-35B-A3B",
  "meta": {
    "title": "Qwen3.5-35B-A3B",
    "slug": "qwen3.5-35b-a3b",
    "provider": "Qwen",
    "description": "Compact Qwen3.5 multimodal MoE (35B total / 3B active) with gated delta networks, 256 experts, and 262K context",
    "date_added": "2026-04-22",
    "date_updated": "2026-09-30",
    "tasks": [
      "multimodal",
      "text"
    ],
    "performance_headline": "Compact Qwen3.5 MoE — single-GPU FP8, 2x GPU or 2x Xeon 6 NUMA nodes or 4x Intel Arc Pro B60/B70 BF16 serving",
    "related_recipes": [
      "Qwen/Qwen3.5-397B-A17B",
      "Qwen/Qwen3.5-122B-A10B"
    ],
    "hardware": {
      "xeon6": "verified",
      "arc_pro_b60": "verified",
      "arc_pro_b70": "verified"
    }
  },
  "recommended_command": {
    "hardware": "arc_pro_b60",
    "strategy": "single_node_tp",
    "variant": "default",
    "node_count": 1,
    "deploy_type": "single_node",
    "env": {
      "VLLM_USE_RUST_FRONTEND": "1"
    },
    "docker_image": "vllm/vllm-openai-xpu:latest",
    "command": "vllm serve Qwen/Qwen3.5-35B-A3B \\\n  --trust-remote-code \\\n  --tensor-parallel-size 4 \\\n  --enforce-eager \\\n  --enable-auto-tool-choice \\\n  --tool-call-parser qwen3_coder \\\n  --reasoning-parser qwen3 \\\n  --mm-encoder-tp-mode data",
    "argv": [
      "vllm",
      "serve",
      "Qwen/Qwen3.5-35B-A3B",
      "--trust-remote-code",
      "--tensor-parallel-size",
      "4",
      "--enforce-eager",
      "--enable-auto-tool-choice",
      "--tool-call-parser",
      "qwen3_coder",
      "--reasoning-parser",
      "qwen3",
      "--mm-encoder-tp-mode",
      "data"
    ],
    "docker_command": "docker run --device /dev/dri \\\n  -v /dev/dri/by-path:/dev/dri/by-path --shm-size=16g \\\n  --privileged --ipc=host -p 8000:8000 \\\n  -v ~/.cache/huggingface:/root/.cache/huggingface \\\n  -e VLLM_USE_RUST_FRONTEND=1 \\\n  vllm/vllm-openai-xpu:latest Qwen/Qwen3.5-35B-A3B \\\n  --trust-remote-code \\\n  --tensor-parallel-size 4 \\\n  --enforce-eager \\\n  --enable-auto-tool-choice \\\n  --tool-call-parser qwen3_coder \\\n  --reasoning-parser qwen3 \\\n  --mm-encoder-tp-mode data",
    "docker_argv": [
      "docker",
      "run",
      "--device",
      "/dev/dri",
      "-v",
      "/dev/dri/by-path:/dev/dri/by-path",
      "--shm-size",
      "16g",
      "--privileged",
      "--ipc=host",
      "-p",
      "8000:8000",
      "-v",
      "~/.cache/huggingface:/root/.cache/huggingface",
      "-e",
      "VLLM_USE_RUST_FRONTEND=1",
      "vllm/vllm-openai-xpu:latest",
      "Qwen/Qwen3.5-35B-A3B",
      "--trust-remote-code",
      "--tensor-parallel-size",
      "4",
      "--enforce-eager",
      "--enable-auto-tool-choice",
      "--tool-call-parser",
      "qwen3_coder",
      "--reasoning-parser",
      "qwen3",
      "--mm-encoder-tp-mode",
      "data"
    ],
    "strategy_spec": {
      "name": "single_node_tp",
      "deploy_type": "single_node",
      "display_name": "Tensor Parallel",
      "short_label": "TP",
      "orientation": "latency",
      "description": "Single-node tensor parallel. Splits the model across all local GPUs. TP size is set to the GPU count at deploy time. The simplest multi-GPU strategy — works for all model architectures.\n",
      "hardware_match": {
        "min_gpus": 1,
        "max_gpus": 8,
        "multi_node": false
      },
      "vllm_args": [],
      "parallel_flag": "--tensor-parallel-size"
    },
    "hardware_profile": {
      "brand": "Intel",
      "generation": "xpu",
      "display_name": "Arc Pro B60",
      "description": "Intel Arc Pro B60 24 GB · 4-card node · vLLM XPU backend",
      "gpu_count": 4,
      "vram_gb": 96,
      "multi_node": false,
      "restricted": true
    },
    "alternatives": {
      "single_node_tep": "/Qwen/Qwen3.5-35B-A3B/strategies/single_node_tep.json",
      "single_node_dep": "/Qwen/Qwen3.5-35B-A3B/strategies/single_node_dep.json",
      "multi_node_tp": "/Qwen/Qwen3.5-35B-A3B/strategies/multi_node_tp.json",
      "multi_node_dep": "/Qwen/Qwen3.5-35B-A3B/strategies/multi_node_dep.json",
      "multi_node_tep": "/Qwen/Qwen3.5-35B-A3B/strategies/multi_node_tep.json"
    },
    "by_hardware": {
      "arc_pro_b60": "/Qwen/Qwen3.5-35B-A3B/hw/arc_pro_b60.json",
      "xeon6": "/Qwen/Qwen3.5-35B-A3B/hw/xeon6.json",
      "arc_pro_b70": "/Qwen/Qwen3.5-35B-A3B/hw/arc_pro_b70.json"
    }
  },
  "model": {
    "model_id": "Qwen/Qwen3.5-35B-A3B",
    "min_vllm_version": "0.17.0",
    "default_frontend": "rust",
    "architecture": "moe",
    "parameter_count": "35B",
    "active_parameters": "3B",
    "context_length": 262144,
    "base_args": [
      "--trust-remote-code"
    ],
    "base_env": {},
    "install": {
      "pip": {
        "command": "uv venv\nsource .venv/bin/activate\nuv pip install -U vllm --torch-backend auto"
      },
      "docker": {
        "command": "docker pull vllm/vllm-openai:latest"
      }
    }
  },
  "features": {
    "tool_calling": {
      "description": "Enable automatic tool choice with Qwen3 Coder parser",
      "args": [
        "--enable-auto-tool-choice",
        "--tool-call-parser",
        "qwen3_coder"
      ]
    },
    "reasoning": {
      "description": "Enable chain-of-thought reasoning with Qwen3 parser",
      "args": [
        "--reasoning-parser",
        "qwen3"
      ]
    },
    "spec_decoding": {
      "description": "Multi-token prediction speculative decoding for lower latency",
      "args": [
        "--speculative-config",
        "{\"method\":\"mtp\",\"num_speculative_tokens\":1}"
      ]
    },
    "text_only": {
      "description": "Skip loading the vision encoder for text-only workloads — frees VRAM for KV cache. Mutually exclusive with encoder_parallel.",
      "args": [
        "--language-model-only"
      ]
    },
    "encoder_parallel": {
      "description": "Run the vision encoder in data-parallel mode — avoids TP comm overhead on the small encoder. Mutually exclusive with text_only.",
      "args": [
        "--mm-encoder-tp-mode",
        "data"
      ]
    },
    "long_context": {
      "description": "YaRN RoPE scaling past the native 262,144-token window to ~1,010,000 tokens. Static YaRN holds the scaling factor constant regardless of input length, so leave this off unless you actually serve long prompts; for ~524K, halve the factor to 2.0.",
      "args": [
        "--hf-overrides",
        "{\"text_config\": {\"rope_parameters\": {\"mrope_interleaved\": true, \"mrope_section\": [11, 11, 10], \"rope_type\": \"yarn\", \"rope_theta\": 10000000, \"partial_rotary_factor\": 0.25, \"factor\": 4.0, \"original_max_position_embeddings\": 262144}}}",
        "--max-model-len",
        "1010000"
      ],
      "env": {
        "VLLM_ALLOW_LONG_MAX_MODEL_LEN": "1"
      }
    }
  },
  "opt_in_features": [
    "spec_decoding",
    "text_only",
    "long_context"
  ],
  "variants": {
    "default": {
      "precision": "bf16",
      "vram_minimum_gb": 84,
      "description": "Full precision BF16 — fits on 1x H200, 2x H100, 2x Xeon 6 NUMA nodes, or 4x Intel Arc Pro B60 / B70",
      "supported_hardware": [
        "xeon6",
        "arc_pro_b60",
        "arc_pro_b70"
      ]
    },
    "fp8": {
      "model_id": "Qwen/Qwen3.5-35B-A3B-FP8",
      "precision": "fp8",
      "vram_minimum_gb": 42,
      "description": "Qwen official FP8 checkpoint — single-GPU serving",
      "supported_hardware": [
        "arc_pro_b60",
        "arc_pro_b70"
      ],
      "json": "/Qwen/Qwen3.5-35B-A3B-FP8.json"
    },
    "gptq_int4": {
      "model_id": "Qwen/Qwen3.5-35B-A3B-GPTQ-Int4",
      "precision": "int4",
      "vram_minimum_gb": 21,
      "description": "GPTQ Int4 checkpoint — fits on a single 24GB GPU",
      "supported_hardware": [
        "arc_pro_b60",
        "arc_pro_b70"
      ],
      "json": "/Qwen/Qwen3.5-35B-A3B-GPTQ-Int4.json"
    },
    "redhat_fp8": {
      "model_id": "RedHatAI/Qwen3.5-35B-A3B-FP8-dynamic",
      "precision": "fp8",
      "label": "FP8 Dynamic (RedHat)",
      "vram_minimum_gb": 42,
      "description": "RedHatAI FP8 (E4M3) with dynamic per-token activation quantization",
      "supported_hardware": [
        "arc_pro_b60",
        "arc_pro_b70"
      ],
      "extra_args": [
        "--kv-cache-dtype",
        "fp8"
      ],
      "json": "/RedHatAI/Qwen3.5-35B-A3B-FP8-dynamic.json"
    }
  },
  "hardware_overrides": {
    "cpu": {
      "extra_env": {
        "VLLM_ENGINE_READY_TIMEOUT_S": "1800"
      }
    },
    "xpu": {
      "extra_args": [
        "--enforce-eager"
      ]
    }
  },
  "guide": "## Overview\n\n[Qwen3.5-35B-A3B](https://huggingface.co/Qwen/Qwen3.5-35B-A3B) is the smallest\nMoE in the Qwen3.5 family, sharing the gated delta networks architecture with\n35B total parameters and 3B activated per token (256 experts). With FP8 weights\nit fits on a single 80 GB GPU and supports the full 262K context.\n\n## Prerequisites\n\n- **vLLM version:** >= 0.17.0\n- **Hardware (BF16):** 1x H200, 2x H100, 2x Xeon 6 NUMA nodes, 4x Intel Arc Pro B60 / B70\n- **Hardware (FP8):** single H100/H200\n- **Hardware (Int4):** single 24 GB GPU\n\n### Pip Install\n\n#### NVIDIA\n```bash\nuv venv\nsource .venv/bin/activate\nuv pip install -U vllm --torch-backend=auto\n```\n\n#### CPU\nFor Intel and AMD x86 CPUs, follow the [CPU pre-built wheels](https://docs.vllm.ai/en/latest/getting_started/installation/cpu/#pre-built-wheels) installation instructions.\n```bash\nuv venv\nsource .venv/bin/activate\nexport VLLM_VERSION=$(curl -s https://api.github.com/repos/vllm-project/vllm/releases/latest | jq -r .tag_name | sed 's/^v//')\nuv pip install https://github.com/vllm-project/vllm/releases/download/v${VLLM_VERSION}/vllm-${VLLM_VERSION}+cpu-cp38-abi3-manylinux_2_35_x86_64.whl --torch-backend cpu\n```\n\n### Docker\n```bash\ndocker pull vllm/vllm-openai-xpu:latest  # Intel XPU (B60 / B70)\n```\n\n## Launching the Server\n\n### Single-GPU FP8\n\n```bash\nvllm serve Qwen/Qwen3.5-35B-A3B-FP8 \\\n  --max-model-len 262144 \\\n  --reasoning-parser qwen3\n```\n\n### BF16 on 2xH200 (TP2)\n\n```bash\nvllm serve Qwen/Qwen3.5-35B-A3B \\\n  --tensor-parallel-size 2 \\\n  --max-model-len 262144 \\\n  --reasoning-parser qwen3\n```\n\n### Intel Xeon 6 Deployment via Docker\n\nLaunch the x86 CPU vLLM Docker container for `Qwen/Qwen3.5-35B-A3B`:\n\n```bash\ndocker run \\\n  --privileged --ipc=host -p 8000:8000 \\\n  -v ~/.cache/huggingface:/root/.cache/huggingface \\\n  -e VLLM_ENGINE_READY_TIMEOUT_S=1800 \\\n  vllm/vllm-openai-cpu:latest-x86_64 Qwen/Qwen3.5-35B-A3B \\\n  --trust-remote-code \\\n  --tensor-parallel-size 1 \\\n  --enable-auto-tool-choice \\\n  --tool-call-parser qwen3_coder \\\n  --reasoning-parser qwen3 \\\n  --mm-encoder-tp-mode data\n```\n\n### MTP speculative decoding\n\n```bash\nvllm serve Qwen/Qwen3.5-35B-A3B-FP8 \\\n  --speculative-config '{\"method\": \"mtp\", \"num_speculative_tokens\": 1}' \\\n  --reasoning-parser qwen3\n```\n\n### Docker (Intel XPU B60 / B70)\n\nValidated on 4× Intel Arc Pro B60 / B70 (B60 24 GB, B70 32 GB per card) with the\nofficial vLLM XPU image `vllm/vllm-openai-xpu:latest`.\n\n```bash\ndocker run --device /dev/dri \\\n  -v /dev/dri/by-path:/dev/dri/by-path --shm-size=16g \\\n  --privileged --ipc=host -p 8000:8000 \\\n  -v ~/.cache/huggingface:/root/.cache/huggingface \\\n  vllm/vllm-openai-xpu:latest \\\n  Qwen/Qwen3.5-35B-A3B \\\n  --reasoning-parser qwen3 \\\n  --tensor-parallel-size 4 \\\n  --max-model-len 8192 \\\n  --enforce-eager\n```\n\n## Client Usage\n\n```python\nfrom openai import OpenAI\n\nclient = OpenAI(api_key=\"EMPTY\", base_url=\"http://localhost:8000/v1\")\nresp = client.chat.completions.create(\n    model=\"Qwen/Qwen3.5-35B-A3B\",\n    messages=[{\"role\": \"user\", \"content\": \"Explain gated delta networks in one paragraph.\"}],\n    max_tokens=512,\n)\nprint(resp.choices[0].message.content)\n```\n\n## Troubleshooting\n\n- **CPU startup timeout:** a cold weight download followed by CPU compilation\n  can exceed the default 600-second engine readiness timeout. The CPU recipe\n  sets `VLLM_ENGINE_READY_TIMEOUT_S=1800`; set any external health-check\n  deadline higher (for example, 2100 seconds). Increasing only the external\n  deadline does not prevent the frontend from timing out.\n- **CUDA graph / Mamba cache size error:** reduce `--max-cudagraph-capture-size`\n  (default 512). See [vLLM PR #34571](https://github.com/vllm-project/vllm/pull/34571).\n- **Disable reasoning:** add `--default-chat-template-kwargs '{\"enable_thinking\": false}'`.\n- **Prefix Caching (Mamba):** currently experimental in \"align\" mode.\n\n## References\n\n- [Model card](https://huggingface.co/Qwen/Qwen3.5-35B-A3B)\n- [FP8 checkpoint](https://huggingface.co/Qwen/Qwen3.5-35B-A3B-FP8)\n- [GPTQ-Int4 checkpoint](https://huggingface.co/Qwen/Qwen3.5-35B-A3B-GPTQ-Int4)\n- [Qwen3.5-397B-A17B recipe](../Qwen3.5-397B-A17B)\n",
  "source": {
    "source": "vllm",
    "repo": "vllm-project/vllm",
    "ref": "main",
    "sha": "3a90cc6fd0504b52bf47d58e8d70cff4b03f2c19",
    "path": "recipes/models/Qwen/Qwen3.5-35B-A3B.yaml",
    "edit_url": "https://github.com/vllm-project/vllm/edit/main/recipes/models/Qwen/Qwen3.5-35B-A3B.yaml",
    "overlays": []
  }
}