{ "hf_id": "Qwen/Qwen3.5-122B-A10B", "meta": { "title": "Qwen3.5-122B-A10B", "slug": "qwen3.5-122b-a10b", "provider": "Qwen", "description": "Mid-size Qwen3.5 multimodal MoE (122B total / 10B active) with gated delta networks, 256 experts, and 262K context", "date_added": "2026-04-22", "date_updated": "2026-10-07", "tasks": [ "multimodal", "text" ], "performance_headline": "Qwen3.5 mid-tier MoE — fits on 4x H200 BF16 or 2x H200 FP8", "related_recipes": [ "Qwen/Qwen3.5-397B-A17B", "Qwen/Qwen3.5-35B-A3B" ], "hardware": { "mi325x": "verified", "mi355x": "verified" } }, "recommended_command": { "hardware": "h200", "strategy": "single_node_tp", "variant": "default", "node_count": 1, "deploy_type": "single_node", "env": { "VLLM_USE_RUST_FRONTEND": "1" }, "docker_image": "vllm/vllm-openai:latest", "command": "vllm serve Qwen/Qwen3.5-122B-A10B \\\n --trust-remote-code \\\n --tensor-parallel-size 8 \\\n --enable-auto-tool-choice \\\n --tool-call-parser qwen3_coder \\\n --reasoning-parser qwen3 \\\n --mm-encoder-tp-mode data", "argv": [ "vllm", "serve", "Qwen/Qwen3.5-122B-A10B", "--trust-remote-code", "--tensor-parallel-size", "8", "--enable-auto-tool-choice", "--tool-call-parser", "qwen3_coder", "--reasoning-parser", "qwen3", "--mm-encoder-tp-mode", "data" ], "docker_command": "docker run --gpus all \\\n --privileged --ipc=host -p 8000:8000 \\\n -v ~/.cache/huggingface:/root/.cache/huggingface \\\n -e VLLM_USE_RUST_FRONTEND=1 \\\n vllm/vllm-openai:latest Qwen/Qwen3.5-122B-A10B \\\n --trust-remote-code \\\n --tensor-parallel-size 8 \\\n --enable-auto-tool-choice \\\n --tool-call-parser qwen3_coder \\\n --reasoning-parser qwen3 \\\n --mm-encoder-tp-mode data", "docker_argv": [ "docker", "run", "--gpus", "all", "--privileged", "--ipc=host", "-p", "8000:8000", "-v", "~/.cache/huggingface:/root/.cache/huggingface", "-e", "VLLM_USE_RUST_FRONTEND=1", "vllm/vllm-openai:latest", "Qwen/Qwen3.5-122B-A10B", "--trust-remote-code", "--tensor-parallel-size", "8", "--enable-auto-tool-choice", "--tool-call-parser", "qwen3_coder", "--reasoning-parser", "qwen3", "--mm-encoder-tp-mode", "data" ], "strategy_spec": { "name": "single_node_tp", "deploy_type": "single_node", "display_name": "Tensor Parallel", "short_label": "TP", "orientation": "latency", "description": "Single-node tensor parallel. Splits the model across all local GPUs. TP size is set to the GPU count at deploy time. The simplest multi-GPU strategy — works for all model architectures.\n", "hardware_match": { "min_gpus": 1, "max_gpus": 8, "multi_node": false }, "vllm_args": [], "parallel_flag": "--tensor-parallel-size" }, "hardware_profile": { "brand": "NVIDIA", "generation": "hopper", "compute_arch": "sm90", "display_name": "H200", "description": "NVIDIA H200 SXM 141 GB HBM3e · 8-GPU HGX node", "gpu_count": 8, "vram_gb": 1128, "multi_node": false }, "alternatives": { "single_node_tep": "/Qwen/Qwen3.5-122B-A10B/strategies/single_node_tep.json", "single_node_dep": "/Qwen/Qwen3.5-122B-A10B/strategies/single_node_dep.json", "multi_node_tp": "/Qwen/Qwen3.5-122B-A10B/strategies/multi_node_tp.json", "multi_node_dep": "/Qwen/Qwen3.5-122B-A10B/strategies/multi_node_dep.json", "multi_node_tep": "/Qwen/Qwen3.5-122B-A10B/strategies/multi_node_tep.json", "pd_cluster": "/Qwen/Qwen3.5-122B-A10B/strategies/pd_cluster.json", "kv_store_centralized_mooncake": "/Qwen/Qwen3.5-122B-A10B/strategies/kv_store_centralized_mooncake.json", "kv_store_distributed_mooncake": "/Qwen/Qwen3.5-122B-A10B/strategies/kv_store_distributed_mooncake.json" }, "by_hardware": { "h200": "/Qwen/Qwen3.5-122B-A10B/hw/h200.json", "h100": "/Qwen/Qwen3.5-122B-A10B/hw/h100.json", "b200": "/Qwen/Qwen3.5-122B-A10B/hw/b200.json", "gb200": "/Qwen/Qwen3.5-122B-A10B/hw/gb200.json", "b300": "/Qwen/Qwen3.5-122B-A10B/hw/b300.json", "gb300": "/Qwen/Qwen3.5-122B-A10B/hw/gb300.json", "mi300x": "/Qwen/Qwen3.5-122B-A10B/hw/mi300x.json", "mi325x": "/Qwen/Qwen3.5-122B-A10B/hw/mi325x.json", "mi350x": "/Qwen/Qwen3.5-122B-A10B/hw/mi350x.json", "mi355x": "/Qwen/Qwen3.5-122B-A10B/hw/mi355x.json" } }, "model": { "model_id": "Qwen/Qwen3.5-122B-A10B", "min_vllm_version": "0.17.0", "default_frontend": "rust", "docker_image": { "amd": "vllm/vllm-openai-rocm:v0.31.0" }, "architecture": "moe", "parameter_count": "122B", "active_parameters": "10B", "context_length": 262144, "base_args": [ "--trust-remote-code" ], "base_env": {}, "install": { "pip": { "command": "uv venv\nsource .venv/bin/activate\nuv pip install -U vllm --torch-backend auto" }, "docker": { "command": "docker pull vllm/vllm-openai:latest" } } }, "features": { "tool_calling": { "description": "Enable automatic tool choice with Qwen3 Coder parser", "args": [ "--enable-auto-tool-choice", "--tool-call-parser", "qwen3_coder" ] }, "reasoning": { "description": "Enable chain-of-thought reasoning with Qwen3 parser", "args": [ "--reasoning-parser", "qwen3" ] }, "spec_decoding": { "description": "Multi-token prediction speculative decoding for lower latency", "args": [ "--speculative-config", "{\"method\":\"mtp\",\"num_speculative_tokens\":1}" ] }, "text_only": { "description": "Skip loading the vision encoder for text-only workloads — frees VRAM for KV cache. Mutually exclusive with encoder_parallel.", "args": [ "--language-model-only" ] }, "encoder_parallel": { "description": "Run the vision encoder in data-parallel mode — avoids TP comm overhead on the small encoder. Mutually exclusive with text_only.", "args": [ "--mm-encoder-tp-mode", "data" ] }, "long_context": { "description": "YaRN RoPE scaling past the native 262,144-token window to ~1,010,000 tokens. Static YaRN holds the scaling factor constant regardless of input length, so leave this off unless you actually serve long prompts; for ~524K, halve the factor to 2.0.", "args": [ "--hf-overrides", "{\"text_config\": {\"rope_parameters\": {\"mrope_interleaved\": true, \"mrope_section\": [11, 11, 10], \"rope_type\": \"yarn\", \"rope_theta\": 10000000, \"partial_rotary_factor\": 0.25, \"factor\": 4.0, \"original_max_position_embeddings\": 262144}}}", "--max-model-len", "1010000" ], "env": { "VLLM_ALLOW_LONG_MAX_MODEL_LEN": "1" } } }, "opt_in_features": [ "spec_decoding", "text_only", "long_context" ], "variants": { "default": { "precision": "bf16", "vram_minimum_gb": 293, "description": "Full precision BF16 — requires 4x H200 or equivalent" }, "fp8": { "model_id": "Qwen/Qwen3.5-122B-A10B-FP8", "precision": "fp8", "vram_minimum_gb": 147, "description": "Qwen official FP8 checkpoint — fits on 2x H200", "hardware_overrides": { "amd": { "extra_args": [ "--kv-cache-dtype", "fp8" ] } }, "json": "/Qwen/Qwen3.5-122B-A10B-FP8.json" }, "nvfp4": { "model_id": "RedHatAI/Qwen3.5-122B-A10B-NVFP4", "precision": "nvfp4", "vram_minimum_gb": 74, "description": "RedHatAI compressed-tensors NVFP4 quantization for Blackwell GPUs", "extra_args": [ "--kv-cache-dtype", "fp8" ], "json": "/RedHatAI/Qwen3.5-122B-A10B-NVFP4.json" }, "gptq_int4": { "model_id": "Qwen/Qwen3.5-122B-A10B-GPTQ-Int4", "precision": "int4", "vram_minimum_gb": 74, "description": "GPTQ Int4 checkpoint — single-GPU serving on 80GB hardware", "json": "/Qwen/Qwen3.5-122B-A10B-GPTQ-Int4.json" }, "redhat_fp8": { "model_id": "RedHatAI/Qwen3.5-122B-A10B-FP8-dynamic", "precision": "fp8", "label": "FP8 Dynamic (RedHat)", "vram_minimum_gb": 147, "description": "RedHatAI FP8 (E4M3) with dynamic per-token activation quantization", "extra_args": [ "--kv-cache-dtype", "fp8" ], "json": "/RedHatAI/Qwen3.5-122B-A10B-FP8-dynamic.json" } }, "hardware_overrides": { "amd": { "extra_args": [ "--attention-backend", "ROCM_AITER_FA" ], "extra_env": { "VLLM_ROCM_USE_AITER": "1" } } }, "guide": "## Overview\n\n[Qwen3.5-122B-A10B](https://huggingface.co/Qwen/Qwen3.5-122B-A10B) is a mid-tier\nmember of the Qwen3.5 family, sharing the gated delta networks MoE architecture\nwith 122B total parameters and 10B activated per token (256 experts). It is\nmultimodal (vision + text) and natively supports 262K context.\n\n## Prerequisites\n\n- **vLLM version:** >= 0.17.0\n- **Hardware (BF16):** 4x H200 or 8x H100\n- **Hardware (FP8):** 2x H200 or 4x H100\n- **Hardware (Int4):** single 80 GB GPU\n\n### Install vLLM\n\n```bash\nuv venv\nsource .venv/bin/activate\nuv pip install -U vllm --torch-backend=auto\n```\n\n## Launching the Server\n\n### BF16 on 4xH200 (TP4)\n\n```bash\nvllm serve Qwen/Qwen3.5-122B-A10B \\\n --tensor-parallel-size 4 \\\n --max-model-len 262144 \\\n --reasoning-parser qwen3\n```\n\n### FP8 on 2xH200 (TP2)\n\n```bash\nvllm serve Qwen/Qwen3.5-122B-A10B-FP8 \\\n --tensor-parallel-size 2 \\\n --max-model-len 262144 \\\n --reasoning-parser qwen3\n```\n\n### Throughput-focused (text-only, EP)\n\n```bash\nvllm serve Qwen/Qwen3.5-122B-A10B-FP8 \\\n -dp 4 --enable-expert-parallel \\\n --language-model-only \\\n --reasoning-parser qwen3 \\\n --enable-prefix-caching\n```\n\n### MTP speculative decoding\n\n```bash\nvllm serve Qwen/Qwen3.5-122B-A10B-FP8 \\\n --tensor-parallel-size 2 \\\n --speculative-config '{\"method\": \"mtp\", \"num_speculative_tokens\": 1}' \\\n --reasoning-parser qwen3\n```\n\n### AMD (MI325X / MI355X, single GPU, FP8)\n\nThe FP8 checkpoint fits one MI300X-class GPU. Three settings matter on ROCm and none is on by\ndefault: `VLLM_ROCM_USE_AITER=1` switches the MoE to the AITER FP8 kernels,\n`--attention-backend ROCM_AITER_FA` keeps decode flat across context (the auto-selected\n`ROCM_ATTN` decays with prompt length on this hybrid GDN model), and `--kv-cache-dtype fp8`\nkeeps MTP flat across context too.\n\n```bash\nVLLM_ROCM_USE_AITER=1 vllm serve Qwen/Qwen3.5-122B-A10B-FP8 \\\n --attention-backend ROCM_AITER_FA \\\n --kv-cache-dtype fp8 --max-num-seqs 64 \\\n --reasoning-parser qwen3\n```\n\n`--max-num-seqs 64` because the default exceeds the Mamba block budget on 192 GB. MTP\n(`--speculative-config '{\"method\":\"mtp\",\"num_speculative_tokens\":1}'`) works with the same\nflags. Single-stream decode (tokens/s at ~200 / ~2k / ~12k / ~30k prompt tokens, 128 output\ntokens, prefix caching off), measured 2026-10-07 on `vllm/vllm-openai-rocm:v0.31.0`:\n\n| | MI325X | MI355X |\n|---|---|---|\n| AITER + `ROCM_AITER_FA` | 128 / 127 / 127 / 127 | 143 / 143 / 143 / 143 |\n| + MTP | 184 / 169 / 172 / 164 | 204 / 193 / 195 / 190 |\n\nThis recipe used `ROCM_AITER_UNIFIED_ATTN` before v0.31.0. Avoid it on MI350X/MI355X with\nv0.31.0: there it runs AITER's new gfx950 Gluon kernel, which decodes at 110 tokens/s and,\nwith MTP, falls from 200 to 72 tokens/s between ~200 and ~30k prompt tokens\n([vllm-project/vllm#60408](https://github.com/vllm-project/vllm/issues/60408)). MI325X is\nunaffected (126 tokens/s; 182 to 162 with MTP).\n\n## Client Usage\n\n```python\nfrom openai import OpenAI\n\nclient = OpenAI(api_key=\"EMPTY\", base_url=\"http://localhost:8000/v1\")\nresp = client.chat.completions.create(\n model=\"Qwen/Qwen3.5-122B-A10B\",\n messages=[{\"role\": \"user\", \"content\": \"Summarize the gated delta networks paper.\"}],\n max_tokens=512,\n)\nprint(resp.choices[0].message.content)\n```\n\n## Troubleshooting\n\n- **CUDA graph / Mamba cache size error:** reduce `--max-cudagraph-capture-size`\n (default 512). See [vLLM PR #34571](https://github.com/vllm-project/vllm/pull/34571).\n- **Disable reasoning:** add `--default-chat-template-kwargs '{\"enable_thinking\": false}'`.\n- **Prefix Caching (Mamba):** currently experimental in \"align\" mode.\n\n## References\n\n- [Model card](https://huggingface.co/Qwen/Qwen3.5-122B-A10B)\n- [FP8 checkpoint](https://huggingface.co/Qwen/Qwen3.5-122B-A10B-FP8)\n- [GPTQ-Int4 checkpoint](https://huggingface.co/Qwen/Qwen3.5-122B-A10B-GPTQ-Int4)\n- [Qwen3.5-397B-A17B recipe](../Qwen3.5-397B-A17B)\n", "source": { "source": "vllm", "repo": "vllm-project/vllm", "ref": "main", "sha": "564ac0de1b228568cafcbfbe8ccaccc82d6023eb", "path": "recipes/models/Qwen/Qwen3.5-122B-A10B.yaml", "edit_url": "https://github.com/vllm-project/vllm/edit/main/recipes/models/Qwen/Qwen3.5-122B-A10B.yaml", "overlays": [] } }