{ "hf_id": "Qwen/Qwen3.8-2.4T-A95B", "meta": { "title": "Qwen3.8-2.4T-A95B", "slug": "qwen3.8-2.4t-a95b", "provider": "Qwen", "description": "2.4T-parameter hybrid-attention MoE (~95B active) with linear attention on 69 of 92 layers, 512 routed experts, a built-in MTP draft head, 262K native context window and extensible to 1M context", "date_added": "2026-08-10", "date_updated": "2026-09-14", "tasks": [ "text" ], "performance_headline": "NVFP4 W4A4 on 8 Blackwell GPUs: 304 output tok/s per user with MTP-3, up to 4300 total tok/s per GPU at DEP16", "related_recipes": [ "Qwen/Qwen3.5-397B-A17B" ], "default_hardware": "b300", "hardware": { "b300": "verified", "gb300": "verified" } }, "recommended_command": { "hardware": "b300", "strategy": "multi_node_tp", "variant": "default", "node_count": 2, "deploy_type": "multi_node", "env": { "VLLM_ENGINE_READY_TIMEOUT_S": "3600", "GLOO_SOCKET_IFNAME": "$IFACE_NAME", "NCCL_SOCKET_IFNAME": "$IFACE_NAME", "VLLM_USE_RUST_FRONTEND": "1" }, "docker_image": "vllm/vllm-openai:qwen38", "head_command": "vllm serve Qwen/Qwen3.8-2.4T-A95B \\\n --tensor-parallel-size 16 \\\n --nnodes 2 \\\n --node-rank 0 \\\n --master-addr $HEAD_IP \\\n --enable-auto-tool-choice \\\n --tool-call-parser qwen3_coder \\\n --reasoning-parser qwen3", "worker_command": "vllm serve Qwen/Qwen3.8-2.4T-A95B \\\n --tensor-parallel-size 16 \\\n --nnodes 2 \\\n --node-rank 1 \\\n --master-addr $HEAD_IP \\\n --headless \\\n --enable-auto-tool-choice \\\n --tool-call-parser qwen3_coder \\\n --reasoning-parser qwen3", "worker_commands": [ "vllm serve Qwen/Qwen3.8-2.4T-A95B \\\n --tensor-parallel-size 16 \\\n --nnodes 2 \\\n --node-rank 1 \\\n --master-addr $HEAD_IP \\\n --headless \\\n --enable-auto-tool-choice \\\n --tool-call-parser qwen3_coder \\\n --reasoning-parser qwen3" ], "head_argv": [ "vllm", "serve", "Qwen/Qwen3.8-2.4T-A95B", "--tensor-parallel-size", "16", "--nnodes", "2", "--node-rank", "0", "--master-addr", "$HEAD_IP", "--enable-auto-tool-choice", "--tool-call-parser", "qwen3_coder", "--reasoning-parser", "qwen3" ], "worker_argv": [ "vllm", "serve", "Qwen/Qwen3.8-2.4T-A95B", "--tensor-parallel-size", "16", "--nnodes", "2", "--node-rank", "1", "--master-addr", "$HEAD_IP", "--headless", "--enable-auto-tool-choice", "--tool-call-parser", "qwen3_coder", "--reasoning-parser", "qwen3" ], "worker_argvs": [ [ "vllm", "serve", "Qwen/Qwen3.8-2.4T-A95B", "--tensor-parallel-size", "16", "--nnodes", "2", "--node-rank", "1", "--master-addr", "$HEAD_IP", "--headless", "--enable-auto-tool-choice", "--tool-call-parser", "qwen3_coder", "--reasoning-parser", "qwen3" ] ], "head_docker_command": "docker run --gpus all \\\n --privileged --ipc=host -p 8000:8000 \\\n -v ~/.cache/huggingface:/root/.cache/huggingface \\\n -e VLLM_ENGINE_READY_TIMEOUT_S=3600 \\\n -e GLOO_SOCKET_IFNAME=$IFACE_NAME \\\n -e NCCL_SOCKET_IFNAME=$IFACE_NAME \\\n -e VLLM_USE_RUST_FRONTEND=1 \\\n vllm/vllm-openai:qwen38 Qwen/Qwen3.8-2.4T-A95B \\\n --tensor-parallel-size 16 \\\n --nnodes 2 \\\n --node-rank 0 \\\n --master-addr $HEAD_IP \\\n --enable-auto-tool-choice \\\n --tool-call-parser qwen3_coder \\\n --reasoning-parser qwen3", "worker_docker_command": "docker run --gpus all \\\n --privileged --ipc=host -p 8000:8000 \\\n -v ~/.cache/huggingface:/root/.cache/huggingface \\\n -e VLLM_ENGINE_READY_TIMEOUT_S=3600 \\\n -e GLOO_SOCKET_IFNAME=$IFACE_NAME \\\n -e NCCL_SOCKET_IFNAME=$IFACE_NAME \\\n -e VLLM_USE_RUST_FRONTEND=1 \\\n vllm/vllm-openai:qwen38 Qwen/Qwen3.8-2.4T-A95B \\\n --tensor-parallel-size 16 \\\n --nnodes 2 \\\n --node-rank 1 \\\n --master-addr $HEAD_IP \\\n --headless \\\n --enable-auto-tool-choice \\\n --tool-call-parser qwen3_coder \\\n --reasoning-parser qwen3", "worker_docker_commands": [ "docker run --gpus all \\\n --privileged --ipc=host -p 8000:8000 \\\n -v ~/.cache/huggingface:/root/.cache/huggingface \\\n -e VLLM_ENGINE_READY_TIMEOUT_S=3600 \\\n -e GLOO_SOCKET_IFNAME=$IFACE_NAME \\\n -e NCCL_SOCKET_IFNAME=$IFACE_NAME \\\n -e VLLM_USE_RUST_FRONTEND=1 \\\n vllm/vllm-openai:qwen38 Qwen/Qwen3.8-2.4T-A95B \\\n --tensor-parallel-size 16 \\\n --nnodes 2 \\\n --node-rank 1 \\\n --master-addr $HEAD_IP \\\n --headless \\\n --enable-auto-tool-choice \\\n --tool-call-parser qwen3_coder \\\n --reasoning-parser qwen3" ], "head_docker_argv": [ "docker", "run", "--gpus", "all", "--privileged", "--ipc=host", "-p", "8000:8000", "-v", "~/.cache/huggingface:/root/.cache/huggingface", "-e", "VLLM_ENGINE_READY_TIMEOUT_S=3600", "-e", "GLOO_SOCKET_IFNAME=$IFACE_NAME", "-e", "NCCL_SOCKET_IFNAME=$IFACE_NAME", "-e", "VLLM_USE_RUST_FRONTEND=1", "vllm/vllm-openai:qwen38", "Qwen/Qwen3.8-2.4T-A95B", "--tensor-parallel-size", "16", "--nnodes", "2", "--node-rank", "0", "--master-addr", "$HEAD_IP", "--enable-auto-tool-choice", "--tool-call-parser", "qwen3_coder", "--reasoning-parser", "qwen3" ], "worker_docker_argv": [ "docker", "run", "--gpus", "all", "--privileged", "--ipc=host", "-p", "8000:8000", "-v", "~/.cache/huggingface:/root/.cache/huggingface", "-e", "VLLM_ENGINE_READY_TIMEOUT_S=3600", "-e", "GLOO_SOCKET_IFNAME=$IFACE_NAME", "-e", "NCCL_SOCKET_IFNAME=$IFACE_NAME", "-e", "VLLM_USE_RUST_FRONTEND=1", "vllm/vllm-openai:qwen38", "Qwen/Qwen3.8-2.4T-A95B", "--tensor-parallel-size", "16", "--nnodes", "2", "--node-rank", "1", "--master-addr", "$HEAD_IP", "--headless", "--enable-auto-tool-choice", "--tool-call-parser", "qwen3_coder", "--reasoning-parser", "qwen3" ], "worker_docker_argvs": [ [ "docker", "run", "--gpus", "all", "--privileged", "--ipc=host", "-p", "8000:8000", "-v", "~/.cache/huggingface:/root/.cache/huggingface", "-e", "VLLM_ENGINE_READY_TIMEOUT_S=3600", "-e", "GLOO_SOCKET_IFNAME=$IFACE_NAME", "-e", "NCCL_SOCKET_IFNAME=$IFACE_NAME", "-e", "VLLM_USE_RUST_FRONTEND=1", "vllm/vllm-openai:qwen38", "Qwen/Qwen3.8-2.4T-A95B", "--tensor-parallel-size", "16", "--nnodes", "2", "--node-rank", "1", "--master-addr", "$HEAD_IP", "--headless", "--enable-auto-tool-choice", "--tool-call-parser", "qwen3_coder", "--reasoning-parser", "qwen3" ] ], "strategy_spec": { "name": "multi_node_tp", "deploy_type": "multi_node", "display_name": "Multi-Node Tensor Parallel", "short_label": "Multi-node TP", "orientation": "latency", "description": "Multi-node TP. Splits the model across GPUs spanning multiple nodes. TP is set to total GPU count across all nodes. Head node serves requests, worker nodes run with --headless. Requires InfiniBand interconnect.\n", "hardware_match": { "min_gpus": 2, "multi_node": true }, "parallel_flag": "--tensor-parallel-size", "env": { "GLOO_SOCKET_IFNAME": "$IFACE_NAME", "NCCL_SOCKET_IFNAME": "$IFACE_NAME" } }, "hardware_profile": { "brand": "NVIDIA", "generation": "blackwell", "compute_arch": "sm103", "display_name": "B300", "description": "NVIDIA B300 SXM 268 GB HBM3e · 8-GPU HGX B300 node (Blackwell Ultra)", "gpu_count": 8, "vram_gb": 2144, "multi_node": false }, "alternatives": { "single_node_tp": "/Qwen/Qwen3.8-2.4T-A95B/strategies/single_node_tp.json", "single_node_tep": "/Qwen/Qwen3.8-2.4T-A95B/strategies/single_node_tep.json", "single_node_dep": "/Qwen/Qwen3.8-2.4T-A95B/strategies/single_node_dep.json", "multi_node_tep": "/Qwen/Qwen3.8-2.4T-A95B/strategies/multi_node_tep.json", "multi_node_dep": "/Qwen/Qwen3.8-2.4T-A95B/strategies/multi_node_dep.json", "multi_node_tp_pp": "/Qwen/Qwen3.8-2.4T-A95B/strategies/multi_node_tp_pp.json", "multi_node_tp_dp": "/Qwen/Qwen3.8-2.4T-A95B/strategies/multi_node_tp_dp.json", "pd_cluster": "/Qwen/Qwen3.8-2.4T-A95B/strategies/pd_cluster.json", "kv_store_centralized_mooncake": "/Qwen/Qwen3.8-2.4T-A95B/strategies/kv_store_centralized_mooncake.json", "kv_store_distributed_mooncake": "/Qwen/Qwen3.8-2.4T-A95B/strategies/kv_store_distributed_mooncake.json" }, "by_hardware": { "b300": "/Qwen/Qwen3.8-2.4T-A95B/hw/b300.json", "h100": "/Qwen/Qwen3.8-2.4T-A95B/hw/h100.json", "h200": "/Qwen/Qwen3.8-2.4T-A95B/hw/h200.json", "b200": "/Qwen/Qwen3.8-2.4T-A95B/hw/b200.json", "gb200": "/Qwen/Qwen3.8-2.4T-A95B/hw/gb200.json", "gb300": "/Qwen/Qwen3.8-2.4T-A95B/hw/gb300.json", "mi300x": "/Qwen/Qwen3.8-2.4T-A95B/hw/mi300x.json", "mi325x": "/Qwen/Qwen3.8-2.4T-A95B/hw/mi325x.json", "mi350x": "/Qwen/Qwen3.8-2.4T-A95B/hw/mi350x.json", "mi355x": "/Qwen/Qwen3.8-2.4T-A95B/hw/mi355x.json" } }, "model": { "model_id": "Qwen/Qwen3.8-2.4T-A95B", "min_vllm_version": "nightly", "default_frontend": "rust", "nightly_required": true, "docker_image": { "nvidia": "vllm/vllm-openai:qwen38", "amd": "vllm/vllm-openai-rocm:qwen38" }, "architecture": "moe", "parameter_count": "2.4T", "active_parameters": "95B", "context_length": 262144, "base_args": [], "base_env": { "VLLM_ENGINE_READY_TIMEOUT_S": "3600" }, "install": { "pip": { "command": "uv venv\nsource .venv/bin/activate\nuv pip install -U vllm --pre \\\n --extra-index-url https://wheels.vllm.ai/nightly/cu130 \\\n --extra-index-url https://download.pytorch.org/whl/cu130 \\\n --index-strategy unsafe-best-match", "note": "vLLM nightly isn't released yet — nightly required. Nightly wheels are CUDA 13 (cu130) only; the host needs an r580+ NVIDIA driver." }, "docker": { "command": "docker pull vllm/vllm-openai:qwen38" }, "extras": [ { "note": "config.json declares transformers_version 5.4.0; older releases do not know this model_type.", "command": "uv pip install -U \"transformers>=5.4.0\"" } ] } }, "features": { "tool_calling": { "description": "Enable automatic tool choice. The chat template emits tool calls as XML, which is the qwen3_coder format — not JSON.", "args": [ "--enable-auto-tool-choice", "--tool-call-parser", "qwen3_coder" ] }, "reasoning": { "description": "Parse blocks separately from the final answer. The template opens every assistant turn with , so without this the reasoning lands in message.content.", "args": [ "--reasoning-parser", "qwen3" ] }, "spec_decoding": { "description": "Multi-token prediction using the draft head shipped inside the checkpoint — no separate speculator repo needed.", "args": [ "--speculative-config", "{\"method\":\"mtp\",\"num_speculative_tokens\":3}" ] } }, "opt_in_features": [ "spec_decoding" ], "variants": { "default": { "precision": "bf16", "vram_minimum_gb": 5871, "description": "Full-precision BF16 — 4.45 TiB of weights: 24 GPUs, i.e. 3x 8xB300 nodes or 6 GB300 NVL4 trays." }, "fp8": { "model_id": "Qwen/Qwen3.8-2.4T-A95B-FP8", "precision": "fp8", "vram_minimum_gb": 2996, "description": "Official block-scaled FP8 checkpoint — 2.27 TiB: 16 GPUs, i.e. 2x 8xB300 nodes or 4 GB300 NVL4 trays (3 trays hold it, but TP12 is invalid — 64 attention heads are not divisible by 12).", "json": "/Qwen/Qwen3.8-2.4T-A95B-FP8.json" }, "mxfp4": { "model_id": "Inferact/Qwen3.8-2.4T-A95B-MXFP4", "precision": "mxfp4", "label": "MXFP4 (AMD)", "vram_minimum_gb": 1917, "supported_hardware": [ "mi300x", "mi325x", "mi355x" ], "description": "MXFP4 build for AMD Instinct — 1.45 TiB: 8 GPUs, a single 8xMI355X or 8xMI325X node.", "json": "/Inferact/Qwen3.8-2.4T-A95B-MXFP4.json" }, "nvfp4": { "model_id": "Inferact/Qwen3.8-2.4T-A95B-NVFP4", "precision": "nvfp4", "label": "NVFP4 (W4A4)", "vram_minimum_gb": 1737, "supported_hardware": [ "b200", "b300", "gb200", "gb300" ], "extra_args": [ "--linear-backend", "flashinfer_cutedsl" ], "description": "NVFP4 build with 4-bit activations for NVIDIA Blackwell — 1.32 TiB, the smallest footprint: 8 GPUs, i.e. one 8xB300 node or 2 GB300 NVL4 trays.", "json": "/Inferact/Qwen3.8-2.4T-A95B-NVFP4.json" }, "nvfp4_reap25": { "model_id": "RedHatAI/Qwen3.8-2.4T-A95B-NVFP4-REAP-25", "precision": "nvfp4", "label": "NVFP4-REAP-25 (RedHatAI)", "vram_minimum_gb": 1373, "supported_hardware": [ "b200", "b300", "gb200", "gb300" ], "description": "NVFP4 quantization + 25% uniform expert pruning via REAP — 4.25x compression with ~99% accuracy recovery", "json": "/RedHatAI/Qwen3.8-2.4T-A95B-NVFP4-REAP-25.json" } }, "strategy_min_gpus": { "gb300": 8 }, "hardware_overrides": {}, "guide": "## Overview\n\nQwen3.8-2.4T-A95B is a 2.4-trillion-parameter Mixture-of-Experts model with roughly 95B\nparameters active per token — 512 routed experts with 10 active, plus one shared expert,\nover a 92-layer hybrid-attention backbone.\n\n**The layer mix is the interesting part.** Only 23 of the 92 layers run full attention\n(`full_attention_interval: 4`); the other 69 run linear attention with a constant\nrecurrent state. This is a text-only causal LM: the architecture is\n`Qwen3_5MoeForCausalLM`, `config.json` has no `vision_config`, and the weight index\ncontains zero vision tensors.\n\n## Prerequisites\n\n- **vLLM: nightly.** for the best performance\n- **transformers >= 5.4.0**, per the version recorded in `config.json`.\n\n```bash\nuv venv && source .venv/bin/activate\nuv pip install -U vllm --extra-index-url https://wheels.vllm.ai/nightly\nuv pip install -U \"transformers>=5.4.0\"\n```\n\n## Choosing a variant\n\nCounted in **GPUs, not nodes** — a \"node\" is 8 GPUs on B300/MI355X/H200 but only 4 on\nthe GB200/GB300 NVL4 trays, and that difference decides whether a variant fits.\n\n| Variant | Weights | Sized | B300 (268 GB) | MI355X (288 GB) | H200 (141 GB) | GB300 tray (4 GPU) |\n|---|---:|---:|---:|---:|---:|---:|\n| BF16 | 4.45 TiB | 5871 GB | 24 GPUs | 24 GPUs | 48 GPUs | 6 trays |\n| FP8 | 2.27 TiB | 2996 GB | 16 GPUs | 16 GPUs | 32 GPUs | **4 trays (TP16)** |\n| MXFP4 (AMD) | 1.45 TiB | 1917 GB | — | **8 GPUs** | 16 GPUs | — |\n| NVFP4 W4A4 (NVIDIA) | 1.32 TiB | 1737 GB | **8 GPUs** | — | 16 GPUs | **2 trays (TP8)** |\n| NVFP4-REAP-25 (RedHatAI) | 1.04 TiB | 1373 GB | **8 GPUs** | — | 16 GPUs | **2 trays (TP8)** |\n\n**TP must divide the 64 attention heads**, so only 1/2/4/8/16/32 are legal. This rounds\nsome layouts up: FP8 needs 2325 GiB, which is 3 GB300 trays by capacity, but TP12 is\ninvalid — so FP8 is a **16-GPU / 4-tray** deployment on GB300, not 12.\n\n## Launch commands\n\n### Low latency\n\nNVFP4 W4A4, TP8 (one 8xB300 node, or two GB300 trays):\n\n```bash\nvllm serve Inferact/Qwen3.8-2.4T-A95B-NVFP4 \\\n --tensor-parallel-size 8 \\\n --max-model-len 262144 \\\n --kv-cache-dtype fp8 \\\n --reasoning-parser qwen3 \\\n --enable-auto-tool-choice --tool-call-parser qwen3_coder\n```\n\nFP8, TP16 across two 8-GPU nodes. Same command on both, changing only `--node-rank`;\nrank > 0 additionally takes `--headless` (it must not start an API server).\n\n```bash\nvllm serve Qwen/Qwen3.8-2.4T-A95B-FP8 \\\n --tensor-parallel-size 16 \\\n --nnodes 2 --node-rank 0 --master-addr $HEAD_ADDR \\\n --max-model-len 262144 \\\n --kv-cache-dtype fp8 \\\n --reasoning-parser qwen3\n```\n\nAdd `--speculative-config '{\"method\":\"mtp\",\"num_speculative_tokens\":3}'` for the\nlatency numbers below — MTP-3 is worth roughly 2.3x on per-user output rate.\n\nMXFP4 on 8xMI355X — swap the model id and drop `--kv-cache-dtype fp8` unless you have\nconfirmed the ROCm path supports it in your build.\n\n### High throughput\n\nExpert parallelism, not plain TP, and a one-sided NVLink all2all. The recipe sets\n`--all2all-backend flashinfer_nvlink_one_sided` automatically on the EP strategies when\nthe hardware is Blackwell. The MoE kernel differs by precision, so set it explicitly:\n\n- **FP8, TP4·DP4 + EP16** (`multi_node_tp_dp`, 4 GB300 trays): add\n `--moe-backend flashinfer_trtllm --enable-expert-parallel`. See\n .\n- **NVFP4, DEP16** (`multi_node_dep`, 4 trays): add `--moe-backend flashinfer_cutedsl`.\n\nThese MoE backends are deliberately NOT pinned in the variant args: they are measured\nfor the EP layouts above, and the plain-TP path has not been measured with them. vLLM\nauto-selects `FLASHINFER_TRTLLM` for NVFP4 under TP, which is what the TP8 numbers below\nwere taken with.\n\n### Pipeline parallelism\n\n`multi_node_tp_pp` **works** on this architecture — verified 2026-08-13 on GB300 with\nTP4 x PP3 across 3 trays on the FP8 checkpoint: it loads, captures full CUDA graphs\n(51 piecewise + 35 full-decode), and generates correctly. Earlier revisions of this\nrecipe warned that hybrid linear-attention models had historically lagged on PP support;\nthat caveat is resolved. It remains a fallback rather than a default — PP exists for GPU\ncounts that pure TP cannot express (12 GPUs is 3 trays but TP12 is illegal), and TP or\nEP is faster whenever the GPU count is a power of two.\n\n## Performance\n\nTwo operating points, because they want different layouts. Latency wants TP and\nspeculative decoding; throughput wants expert parallelism.\n\n**Low latency** — output tokens/s per user:\n\n| Variant | Layout | no MTP | MTP-3 | with kernel opts |\n|---|---|---:|---:|---:|\n| FP8 | TP16 | 130 | **307** | 150 / 360 (target) |\n| NVFP4 | TP8 | 133 | **304** | — |\n\n**High throughput** — total (input+output) tokens/s per GPU:\n\n| Variant | Layout | Backends | total TPS/GPU |\n|---|---|---|---:|\n| FP8 | TP4·DP4 + EP16 | one-sided all2all + trtllm-gen MoE + activation opt | up to **3200** |\n| NVFP4 | DEP16 | one-sided all2all + cute-dsl MoE | up to **4300** |\n\nThe pending kernel work behind the \"target\" column is gdn decode fusion, split-k GEMM,\nand all-reduce fusion.\n\n**Verification run** (what earned `gb300: verified`) — NVFP4 W4A4, TP8 across 2 trays,\n8k in / 1k out, vLLM `0.1.dev19754+g3a0914114`, no EP, default MoE backend:\n\n| Concurrency | TPS/user | total TPS/GPU | TPOT |\n|---:|---:|---:|---:|\n| 1 | 101 | 108 | 9.9 ms |\n| 32 | 41.9 | 1338 | 23.9 ms |\n| 64 | 26.9 | 1715 | 37.2 ms |\n| 128 | 18.3 | 2170 | 54.7 ms |\n\nThat 2170 against 4300 is the cost of plain TP with the default all2all — it is the\nmeasurement the EP + one-sided-NVLink defaults above are based on.\n\nThree things that moved the numbers materially:\n\n- **Use MTP depth 3, not 1.** MTP-1 was measured at 64.8% acceptance and is not worth\n it: +3.4% at concurrency 1 and a *loss* above it (−9% at 128, −23% at 256), because\n the draft pass displaces real work once the batch is compute-bound. Depth 3 is where\n speculation pays (~2.3x per-user).\n- **`--load-format fastsafetensors --safetensors-load-strategy lazy`** cut weight load\n from 545 s to 306 s (−44%) on a 1.32 TiB checkpoint over shared storage.\n- **Size `--max-model-len` to the workload.** At 262144 the engine reserved KV for 25\n concurrent requests; at 9240 (8k in + 1k out) the same 70 GiB of KV serves 506.\n\n## Client usage\n\n`generation_config.json` ships `temperature: 1.0`, `top_p: 0.95`, `top_k: 20`.\n\n```python\nfrom openai import OpenAI\n\nclient = OpenAI(api_key=\"EMPTY\", base_url=\"http://localhost:8000/v1\", timeout=3600)\n\nresp = client.chat.completions.create(\n model=\"Qwen/Qwen3.8-2.4T-A95B\",\n messages=[{\"role\": \"user\", \"content\": \"Give me three primes above 100.\"}],\n temperature=1.0, top_p=0.95, max_tokens=2048,\n)\nprint(resp.choices[0].message.content)\n```\n\n## Processing Ultra-Long Texts\nThe model has 262k native context length and it can be extended to 1M with --max-model-len\nflag. The best value can be picked based on the use case and GPU ram usage tradeoff. \n\nFor example, to enable full 1M context length on this model:\n\n```bash\nVLLM_ALLOW_LONG_MAX_MODEL_LEN=1 vllm serve Qwen/Qwen3.8-2.4T-A95B \\\n --max-model-len 1010000 \\\n --hf-overrides '{\"max_position_embeddings\": 1010000}' \\\n ...\n```\n\n## Troubleshooting\n\n**CUDA graph / recurrent-state cache size error.** On hybrid models an\n`assert num_cache_lines >= batch` failure means the CUDA-graph capture size exceeds the\nstate cache. Reduce `--max-cudagraph-capture-size` (default 512), or fall back to\n`--compilation-config '{\"cudagraph_mode\":\"FULL_DECODE_ONLY\"}'` if full capture\nmisbehaves on this architecture. See .\n\n**Engine start times out.** Raise `VLLM_ENGINE_READY_TIMEOUT_S` (set to 3600 by this\nrecipe) and prefer a real `/v1/chat/completions` probe over `/health` for readiness —\nloading terabytes plus kernel JIT takes minutes.\n\nThe linear backend feature \"--linear-backend flashinfer_cutedsl\" is Docker only until\nFlashInfer version upgrade in the vllm build. The NVFP4 variant sets that flag, so on a\npip install either use the Docker image pinned above or drop the flag.\n\n## References\n\n- Model card: https://huggingface.co/Qwen/Qwen3.8-2.4T-A95B\n- FP8 checkpoint: https://huggingface.co/Qwen/Qwen3.8-2.4T-A95B-FP8\n- MXFP4 build (AMD): https://huggingface.co/Inferact/Qwen3.8-2.4T-A95B-MXFP4\n- NVFP4 W4A4 build (NVIDIA): https://huggingface.co/Inferact/Qwen3.8-2.4T-A95B-NVFP4\n- NVFP4-REAP-25 build (RedHatAI): https://huggingface.co/RedHatAI/Qwen3.8-2.4T-A95B-NVFP4-REAP-25\n- REAP: Router-weighted Expert Activation Pruning: https://arxiv.org/abs/2510.13999\n- vLLM documentation: https://docs.vllm.ai/\n", "source": { "source": "vllm", "repo": "vllm-project/vllm", "ref": "main", "sha": "564ac0de1b228568cafcbfbe8ccaccc82d6023eb", "path": "recipes/models/Qwen/Qwen3.8-2.4T-A95B.yaml", "edit_url": "https://github.com/vllm-project/vllm/edit/main/recipes/models/Qwen/Qwen3.8-2.4T-A95B.yaml", "overlays": [] } }