# Qwen/Qwen3.8-2.4T-A95B — vLLM serving recipe

> 2.4T-parameter hybrid-attention MoE (~95B active) with linear attention on 69 of 92 layers, 512 routed experts, a built-in MTP draft head, 262K native context window and extensible to 1M context

Configuration: hardware `b300` (8× B300) · strategy `multi_node_tp` (Multi-Node Tensor Parallel) · 2 nodes · variant `default` (BF16) · features `tool_calling`, `reasoning` · frontend `rust`

- Interactive page: https://recipes.vllm.ai/Qwen/Qwen3.8-2.4T-A95B
- This configuration as JSON: https://recipes.vllm.ai/Qwen/Qwen3.8-2.4T-A95B.json?hardware=b300

The commands below were rendered for exactly this configuration and take precedence over any example in the guide.

## Other configurations

Change the query string (the same parameters as the page URL) and fetch this URL again. Unknown or incompatible values fall back to the recipe default, so check the Configuration line.

- `hardware`: `h100`, `h200`, `b200`, `gb200`, `b300`, `gb300`, `mi300x`, `mi325x`, `mi350x`, `mi355x`
- `variant`: `default`, `fp8`, `mxfp4`, `nvfp4`, `nvfp4_reap25`
- `strategy`: `single_node_tp`, `single_node_tep`, `single_node_dep`, `multi_node_tp`, `multi_node_tep`, `multi_node_dep`, `multi_node_tp_pp`, `multi_node_tp_dp`, `pd_cluster` (multi-node strategies also need `nodes=<n>`)
- `features`: comma-separated list of `tool_calling`, `reasoning`, `spec_decoding` (default: `tool_calling`, `reasoning`)

## Install

Requires vLLM >= nightly (not released yet — nightly wheel).

vLLM nightly isn't released yet — nightly required. Nightly wheels are CUDA 13 (cu130) only; the host needs an r580+ NVIDIA driver.

```bash
uv venv
source .venv/bin/activate
uv pip install -U vllm --pre \
  --extra-index-url https://wheels.vllm.ai/nightly/cu130 \
  --extra-index-url https://download.pytorch.org/whl/cu130 \
  --index-strategy unsafe-best-match
```

Docker image: `vllm/vllm-openai:qwen38`

Extra dependencies:

- config.json declares transformers_version 5.4.0; older releases do not know this model_type.

```bash
uv pip install -U "transformers>=5.4.0"
```

## Serve

Run one command per node: the head (node rank 0) plus 1 worker. Each node's command differs — don't replay one on every node.

### Head (node 0)

```bash
export VLLM_ENGINE_READY_TIMEOUT_S=3600
export GLOO_SOCKET_IFNAME=$IFACE_NAME
export NCCL_SOCKET_IFNAME=$IFACE_NAME
export VLLM_USE_RUST_FRONTEND=1
vllm serve Qwen/Qwen3.8-2.4T-A95B \
  --tensor-parallel-size 16 \
  --nnodes 2 \
  --node-rank 0 \
  --master-addr $HEAD_IP \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --reasoning-parser qwen3
```

### Node 1

```bash
export VLLM_ENGINE_READY_TIMEOUT_S=3600
export GLOO_SOCKET_IFNAME=$IFACE_NAME
export NCCL_SOCKET_IFNAME=$IFACE_NAME
export VLLM_USE_RUST_FRONTEND=1
vllm serve Qwen/Qwen3.8-2.4T-A95B \
  --tensor-parallel-size 16 \
  --nnodes 2 \
  --node-rank 1 \
  --master-addr $HEAD_IP \
  --headless \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --reasoning-parser qwen3
```

Docker equivalent:

```bash
# Head
docker run --gpus all \
  --privileged --ipc=host -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -e VLLM_ENGINE_READY_TIMEOUT_S=3600 \
  -e GLOO_SOCKET_IFNAME=$IFACE_NAME \
  -e NCCL_SOCKET_IFNAME=$IFACE_NAME \
  -e VLLM_USE_RUST_FRONTEND=1 \
  vllm/vllm-openai:qwen38 Qwen/Qwen3.8-2.4T-A95B \
  --tensor-parallel-size 16 \
  --nnodes 2 \
  --node-rank 0 \
  --master-addr $HEAD_IP \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --reasoning-parser qwen3
```

## Guide

> Background reference. Commands in it are examples and may target another configuration — use the Serve section above.

### Overview

Qwen3.8-2.4T-A95B is a 2.4-trillion-parameter Mixture-of-Experts model with roughly 95B
parameters active per token — 512 routed experts with 10 active, plus one shared expert,
over a 92-layer hybrid-attention backbone.

**The layer mix is the interesting part.** Only 23 of the 92 layers run full attention
(`full_attention_interval: 4`); the other 69 run linear attention with a constant
recurrent state. This is a text-only causal LM: the architecture is
`Qwen3_5MoeForCausalLM`, `config.json` has no `vision_config`, and the weight index
contains zero vision tensors.

### Prerequisites

- **vLLM: nightly.** for the best performance
- **transformers >= 5.4.0**, per the version recorded in `config.json`.

```bash
uv venv && source .venv/bin/activate
uv pip install -U vllm --extra-index-url https://wheels.vllm.ai/nightly
uv pip install -U "transformers>=5.4.0"
```

### Choosing a variant

Counted in **GPUs, not nodes** — a "node" is 8 GPUs on B300/MI355X/H200 but only 4 on
the GB200/GB300 NVL4 trays, and that difference decides whether a variant fits.

| Variant | Weights | Sized | B300 (268 GB) | MI355X (288 GB) | H200 (141 GB) | GB300 tray (4 GPU) |
|---|---:|---:|---:|---:|---:|---:|
| BF16 | 4.45 TiB | 5871 GB | 24 GPUs | 24 GPUs | 48 GPUs | 6 trays |
| FP8 | 2.27 TiB | 2996 GB | 16 GPUs | 16 GPUs | 32 GPUs | **4 trays (TP16)** |
| MXFP4 (AMD) | 1.45 TiB | 1917 GB | — | **8 GPUs** | 16 GPUs | — |
| NVFP4 W4A4 (NVIDIA) | 1.32 TiB | 1737 GB | **8 GPUs** | — | 16 GPUs | **2 trays (TP8)** |
| NVFP4-REAP-25 (RedHatAI) | 1.04 TiB | 1373 GB | **8 GPUs** | — | 16 GPUs | **2 trays (TP8)** |

**TP must divide the 64 attention heads**, so only 1/2/4/8/16/32 are legal. This rounds
some layouts up: FP8 needs 2325 GiB, which is 3 GB300 trays by capacity, but TP12 is
invalid — so FP8 is a **16-GPU / 4-tray** deployment on GB300, not 12.

### Launch commands

#### Low latency

NVFP4 W4A4, TP8 (one 8xB300 node, or two GB300 trays):

```bash
vllm serve Inferact/Qwen3.8-2.4T-A95B-NVFP4 \
  --tensor-parallel-size 8 \
  --max-model-len 262144 \
  --kv-cache-dtype fp8 \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice --tool-call-parser qwen3_coder
```

FP8, TP16 across two 8-GPU nodes. Same command on both, changing only `--node-rank`;
rank > 0 additionally takes `--headless` (it must not start an API server).

```bash
vllm serve Qwen/Qwen3.8-2.4T-A95B-FP8 \
  --tensor-parallel-size 16 \
  --nnodes 2 --node-rank 0 --master-addr $HEAD_ADDR \
  --max-model-len 262144 \
  --kv-cache-dtype fp8 \
  --reasoning-parser qwen3
```

Add `--speculative-config '{"method":"mtp","num_speculative_tokens":3}'` for the
latency numbers below — MTP-3 is worth roughly 2.3x on per-user output rate.

MXFP4 on 8xMI355X — swap the model id and drop `--kv-cache-dtype fp8` unless you have
confirmed the ROCm path supports it in your build.

#### High throughput

Expert parallelism, not plain TP, and a one-sided NVLink all2all. The recipe sets
`--all2all-backend flashinfer_nvlink_one_sided` automatically on the EP strategies when
the hardware is Blackwell. The MoE kernel differs by precision, so set it explicitly:

- **FP8, TP4·DP4 + EP16** (`multi_node_tp_dp`, 4 GB300 trays): add
  `--moe-backend flashinfer_trtllm --enable-expert-parallel`. See
  <https://github.com/vllm-project/vllm/pull/51924>.
- **NVFP4, DEP16** (`multi_node_dep`, 4 trays): add `--moe-backend flashinfer_cutedsl`.

These MoE backends are deliberately NOT pinned in the variant args: they are measured
for the EP layouts above, and the plain-TP path has not been measured with them. vLLM
auto-selects `FLASHINFER_TRTLLM` for NVFP4 under TP, which is what the TP8 numbers below
were taken with.

#### Pipeline parallelism

`multi_node_tp_pp` **works** on this architecture — verified 2026-08-13 on GB300 with
TP4 x PP3 across 3 trays on the FP8 checkpoint: it loads, captures full CUDA graphs
(51 piecewise + 35 full-decode), and generates correctly. Earlier revisions of this
recipe warned that hybrid linear-attention models had historically lagged on PP support;
that caveat is resolved. It remains a fallback rather than a default — PP exists for GPU
counts that pure TP cannot express (12 GPUs is 3 trays but TP12 is illegal), and TP or
EP is faster whenever the GPU count is a power of two.

### Performance

Two operating points, because they want different layouts. Latency wants TP and
speculative decoding; throughput wants expert parallelism.

**Low latency** — output tokens/s per user:

| Variant | Layout | no MTP | MTP-3 | with kernel opts |
|---|---|---:|---:|---:|
| FP8 | TP16 | 130 | **307** | 150 / 360 (target) |
| NVFP4 | TP8 | 133 | **304** | — |

**High throughput** — total (input+output) tokens/s per GPU:

| Variant | Layout | Backends | total TPS/GPU |
|---|---|---|---:|
| FP8 | TP4·DP4 + EP16 | one-sided all2all + trtllm-gen MoE + activation opt | up to **3200** |
| NVFP4 | DEP16 | one-sided all2all + cute-dsl MoE | up to **4300** |

The pending kernel work behind the "target" column is gdn decode fusion, split-k GEMM,
and all-reduce fusion.

**Verification run** (what earned `gb300: verified`) — NVFP4 W4A4, TP8 across 2 trays,
8k in / 1k out, vLLM `0.1.dev19754+g3a0914114`, no EP, default MoE backend:

| Concurrency | TPS/user | total TPS/GPU | TPOT |
|---:|---:|---:|---:|
| 1 | 101 | 108 | 9.9 ms |
| 32 | 41.9 | 1338 | 23.9 ms |
| 64 | 26.9 | 1715 | 37.2 ms |
| 128 | 18.3 | 2170 | 54.7 ms |

That 2170 against 4300 is the cost of plain TP with the default all2all — it is the
measurement the EP + one-sided-NVLink defaults above are based on.

Three things that moved the numbers materially:

- **Use MTP depth 3, not 1.** MTP-1 was measured at 64.8% acceptance and is not worth
  it: +3.4% at concurrency 1 and a *loss* above it (−9% at 128, −23% at 256), because
  the draft pass displaces real work once the batch is compute-bound. Depth 3 is where
  speculation pays (~2.3x per-user).
- **`--load-format fastsafetensors --safetensors-load-strategy lazy`** cut weight load
  from 545 s to 306 s (−44%) on a 1.32 TiB checkpoint over shared storage.
- **Size `--max-model-len` to the workload.** At 262144 the engine reserved KV for 25
  concurrent requests; at 9240 (8k in + 1k out) the same 70 GiB of KV serves 506.

### Client usage

`generation_config.json` ships `temperature: 1.0`, `top_p: 0.95`, `top_k: 20`.

```python
from openai import OpenAI

client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1", timeout=3600)

resp = client.chat.completions.create(
    model="Qwen/Qwen3.8-2.4T-A95B",
    messages=[{"role": "user", "content": "Give me three primes above 100."}],
    temperature=1.0, top_p=0.95, max_tokens=2048,
)
print(resp.choices[0].message.content)
```

### Processing Ultra-Long Texts
The model has 262k native context length and it can be extended to 1M with --max-model-len
flag. The best value can be picked based on the use case and GPU ram usage tradeoff. 

For example, to enable full 1M context length on this model:

```bash
VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 vllm serve Qwen/Qwen3.8-2.4T-A95B \
  --max-model-len 1010000 \
  --hf-overrides '{"max_position_embeddings": 1010000}' \
  ...
```

### Troubleshooting

**CUDA graph / recurrent-state cache size error.** On hybrid models an
`assert num_cache_lines >= batch` failure means the CUDA-graph capture size exceeds the
state cache. Reduce `--max-cudagraph-capture-size` (default 512), or fall back to
`--compilation-config '{"cudagraph_mode":"FULL_DECODE_ONLY"}'` if full capture
misbehaves on this architecture. See <https://github.com/vllm-project/vllm/pull/34571>.

**Engine start times out.** Raise `VLLM_ENGINE_READY_TIMEOUT_S` (set to 3600 by this
recipe) and prefer a real `/v1/chat/completions` probe over `/health` for readiness —
loading terabytes plus kernel JIT takes minutes.

The linear backend feature "--linear-backend flashinfer_cutedsl" is Docker only until
FlashInfer version upgrade in the vllm build. The NVFP4 variant sets that flag, so on a
pip install either use the Docker image pinned above or drop the flag.

### References

- Model card: https://huggingface.co/Qwen/Qwen3.8-2.4T-A95B
- FP8 checkpoint: https://huggingface.co/Qwen/Qwen3.8-2.4T-A95B-FP8
- MXFP4 build (AMD): https://huggingface.co/Inferact/Qwen3.8-2.4T-A95B-MXFP4
- NVFP4 W4A4 build (NVIDIA): https://huggingface.co/Inferact/Qwen3.8-2.4T-A95B-NVFP4
- NVFP4-REAP-25 build (RedHatAI): https://huggingface.co/RedHatAI/Qwen3.8-2.4T-A95B-NVFP4-REAP-25
- REAP: Router-weighted Expert Activation Pruning: https://arxiv.org/abs/2510.13999
- vLLM documentation: https://docs.vllm.ai/