Qwen/Qwen3.8-2.4T-A95B

Overview

Qwen3.8-2.4T-A95B is a 2.4-trillion-parameter Mixture-of-Experts model with roughly 95B parameters active per token — 512 routed experts with 10 active, plus one shared expert, over a 92-layer hybrid-attention backbone.

The layer mix is the interesting part. Only 23 of the 92 layers run full attention (full_attention_interval: 4); the other 69 run linear attention with a constant recurrent state. This is a text-only causal LM: the architecture is Qwen3_5MoeForCausalLM, config.json has no vision_config, and the weight index contains zero vision tensors.

Prerequisites

• vLLM: nightly. for the best performance
• transformers >= 5.4.0, per the version recorded in config.json.

uv venv && source .venv/bin/activate
uv pip install -U vllm --extra-index-url https://wheels.vllm.ai/nightly
uv pip install -U "transformers>=5.4.0"

Choosing a variant

Counted in GPUs, not nodes — a "node" is 8 GPUs on B300/MI355X/H200 but only 4 on the GB200/GB300 NVL4 trays, and that difference decides whether a variant fits.

Variant	Weights	Sized	B300 (268 GB)	MI355X (288 GB)	H200 (141 GB)	GB300 tray (4 GPU)
BF16	4.45 TiB	5871 GB	24 GPUs	24 GPUs	48 GPUs	6 trays
FP8	2.27 TiB	2996 GB	16 GPUs	16 GPUs	32 GPUs	4 trays (TP16)
MXFP4 (AMD)	1.45 TiB	1917 GB	—	8 GPUs	16 GPUs	—
NVFP4 W4A4 (NVIDIA)	1.32 TiB	1737 GB	8 GPUs	—	16 GPUs	2 trays (TP8)
NVFP4-REAP-25 (RedHatAI)	1.04 TiB	1373 GB	8 GPUs	—	16 GPUs	2 trays (TP8)

TP must divide the 64 attention heads, so only 1/2/4/8/16/32 are legal. This rounds some layouts up: FP8 needs 2325 GiB, which is 3 GB300 trays by capacity, but TP12 is invalid — so FP8 is a 16-GPU / 4-tray deployment on GB300, not 12.

Launch commands

Low latency

NVFP4 W4A4, TP8 (one 8xB300 node, or two GB300 trays):

vllm serve Inferact/Qwen3.8-2.4T-A95B-NVFP4 \
  --tensor-parallel-size 8 \
  --max-model-len 262144 \
  --kv-cache-dtype fp8 \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice --tool-call-parser qwen3_coder

FP8, TP16 across two 8-GPU nodes. Same command on both, changing only --node-rank; rank > 0 additionally takes --headless (it must not start an API server).

vllm serve Qwen/Qwen3.8-2.4T-A95B-FP8 \
  --tensor-parallel-size 16 \
  --nnodes 2 --node-rank 0 --master-addr $HEAD_ADDR \
  --max-model-len 262144 \
  --kv-cache-dtype fp8 \
  --reasoning-parser qwen3

Add --speculative-config '{"method":"mtp","num_speculative_tokens":3}' for the latency numbers below — MTP-3 is worth roughly 2.3x on per-user output rate.

MXFP4 on 8xMI355X — swap the model id and drop --kv-cache-dtype fp8 unless you have confirmed the ROCm path supports it in your build.

High throughput

Expert parallelism, not plain TP, and a one-sided NVLink all2all. The recipe sets --all2all-backend flashinfer_nvlink_one_sided automatically on the EP strategies when the hardware is Blackwell. The MoE kernel differs by precision, so set it explicitly:

• FP8, TP4·DP4 + EP16 (multi_node_tp_dp, 4 GB300 trays): add --moe-backend flashinfer_trtllm --enable-expert-parallel. See https://github.com/vllm-project/vllm/pull/51924.
• NVFP4, DEP16 (multi_node_dep, 4 trays): add --moe-backend flashinfer_cutedsl.

These MoE backends are deliberately NOT pinned in the variant args: they are measured for the EP layouts above, and the plain-TP path has not been measured with them. vLLM auto-selects FLASHINFER_TRTLLM for NVFP4 under TP, which is what the TP8 numbers below were taken with.

Pipeline parallelism

multi_node_tp_pp works on this architecture — verified 2026-08-13 on GB300 with TP4 x PP3 across 3 trays on the FP8 checkpoint: it loads, captures full CUDA graphs (51 piecewise + 35 full-decode), and generates correctly. Earlier revisions of this recipe warned that hybrid linear-attention models had historically lagged on PP support; that caveat is resolved. It remains a fallback rather than a default — PP exists for GPU counts that pure TP cannot express (12 GPUs is 3 trays but TP12 is illegal), and TP or EP is faster whenever the GPU count is a power of two.

Performance

Two operating points, because they want different layouts. Latency wants TP and speculative decoding; throughput wants expert parallelism.

Low latency — output tokens/s per user:

Variant	Layout	no MTP	MTP-3	with kernel opts
FP8	TP16	130	307	150 / 360 (target)
NVFP4	TP8	133	304	—

High throughput — total (input+output) tokens/s per GPU:

Variant	Layout	Backends	total TPS/GPU
FP8	TP4·DP4 + EP16	one-sided all2all + trtllm-gen MoE + activation opt	up to 3200
NVFP4	DEP16	one-sided all2all + cute-dsl MoE	up to 4300

The pending kernel work behind the "target" column is gdn decode fusion, split-k GEMM, and all-reduce fusion.

Verification run (what earned gb300: verified) — NVFP4 W4A4, TP8 across 2 trays, 8k in / 1k out, vLLM 0.1.dev19754+g3a0914114, no EP, default MoE backend:

Concurrency	TPS/user	total TPS/GPU	TPOT
1	101	108	9.9 ms
32	41.9	1338	23.9 ms
64	26.9	1715	37.2 ms
128	18.3	2170	54.7 ms

That 2170 against 4300 is the cost of plain TP with the default all2all — it is the measurement the EP + one-sided-NVLink defaults above are based on.

Three things that moved the numbers materially:

• Use MTP depth 3, not 1. MTP-1 was measured at 64.8% acceptance and is not worth it: +3.4% at concurrency 1 and a loss above it (−9% at 128, −23% at 256), because the draft pass displaces real work once the batch is compute-bound. Depth 3 is where speculation pays (~2.3x per-user).
• --load-format fastsafetensors --safetensors-load-strategy lazy cut weight load from 545 s to 306 s (−44%) on a 1.32 TiB checkpoint over shared storage.
• Size --max-model-len to the workload. At 262144 the engine reserved KV for 25 concurrent requests; at 9240 (8k in + 1k out) the same 70 GiB of KV serves 506.

Client usage

generation_config.json ships temperature: 1.0, top_p: 0.95, top_k: 20.

from openai import OpenAI

client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1", timeout=3600)

resp = client.chat.completions.create(
    model="Qwen/Qwen3.8-2.4T-A95B",
    messages=[{"role": "user", "content": "Give me three primes above 100."}],
    temperature=1.0, top_p=0.95, max_tokens=2048,
)
print(resp.choices[0].message.content)

Processing Ultra-Long Texts

The model has 262k native context length and it can be extended to 1M with --max-model-len flag. The best value can be picked based on the use case and GPU ram usage tradeoff.

For example, to enable full 1M context length on this model:

VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 vllm serve Qwen/Qwen3.8-2.4T-A95B \
  --max-model-len 1010000 \
  --hf-overrides '{"max_position_embeddings": 1010000}' \
  ...

Troubleshooting

CUDA graph / recurrent-state cache size error. On hybrid models an assert num_cache_lines >= batch failure means the CUDA-graph capture size exceeds the state cache. Reduce --max-cudagraph-capture-size (default 512), or fall back to --compilation-config '{"cudagraph_mode":"FULL_DECODE_ONLY"}' if full capture misbehaves on this architecture. See https://github.com/vllm-project/vllm/pull/34571.

Engine start times out. Raise VLLM_ENGINE_READY_TIMEOUT_S (set to 3600 by this recipe) and prefer a real /v1/chat/completions probe over /health for readiness — loading terabytes plus kernel JIT takes minutes.

The linear backend feature "--linear-backend flashinfer_cutedsl" is Docker only until FlashInfer version upgrade in the vllm build. The NVFP4 variant sets that flag, so on a pip install either use the Docker image pinned above or drop the flag.

References

• Model card: https://huggingface.co/Qwen/Qwen3.8-2.4T-A95B
• FP8 checkpoint: https://huggingface.co/Qwen/Qwen3.8-2.4T-A95B-FP8
• MXFP4 build (AMD): https://huggingface.co/Inferact/Qwen3.8-2.4T-A95B-MXFP4
• NVFP4 W4A4 build (NVIDIA): https://huggingface.co/Inferact/Qwen3.8-2.4T-A95B-NVFP4
• NVFP4-REAP-25 build (RedHatAI): https://huggingface.co/RedHatAI/Qwen3.8-2.4T-A95B-NVFP4-REAP-25
• REAP: Router-weighted Expert Activation Pruning: https://arxiv.org/abs/2510.13999
• vLLM documentation: https://docs.vllm.ai/