Qwen/Qwen3.8-2.4T-A95B

Overview

Qwen3.8-2.4T-A95B is a 2.4-trillion-parameter Mixture-of-Experts model with roughly 95B parameters active per token — 512 routed experts with 10 active, plus one shared expert, over a 92-layer hybrid-attention backbone.

The layer mix is the interesting part. Only 23 of the 92 layers run full attention (full_attention_interval: 4); the other 69 run linear attention with a constant recurrent state. This is a text-only causal LM: the architecture is Qwen3_5MoeForCausalLM, config.json has no vision_config, and the weight index contains zero vision tensors.

Prerequisites

• vLLM: nightly. for the best performance
• transformers >= 5.4.0, per the version recorded in config.json.

uv venv && source .venv/bin/activate
uv pip install -U vllm --extra-index-url https://wheels.vllm.ai/nightly
uv pip install -U "transformers>=5.4.0"

Choosing a variant

Variant	Weights	Sized	8xB300 (2144 GB)	8xMI355X (2304 GB)	8xH200 (1128 GB)
BF16	4.45 TiB	5871 GB	3 nodes	3 nodes	6 nodes
FP8	2.27 TiB	2996 GB	2 nodes	2 nodes	4 nodes
MXFP4 (AMD)	1.45 TiB	1917 GB	—	1 node	2 nodes
NVFP4 W4A4 (NVIDIA)	1.32 TiB	1737 GB	1 node	—	2 nodes

The two 4-bit builds are what make this model single-node: MXFP4 targets AMD Instinct, NVFP4 W4A4 targets NVIDIA Blackwell.

Launch commands

Single node, NVFP4 W4A4 on 8xB300:

vllm serve Inferact/Qwen3.8-2.4T-A95B-NVFP4-W4A4 \
  --tensor-parallel-size 8 \
  --max-model-len 262144 \
  --kv-cache-dtype fp8 \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice --tool-call-parser qwen3_coder

Single node, MXFP4 on 8xMI355X — swap the model id and drop --kv-cache-dtype fp8 unless you have confirmed the ROCm path supports it in your build.

Two 8xB300 nodes, FP8, pure tensor parallel. Same command on both nodes, changing only --node-rank:

vllm serve Qwen/Qwen3.8-2.4T-A95B-FP8 \
  --tensor-parallel-size 16 \
  --nnodes 2 --node-rank 0 --master-addr $HEAD_ADDR \
  --max-model-len 262144 \
  --kv-cache-dtype fp8 \
  --reasoning-parser qwen3

Pipeline parallelism is offered as multi_node_tp_pp, but hybrid linear-attention models have historically lagged on PP support — confirm it loads before planning a layout around it.

Client usage

generation_config.json ships temperature: 1.0, top_p: 0.95, top_k: 20. Use them; an earlier revision shipped temperature: 0.6, so don't rely on remembered defaults.

from openai import OpenAI

client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1", timeout=3600)

resp = client.chat.completions.create(
    model="Qwen/Qwen3.8-2.4T-A95B",
    messages=[{"role": "user", "content": "Give me three primes above 100."}],
    temperature=1.0, top_p=0.95, max_tokens=2048,
)
print(resp.choices[0].message.content)

Thinking mode

The template opens each assistant turn with <think>, so a short max_tokens can return an empty content — the budget went to reasoning. To turn thinking off:

--default-chat-template-kwargs '{"enable_thinking": false}'

which emits a pre-closed <think></think> block instead.

Multi-turn reasoning history

By default the template strips <think> blocks from older assistant turns, keeping them only after the last user query. Pass preserve_thinking: true in chat_template_kwargs to retain the full history — it changes what the model sees, so it changes behaviour.

Troubleshooting

CUDA graph / recurrent-state cache size error. On hybrid models an assert num_cache_lines >= batch failure means the CUDA-graph capture size exceeds the state cache. Reduce --max-cudagraph-capture-size (default 512), or fall back to --compilation-config '{"cudagraph_mode":"FULL_DECODE_ONLY"}' if full capture misbehaves on this architecture. See https://github.com/vllm-project/vllm/pull/34571.

Engine start times out. Raise VLLM_ENGINE_READY_TIMEOUT_S (set to 3600 by this recipe) and prefer a real /v1/chat/completions probe over /health for readiness — loading terabytes plus kernel JIT takes minutes.

The linear backend feature "--linear-backend flashinfer_cutedsl" is Docker only until FlashInfer version upgrade in the vllm build.

References

• Model card: https://huggingface.co/Qwen/Qwen3.8-2.4T-A95B
• FP8 checkpoint: https://huggingface.co/Qwen/Qwen3.8-2.4T-A95B-FP8
• MXFP4 build (AMD): https://huggingface.co/Inferact/Qwen3.8-2.4T-A95B-MXFP4
• NVFP4 W4A4 build (NVIDIA): https://huggingface.co/Inferact/Qwen3.8-2.4T-A95B-NVFP4
• vLLM documentation: https://docs.vllm.ai/