# Qwen/Qwen3-Next-80B-A3B-Instruct — vLLM serving recipe

> Advanced Qwen3-Next MoE model (80B total / 3B active) with hybrid attention, highly sparse experts, and multi-token prediction.

Configuration: hardware `h200` (8× H200) · strategy `single_node_tp` (Tensor Parallel) · variant `default` (BF16) · features `tool_calling`

- Interactive page: https://recipes.vllm.ai/Qwen/Qwen3-Next-80B-A3B-Instruct
- This configuration as JSON: https://recipes.vllm.ai/Qwen/Qwen3-Next-80B-A3B-Instruct.json?hardware=h200

The commands below were rendered for exactly this configuration and take precedence over any example in the guide.

## Other configurations

Change the query string (the same parameters as the page URL) and fetch this URL again. Unknown or incompatible values fall back to the recipe default, so check the Configuration line.

- `hardware`: `h100`, `h200`, `b200`, `gb200`, `b300`, `gb300`, `mi300x`, `mi325x`, `mi350x`, `mi355x`
- `variant`: `default`, `fp8`, `nvfp4`, `redhat_fp8`, `redhat_fp8_plain`, `redhat_fp8_block`, `redhat_nvfp4`
- `strategy`: `single_node_tp`, `single_node_tep`, `single_node_dep`, `multi_node_tp`, `multi_node_tp_pp`, `multi_node_dep`, `multi_node_tep` (multi-node strategies also need `nodes=<n>`)
- `features`: comma-separated list of `tool_calling`, `spec_decoding`, `long_context` (default: `tool_calling`)

## Install

Requires vLLM >= 0.17.0.

```bash
uv venv
source .venv/bin/activate
uv pip install -U vllm --torch-backend auto
```

Docker image: `vllm/vllm-openai:latest`

## Serve

```bash
vllm serve Qwen/Qwen3-Next-80B-A3B-Instruct \
  --tensor-parallel-size 8 \
  --enable-auto-tool-choice \
  --tool-call-parser hermes
```

Docker equivalent:

```bash
docker run --gpus all \
  --privileged --ipc=host -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai:latest Qwen/Qwen3-Next-80B-A3B-Instruct \
  --tensor-parallel-size 8 \
  --enable-auto-tool-choice \
  --tool-call-parser hermes
```

## Guide

> Background reference. Commands in it are examples and may target another configuration — use the Serve section above.

### Overview

[Qwen3-Next](https://qwen.ai/blog?id=4074cca80393150c248e508aa62983f9cb7d27cd&from=research.latest-advancements-list) is an advanced LLM from the Qwen team featuring:
- A hybrid attention mechanism
- A highly sparse Mixture-of-Experts (MoE) structure
- Training-stability-friendly optimizations
- A multi-token prediction mechanism for faster inference

### Prerequisites

```bash
uv venv
source .venv/bin/activate
uv pip install -U vllm --torch-backend auto
```

### Deployment Configurations

Launch on 4x H200/H20 or 4x A100/A800 GPUs.

#### Basic Multi-GPU (BF16)
```bash
vllm serve Qwen/Qwen3-Next-80B-A3B-Instruct \
  --tensor-parallel-size 4 \
  --served-model-name qwen3-next \
  --enable-prefix-caching
```

If you hit `torch.AcceleratorError: CUDA error: an illegal memory access was encountered`, add `--compilation_config.cudagraph_mode=PIECEWISE`.

#### FP8 (SM90/SM100)
```bash
vllm serve Qwen/Qwen3-Next-80B-A3B-Instruct-FP8 \
  --tensor-parallel-size 4 \
  --enable-prefix-caching
```

On SM100, accelerate with the FlashInfer TRTLLM MoE kernel:
```bash
VLLM_USE_DEEP_GEMM=0 \
vllm serve Qwen/Qwen3-Next-80B-A3B-Instruct-FP8 \
  --tensor-parallel-size 4 \
  --moe-backend flashinfer_trtllm \
  --attention-backend FLASH_ATTN
```

#### MTP (Multi-Token Prediction)
```bash
vllm serve Qwen/Qwen3-Next-80B-A3B-Instruct \
  --tokenizer-mode auto --gpu-memory-utilization 0.8 \
  --speculative-config '{"method": "qwen3_next_mtp", "num_speculative_tokens": 2}' \
  --tensor-parallel-size 4 --no-enable-chunked-prefill
```

#### Tool / Function Calling
```bash
vllm serve ... --tool-call-parser hermes --enable-auto-tool-choice
```

#### AMD (MI300X/MI325X/MI355X)
```bash
uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/0.14.1/rocm700
```
```bash
SAFETENSORS_FAST_GPU=1 \
VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 \
vllm serve Qwen/Qwen3-Next-80B-A3B-Instruct \
  --tensor-parallel-size 4 \
  --max-model-len 32768 \
  --no-enable-prefix-caching \
  --trust-remote-code
```

### Client Usage

Benchmark:
```bash
vllm bench serve \
  --backend vllm \
  --model Qwen/Qwen3-Next-80B-A3B-Instruct \
  --served-model-name qwen3-next \
  --endpoint /v1/completions \
  --dataset-name random \
  --random-input 2048 \
  --random-output 1024 \
  --max-concurrency 10 \
  --num-prompt 100
```

### Troubleshooting

- **Sub-optimal MoE performance warning**: Tune the MoE Triton kernel with [benchmark_moe](https://github.com/vllm-project/vllm/blob/main/benchmarks/kernels/benchmark_moe.py), then set `VLLM_TUNED_CONFIG_FOLDER` to the directory containing the generated config.
- **IMA error in DP mode**: add `--compilation_config.cudagraph_mode=PIECEWISE`.
- For more parallel topologies, see the [Data Parallel Deployment docs](https://docs.vllm.ai/en/latest/serving/data_parallel_deployment.html).

### References

- [Model card](https://huggingface.co/Qwen/Qwen3-Next-80B-A3B-Instruct)
- [FP8 checkpoint](https://huggingface.co/Qwen/Qwen3-Next-80B-A3B-Instruct-FP8)
- [Qwen3-Next blog](https://qwen.ai/blog?id=4074cca80393150c248e508aa62983f9cb7d27cd)