Qwen/Qwen3.8-Flash-Next

Overview

Qwen3.8-Flash-Next is a multimodal, ultra-sparse Mixture-of-Experts model. It has 125B parameters, including an additional 51B N-gram embedding table, while activating 6B parameters per token.

The architecture combines four main ideas:

• GDN + QSA: three of every four layers use Gated DeltaNet to compress history; the fourth uses Qwen Sparse Attention for precise long-range retrieval.
• Gated Residual: four residual branches dynamically control cross-layer reads and writes.
• N-gram Embedding: a 51B lookup memory adds capacity with little per-token compute and can be asynchronously offloaded to host memory. (note that offload currently only runs on Nvidia devices)
• MTP: the built-in Multi-Token Prediction module supports speculative decoding.

The checkpoint natively supports 262,144 tokens. Qwen reports that QSA reaches up to 10.2x prefill and 6.6x decode attention-kernel speedups at one million tokens. On GB300, TP2 is the minimum validated FP8 deployment and TP4 is the recommended full-tray configuration. On an 8x H200 node, use TEP8 for the official FP8 checkpoint; plain TP8 is incompatible with its 128-wide quantization blocks.

Prerequisites

• Runtime: use vllm/vllm-openai:qwen38-flash-next. PyPI installation is not supported for this recipe.
• Hardware: NVIDIA CUDA and AMD ROCm instructions are provided below.
• FP8 memory: the checkpoint is 172.78 GiB. TP2 is the validated minimum on GB300; TP4 is the recommended full-tray configuration.
• BF16 memory: the checkpoint is 335.28 GiB. The validated TP2 configuration used about 190 GiB per GB300 GPU.
• Host memory for N-gram Embedding offload: at least 51 GB plus runtime headroom.

The Install block above generates the pull and Docker launch flow for the dedicated image.

Serving

Recommended FP8 configuration on 4x GB300

Use the official Qwen/Qwen3.8-Flash-Next-FP8 checkpoint tested by this recipe.

vllm serve Qwen/Qwen3.8-Flash-Next-FP8 \
  --tensor-parallel-size 4 \
  --gpu-memory-utilization 0.90 \
  --max-num-seqs 256 \
  --enable-prefix-caching \
  --no-enable-flashinfer-autotune \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --reasoning-parser qwen3

TP2 is the minimum FP8 deployment on GB300. TP4 and TEP4 are validated full-tray configurations, including with MTP3. Enable expert parallelism with --enable-expert-parallel to improve throughput.

To serve BF16, use the Qwen/Qwen3.8-Flash-Next checkpoint with the same arguments.

Recommended FP8 configuration on 8x H200

Use TEP8 with the Triton MoE backend on Hopper:

vllm serve Qwen/Qwen3.8-Flash-Next-FP8 \
  --tensor-parallel-size 8 \
  --enable-expert-parallel \
  --moe-backend triton \
  --gpu-memory-utilization 0.85 \
  --max-num-seqs 256 \
  --enable-prefix-caching \
  --no-enable-flashinfer-autotune \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --reasoning-parser qwen3

Plain TP8 is incompatible with the FP8 checkpoint; use TEP8.

FP8 configuration on a 4x MI355X GPU

export VLLM_ROCM_USE_AITER=1
export VLLM_ROCM_USE_AITER_MOE=0

vllm serve Qwen/Qwen3.8-Flash-Next-FP8 \
--tensor-parallel-size 4 \
--max-model-len auto \
--gpu-memory-utilization 0.9

Full native context

The checkpoint advertises a native 262,144-token context, which vLLM uses when --max-model-len is omitted. Startup and bounded evaluation were validated with this limit configured, but a single 262K-token request was not tested.

MTP speculative decoding

Add the following option:

--speculative-config '{"method":"mtp","num_speculative_tokens":3}'

Offload the 51B N-gram embedding table

Builds containing N-gram Embedding offload can keep the N-gram lookup memory in host RAM and asynchronously prefetch the required rows. The initial implementation supports both ModelRunner V1 and V2. PLE CPU offload is optional for TP and TEP, but required for DEP. The DEP strategy generated by this recipe enables it automatically.

VLLM_PLE_CPU_OFFLOAD=1 \
vllm serve Qwen/Qwen3.8-Flash-Next-FP8 \
  --tensor-parallel-size 4 \
  --max-model-len 262144 \
  --no-enable-flashinfer-autotune \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --reasoning-parser qwen3

Extend to one million tokens with YaRN

The checkpoint is native at 262K. For a one-million-token workload, enable static YaRN explicitly; evaluate shorter-context quality before using this as the default.

VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 \
vllm serve Qwen/Qwen3.8-Flash-Next-FP8 \
  --tensor-parallel-size 4 \
  --rope-scaling '{"rope_type":"yarn","factor":4.0,"original_max_position_embeddings":262144}' \
  --max-model-len 1000000 \
  --no-enable-flashinfer-autotune \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --reasoning-parser qwen3

Verifying

from openai import OpenAI

client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1")
response = client.chat.completions.create(
    model="Qwen/Qwen3.8-Flash-Next-FP8",
    messages=[{
        "role": "user",
        "content": "Explain how Gated DeltaNet and Qwen Sparse Attention complement each other.",
    }],
    max_tokens=8196,
)
print(response.choices[0].message.content)

A correct response should explain that GDN maintains a compact recurrent summary, while QSA selectively retrieves important regions from the full token history.

Troubleshooting

• FP8 on eight GPUs: use TEP8; plain TP8 is incompatible with this checkpoint. TP4 is also supported if GPU memory capacity allows.
• TP1 compilation OOM on GB300: use TP2 or TP4.
• Mamba-cache capacity error at startup: keep --max-num-seqs 256.
• Runtime OOM on large multimodal batches: keep --gpu-memory-utilization 0.90.
• Out of memory while loading: enable N-gram Embedding CPU offload, increase TP size, or reduce --max-model-len to reserve less KV cache.
• XPU/TPU startup error: these platforms are not supported by the initial implementation.
• Pipeline-parallel startup error: N-gram Embedding does not initially support pipeline parallelism; use single-node TP or TEP instead.
• DEP startup failure: DEP requires VLLM_PLE_CPU_OFFLOAD=1; DEP without PLE offload is not supported. The generated DEP command sets this automatically.
• MTP memory pressure: reduce num_speculative_tokens below 3.
• PLE: a network layer that injects N-gram Embeddings into the main model.

References

• Qwen3.8-Flash-Next model card
• Qwen3.8-Flash-Next FP8 checkpoint
• Qwen3.8-Flash-Next: A New Architecture, Towards Ultimate Cost-Efficiency
• Qwen3-Next recipe