Qwen/Qwen3.8-Flash-Next

Overview

Qwen3.8-Flash-Next is a multimodal, ultra-sparse Mixture-of-Experts model. It has 125B parameters, including an additional 51B N-gram embedding table, while activating 6B parameters per token.

The architecture combines four main ideas:

• GDN + QSA: three of every four layers use Gated DeltaNet to compress history; the fourth uses Qwen Sparse Attention for precise long-range retrieval.
• Gated Residual: four residual branches dynamically control cross-layer reads and writes.
• N-gram Embedding: a 51B lookup memory adds capacity with little per-token compute and can be asynchronously offloaded to host memory. (note that offload currently only runs on Nvidia devices)
• MTP: the built-in Multi-Token Prediction module supports speculative decoding.

The checkpoint natively supports 262,144 tokens. Qwen reports that QSA reaches up to 10.2x prefill and 6.6x decode attention-kernel speedups at one million tokens. On GB300, TEP4 is the recommended full-tray configuration. On an 8x H200 node, use TEP8 for the official FP8 checkpoint; plain TP8 is incompatible with its 128-wide quantization blocks.

Prerequisites

• Runtime: use the nightly image, vllm/vllm-openai:nightly on NVIDIA or vllm/vllm-openai-rocm:nightly on AMD, or the nightly PyPI wheels. Stable ≥0.30.0 runs the model, but this recipe is tuned and verified on nightly.
• Hardware: NVIDIA CUDA and AMD ROCm instructions are provided below.
• FP8 memory: the checkpoint is 172.78 GiB; it fits on a single GB300 GPU (288 GB HBM3e).
• BF16 memory: the checkpoint is 335.28 GiB. The validated TP2 configuration used about 190 GiB per GB300 GPU.
• Host memory for N-gram Embedding offload: at least 51 GB plus runtime headroom.

The Install block above generates the nightly-wheel and Docker launch flows.

Serving

Recommended FP8 configuration on 4x GB300

Use the official Qwen/Qwen3.8-Flash-Next-FP8 checkpoint tested by this recipe.

vllm serve Qwen/Qwen3.8-Flash-Next-FP8 \
  --tensor-parallel-size 4 \
  --enable-expert-parallel \
  --gpu-memory-utilization 0.90 \
  --max-num-seqs 256 \
  --enable-prefix-caching \
  --kv-cache-dtype fp8 \
  --attention-config.indexer_kv_dtype fp8 \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}' \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --reasoning-parser qwen3

TEP4 is the recommended full-tray configuration — we recommend it over plain TP4 because expert parallelism is consistently faster for this model. TP4 and TEP4 are both validated full-tray configurations, including with MTP3.

To serve BF16, use the Qwen/Qwen3.8-Flash-Next checkpoint with the same arguments.

Single-GPU deployment on GB200/GB300

The model fits on a single GPU: the FP8 checkpoint on one GB300 (288 GB HBM3e), and the NVFP4 variant on GB200 as well. Single-GPU TP1 suits evaluation and development; multi-GPU TEP remains the recommended deployment for serving throughput.

Recommended FP8 configuration on 8x H200

Use TEP8 with the Triton MoE backend on Hopper:

vllm serve Qwen/Qwen3.8-Flash-Next-FP8 \
  --tensor-parallel-size 8 \
  --enable-expert-parallel \
  --moe-backend triton \
  --gpu-memory-utilization 0.85 \
  --max-num-seqs 256 \
  --enable-prefix-caching \
  --kv-cache-dtype fp8 \
  --attention-config.indexer_kv_dtype fp8 \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}' \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --reasoning-parser qwen3

Plain TP8 is incompatible with the FP8 checkpoint; use TEP8.

Recommended FP8 configuration on 8x H100

H100's 80GB/GPU is not enough headroom for the 51B N-gram/PLE embedding table — it relies on PLE CPU offload, which vLLM enables by default. Use TEP8 with the Triton MoE backend; as on other hardware, expert parallelism is faster than plain TP, and plain TP8 is incompatible with the FP8 checkpoint's 128-wide quantization blocks.

vllm serve Qwen/Qwen3.8-Flash-Next-FP8 \
  --tensor-parallel-size 8 \
  --enable-expert-parallel \
  --moe-backend triton \
  --gpu-memory-utilization 0.85 \
  --max-num-seqs 256 \
  --enable-prefix-caching \
  --kv-cache-dtype fp8 \
  --attention-config.indexer_kv_dtype fp8 \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}' \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --reasoning-parser qwen3

Benchmark MTP with SPEED-Bench (--dataset-name speed_bench), not random.

FP8 configuration on a 4x MI355X GPU

export VLLM_ROCM_USE_AITER=1
export VLLM_ROCM_USE_AITER_MOE=0

vllm serve Qwen/Qwen3.8-Flash-Next-FP8 \
--tensor-parallel-size 4 \
--max-model-len auto \
--gpu-memory-utilization 0.9 \
--speculative-config '{"method":"mtp","num_speculative_tokens":3}'

Full native context

The checkpoint advertises a native 262,144-token context, which vLLM uses when --max-model-len is omitted. Startup and bounded evaluation were validated with this limit configured, but a single 262K-token request was not tested.

MTP speculative decoding

Add the following option:

--speculative-config '{"method":"mtp","num_speculative_tokens":3}'

Offload the 51B N-gram embedding table

Builds containing N-gram Embedding offload can keep the N-gram lookup memory in host RAM and asynchronously prefetch the required rows. The initial implementation supports both ModelRunner V1 and V2. PLE CPU offload is required for DEP, and also required for TP/TEP on 80GB-class GPUs (H100) where the embedding table alone exceeds available headroom — optional only on GPUs with more VRAM per rank (H200 and above). vLLM enables it by default; set it explicitly with --engram-config '{"cpu_offload":true}'.

vllm serve Qwen/Qwen3.8-Flash-Next-FP8 \
  --tensor-parallel-size 4 \
  --enable-expert-parallel \
  --max-model-len 262144 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --reasoning-parser qwen3

Extend to one million tokens with YaRN

The checkpoint is native at 262K. For a one-million-token workload, turn on the Long Context feature above: it applies the model card's YaRN RoPE configuration under text_config (where this checkpoint carries rope_parameters) and raises --max-model-len to 1,000,000. Evaluate shorter-context quality before making it the default — static YaRN holds the scaling factor constant regardless of input length.

Verifying

from openai import OpenAI

client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1")
response = client.chat.completions.create(
    model="Qwen/Qwen3.8-Flash-Next-FP8",
    messages=[{
        "role": "user",
        "content": "Explain how Gated DeltaNet and Qwen Sparse Attention complement each other.",
    }],
    max_tokens=8196,
)
print(response.choices[0].message.content)

A correct response should explain that GDN maintains a compact recurrent summary, while QSA selectively retrieves important regions from the full token history.

Troubleshooting

• FP8 on eight GPUs: use TEP8; plain TP8 is incompatible with this checkpoint. TP4 is also supported if GPU memory capacity allows.
• Mamba-cache capacity error at startup: keep --max-num-seqs 256.
• Runtime OOM on large multimodal batches: keep --gpu-memory-utilization 0.90.
• Out of memory while loading: enable N-gram Embedding CPU offload, increase TP size, or reduce --max-model-len to reserve less KV cache. On 4x H100 80GB, plain TP4 OOMs at engine startup without PLE CPU offload — keep the default --engram-config '{"cpu_offload":true}'.
• XPU/TPU startup error: these platforms are not supported by the initial implementation.
• Pipeline-parallel startup error: N-gram Embedding does not initially support pipeline parallelism; use single-node TP or TEP instead.
• DEP startup failure: DEP requires PLE CPU offload (on by default); DEP without it is not supported.
• MTP memory pressure: reduce num_speculative_tokens below 3.
• PLE: a network layer that injects N-gram Embeddings into the main model.

KV cache offloading

Offloading is available upstream since vLLM v0.30.0 (vllm-project/vllm#57145). Validation used Qwen/Qwen3.8-Flash-Next-FP8 on 4×H200 (TP4), vLLM commit d1b4028d7e, the Python frontend, --enforce-eager, --max-model-len 16384, --max-num-seqs 16 and --max-num-batched-tokens 8192. Qwen validation used the FP8 variant with --limit-mm-per-prompt '{"image":0,"video":0}'. Validation used TieringOffloadingSpec: 8 GiB total CPU tier, blocks_per_chunk: 1, a persistent filesystem root_dir, eight read/write threads and PYTHONHASHSEED=0. Three CPU-tier reloads and three filesystem reloads after restarting the server passed. Reloaded responses matched the cold output, including the reasoning field. The five-shot GSM8K diagnostic scored 31/32; this is a small sample, not the full benchmark. These results cover this text-only configuration, not the builder's larger tier/chunk defaults or other variants.

References

• Qwen3.8-Flash-Next model card
• Qwen3.8-Flash-Next FP8 checkpoint
• Qwen3.8-Flash-Next: A New Architecture, Towards Ultimate Cost-Efficiency
• Qwen3-Next recipe