{ "hf_id": "Qwen/Qwen3.8-Flash-Next", "meta": { "title": "Qwen3.8-Flash-Next", "slug": "qwen3.8-flash-next", "provider": "Qwen", "description": "Qwen4 architecture preview with a 125B-parameter main model, supplemented by an additional 51B N-gram embeddings, with 6B parameters activated per token.", "date_added": "2026-08-26", "date_updated": "2026-10-27", "tasks": [ "multimodal", "text" ], "performance_headline": "Qwen4 architecture preview with 6B active parameters and efficient 262K context", "related_recipes": [ "Qwen/Qwen3-Next-80B-A3B-Instruct" ], "default_variant": { "mi355x": "mxfp4" }, "hardware": { "h200": "verified", "h100": "verified", "gb200": "verified", "gb300": "verified", "mi355x": "verified", "rtx_pro_6000_4x": "verified", "dgx_station_gb300": "verified" } }, "recommended_command": { "hardware": "b200", "strategy": "single_node_tep", "variant": "default", "node_count": 1, "deploy_type": "single_node", "env": { "VLLM_USE_RUST_FRONTEND": "1" }, "docker_image": "vllm/vllm-openai:nightly", "command": "vllm serve Inferact/Qwen3.8-Flash-Next-NVFP4 \\\n --max-num-seqs 256 \\\n --gpu-memory-utilization 0.90 \\\n --enable-prefix-caching \\\n --enable-expert-parallel \\\n --tensor-parallel-size 8 \\\n --kv-cache-dtype fp8 \\\n --attention-config.indexer_kv_dtype fp8 \\\n --enable-auto-tool-choice \\\n --tool-call-parser qwen3_coder \\\n --reasoning-parser qwen3 \\\n --speculative-config '{\"method\":\"mtp\",\"num_speculative_tokens\":3}'", "argv": [ "vllm", "serve", "Inferact/Qwen3.8-Flash-Next-NVFP4", "--max-num-seqs", "256", "--gpu-memory-utilization", "0.90", "--enable-prefix-caching", "--enable-expert-parallel", "--tensor-parallel-size", "8", "--kv-cache-dtype", "fp8", "--attention-config.indexer_kv_dtype", "fp8", "--enable-auto-tool-choice", "--tool-call-parser", "qwen3_coder", "--reasoning-parser", "qwen3", "--speculative-config", "{\"method\":\"mtp\",\"num_speculative_tokens\":3}" ], "docker_command": "docker run --gpus all \\\n --privileged --ipc=host -p 8000:8000 \\\n -v ~/.cache/huggingface:/root/.cache/huggingface \\\n -e VLLM_USE_RUST_FRONTEND=1 \\\n vllm/vllm-openai:nightly Inferact/Qwen3.8-Flash-Next-NVFP4 \\\n --max-num-seqs 256 \\\n --gpu-memory-utilization 0.90 \\\n --enable-prefix-caching \\\n --enable-expert-parallel \\\n --tensor-parallel-size 8 \\\n --kv-cache-dtype fp8 \\\n --attention-config.indexer_kv_dtype fp8 \\\n --enable-auto-tool-choice \\\n --tool-call-parser qwen3_coder \\\n --reasoning-parser qwen3 \\\n --speculative-config '{\"method\":\"mtp\",\"num_speculative_tokens\":3}'", "docker_argv": [ "docker", "run", "--gpus", "all", "--privileged", "--ipc=host", "-p", "8000:8000", "-v", "~/.cache/huggingface:/root/.cache/huggingface", "-e", "VLLM_USE_RUST_FRONTEND=1", "vllm/vllm-openai:nightly", "Inferact/Qwen3.8-Flash-Next-NVFP4", "--max-num-seqs", "256", "--gpu-memory-utilization", "0.90", "--enable-prefix-caching", "--enable-expert-parallel", "--tensor-parallel-size", "8", "--kv-cache-dtype", "fp8", "--attention-config.indexer_kv_dtype", "fp8", "--enable-auto-tool-choice", "--tool-call-parser", "qwen3_coder", "--reasoning-parser", "qwen3", "--speculative-config", "{\"method\":\"mtp\",\"num_speculative_tokens\":3}" ], "strategy_spec": { "name": "single_node_tep", "deploy_type": "single_node", "display_name": "Tensor + Expert Parallel", "short_label": "TEP", "orientation": "balanced", "description": "Single-node TEP. TP splits dense layers and EP splits expert layers across local GPUs. TP must be set to GPU count to avoid OOM from replicated dense layers. For MoE models only.\n", "hardware_match": { "min_gpus": 2, "max_gpus": 8, "multi_node": false, "architecture": "moe" }, "vllm_args": [ "--enable-expert-parallel" ], "parallel_flag": "--tensor-parallel-size" }, "hardware_profile": { "brand": "NVIDIA", "generation": "blackwell", "compute_arch": "sm100", "display_name": "B200", "description": "NVIDIA B200 SXM 180 GB HBM3e · 8-GPU HGX B200 node", "gpu_count": 8, "vram_gb": 1440, "multi_node": false }, "alternatives": { "single_node_tp": "/Qwen/Qwen3.8-Flash-Next/strategies/single_node_tp.json", "single_node_dep": "/Qwen/Qwen3.8-Flash-Next/strategies/single_node_dep.json", "kv_store_centralized_mooncake": "/Qwen/Qwen3.8-Flash-Next/strategies/kv_store_centralized_mooncake.json", "kv_store_distributed_mooncake": "/Qwen/Qwen3.8-Flash-Next/strategies/kv_store_distributed_mooncake.json" }, "by_hardware": { "b200": "/Qwen/Qwen3.8-Flash-Next/hw/b200.json", "gb200": "/Qwen/Qwen3.8-Flash-Next/hw/gb200.json", "b300": "/Qwen/Qwen3.8-Flash-Next/hw/b300.json", "gb300": "/Qwen/Qwen3.8-Flash-Next/hw/gb300.json", "dgx_station_gb300": "/Qwen/Qwen3.8-Flash-Next/hw/dgx_station_gb300.json", "rtx_pro_6000_4x": "/Qwen/Qwen3.8-Flash-Next/hw/rtx_pro_6000_4x.json" } }, "model": { "model_id": "Qwen/Qwen3.8-Flash-Next", "min_vllm_version": "0.30.0", "default_frontend": "rust", "docker_image": { "nvidia": "vllm/vllm-openai:nightly", "amd": "vllm/vllm-openai-rocm:nightly" }, "nightly_required": true, "architecture": "moe", "parameter_count": "176B", "active_parameters": "6B", "context_length": 262144, "base_args": [ "--max-num-seqs", "256", "--gpu-memory-utilization", "0.90", "--enable-prefix-caching" ], "base_env": {}, "install": { "pip": { "command": "uv venv\nsource .venv/bin/activate\nuv pip install -U vllm --pre \\\n --extra-index-url https://wheels.vllm.ai/nightly/cu130 \\\n --extra-index-url https://download.pytorch.org/whl/cu130 \\\n --index-strategy unsafe-best-match", "note": "Nightly wheels are recommended and verified by this recipe; stable vLLM ≥0.30.0 also runs the model." }, "docker": { "command": "docker pull vllm/vllm-openai:nightly", "note": "Use the vLLM nightly image." } } }, "features": { "tool_calling": { "description": "Enable automatic tool choice with the Qwen3 Coder parser", "args": [ "--enable-auto-tool-choice", "--tool-call-parser", "qwen3_coder" ] }, "reasoning": { "description": "Extract thinking output with the Qwen3 reasoning parser", "args": [ "--reasoning-parser", "qwen3" ] }, "spec_decoding": { "description": "Use the checkpoint's Multi-Token Prediction module for faster decoding", "args": [ "--speculative-config", "{\"method\":\"mtp\",\"num_speculative_tokens\":3}" ] }, "text_only": { "description": "Skip the vision encoder for text-only workloads and leave more memory for KV cache", "args": [ "--language-model-only" ] }, "long_context": { "description": "YaRN RoPE scaling past the native 262,144-token window to 1,000,000 tokens. Static YaRN holds the scaling factor constant regardless of input length, so leave this off unless you actually serve long prompts; for ~524K, halve the factor to 2.0.", "args": [ "--hf-overrides", "{\"text_config\": {\"rope_parameters\": {\"mrope_interleaved\": true, \"mrope_section\": [11, 11, 10], \"rope_type\": \"yarn\", \"rope_theta\": 10000000, \"partial_rotary_factor\": 0.25, \"factor\": 4.0, \"original_max_position_embeddings\": 262144}}}", "--max-model-len", "1000000" ], "env": { "VLLM_ALLOW_LONG_MAX_MODEL_LEN": "1" } } }, "opt_in_features": [ "text_only", "long_context" ], "hardware_opt_in_features": { "rtx_pro_6000_4x": [ "spec_decoding" ], "mi355x": [ "spec_decoding" ] }, "variants": { "default": { "model_id": "Inferact/Qwen3.8-Flash-Next-NVFP4", "precision": "nvfp4", "vram_minimum_gb": 130, "description": "Inferact quantized NVFP4 weights with per-layer embeddings in nvfp4 too", "hardware_overrides": { "dgx_station_gb300": { "extra_args": [ "--max-num-seqs", "32", "--gpu-memory-utilization", "0.95", "--max-num-batched-tokens", "8192", "--enable-prefix-caching", "--distributed-executor-backend", "mp", "-cc.mode", "none", "-cc.cudagraph_mode", "full_decode_only" ] }, "rtx_pro_6000_4x": { "extra_args": [ "--max-num-seqs", "16", "--gpu-memory-utilization", "0.95", "--max-num-batched-tokens", "8192", "--enable-prefix-caching", "--moe-backend", "marlin" ] } }, "json": "/Inferact/Qwen3.8-Flash-Next-NVFP4.json" }, "fp8": { "model_id": "Qwen/Qwen3.8-Flash-Next-FP8", "precision": "fp8", "vram_minimum_gb": 250, "description": "Official dynamic FP8 checkpoint with blockwise weight quantization", "hardware_overrides": { "dgx_station_gb300": { "extra_args": [ "--max-num-seqs", "32", "--gpu-memory-utilization", "0.95", "--max-num-batched-tokens", "8192", "--enable-prefix-caching", "--distributed-executor-backend", "mp", "-cc.mode", "none", "-cc.cudagraph_mode", "full_decode_only" ] }, "rtx_pro_6000_4x": { "extra_args": [ "--max-num-seqs", "16", "--gpu-memory-utilization", "0.95", "--max-num-batched-tokens", "8192", "--enable-prefix-caching" ] } }, "json": "/Qwen/Qwen3.8-Flash-Next-FP8.json" }, "bf16": { "model_id": "Qwen/Qwen3.8-Flash-Next", "precision": "bf16", "vram_minimum_gb": 423, "description": "Official BF16 checkpoint" }, "mxfp4": { "model_id": "amd/Qwen3.8-Flash-Next-Quark-MXFP4", "precision": "mxfp4", "label": "MXFP4 (AMD)", "vram_minimum_gb": 219, "description": "AMD Quark MXFP4 checkpoint with MXFP4 routed experts, for MI355X", "supported_hardware": [ "mi355x" ], "tp": { "mi355x": 2 }, "hardware_overrides": { "mi355x": { "extra_args": [ "--gdn-prefill-backend", "aiter_flydsl" ], "extra_env": { "VLLM_ROCM_USE_AITER": "1", "VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS": "1", "AITER_USE_FLYDSL_MOE_SORTING": "1" } } }, "json": "/amd/Qwen3.8-Flash-Next-Quark-MXFP4.json" } }, "kv_offload_support": { "offloading_cpu": "verified", "offloading_fs": "verified" }, "hardware_overrides": { "hopper": { "extra_args": [ "--moe-backend", "triton", "--gpu-memory-utilization", "0.85", "--kv-cache-dtype", "fp8", "--attention-config.indexer_kv_dtype", "fp8" ], "extra_env": {} }, "blackwell": { "extra_args": [ "--kv-cache-dtype", "fp8", "--attention-config.indexer_kv_dtype", "fp8" ] } }, "notes": [ { "title": "MTP is off by default on MI355X", "when": { "hardware": [ "mi355x" ] }, "body": "Speculative decoding with MTP is opt-in on MI355X. The MXFP4 checkpoint\ncan't load its MTP module until\n[its config.json fix](https://huggingface.co/amd/Qwen3.8-Flash-Next-Quark-MXFP4/discussions/1)\nis merged. FP8 and BF16 can still turn MTP on with the speculative\ndecoding toggle.\n" } ], "troubleshooting": [ { "title": "Timed out waiting for engine core processes to start", "when": { "variants": [ "mxfp4" ] }, "body": "On nodes with slow weight loading, a cold TP2 start of the MXFP4\ncheckpoint can take longer than the 600 s default ready timeout. On the\nvalidation node each TP2 rank took 380–450 s to load its weights. Raise\nthe limit with `VLLM_ENGINE_READY_TIMEOUT_S=1800`.\n" } ], "guide": "## Overview\n\n[Qwen3.8-Flash-Next](https://huggingface.co/Qwen/Qwen3.8-Flash-Next) is a\nmultimodal, ultra-sparse Mixture-of-Experts model. It has 125B parameters, \nincluding an additional 51B N-gram embedding table, while activating 6B \nparameters per token.\n\nThe architecture combines four main ideas:\n\n- **GDN + QSA:** three of every four layers use Gated DeltaNet to compress history;\n the fourth uses Qwen Sparse Attention for precise long-range retrieval.\n- **Gated Residual:** four residual branches dynamically control cross-layer reads\n and writes.\n- **N-gram Embedding:** a 51B lookup memory adds capacity with little per-token\n compute and can be asynchronously offloaded to host memory. (note that offload\n currently only runs on Nvidia devices)\n- **MTP:** the built-in Multi-Token Prediction module supports speculative decoding.\n\nThe checkpoint natively supports 262,144 tokens. Qwen reports that QSA reaches up\nto 10.2x prefill and 6.6x decode attention-kernel speedups at one million tokens.\nOn GB300, TEP4 is the recommended full-tray configuration. On an 8x H200 node,\nuse TEP8 for the official FP8 checkpoint; plain TP8 is incompatible with its\n128-wide quantization blocks.\n\n## Prerequisites\n\n- **Runtime:** use the nightly image, `vllm/vllm-openai:nightly` on NVIDIA or\n `vllm/vllm-openai-rocm:nightly` on AMD, or the nightly PyPI wheels. Stable\n ≥0.30.0 runs the model, but this recipe is tuned and verified on nightly.\n- **Hardware:** NVIDIA CUDA and AMD ROCm instructions are provided below.\n- **FP8 memory:** the checkpoint is 172.78 GiB; it fits on a single GB300 GPU\n (288 GB HBM3e).\n- **BF16 memory:** the checkpoint is 335.28 GiB. The validated TP2 configuration\n used about 190 GiB per GB300 GPU.\n- **MXFP4 memory:** the AMD Quark MXFP4 checkpoint is 169.8 GiB; it fits on a\n single MI355X GPU (288 GB HBM3E).\n- **Host memory for N-gram Embedding offload:** at least 51 GB plus\n runtime headroom.\n\nThe Install block above generates the nightly-wheel and Docker launch flows.\n\n## Serving\n\n### Recommended FP8 configuration on 4x GB300\n\nUse the official\n[`Qwen/Qwen3.8-Flash-Next-FP8`](https://huggingface.co/Qwen/Qwen3.8-Flash-Next-FP8)\ncheckpoint tested by this recipe.\n\n```bash\nvllm serve Qwen/Qwen3.8-Flash-Next-FP8 \\\n --tensor-parallel-size 4 \\\n --enable-expert-parallel \\\n --gpu-memory-utilization 0.90 \\\n --max-num-seqs 256 \\\n --enable-prefix-caching \\\n --kv-cache-dtype fp8 \\\n --attention-config.indexer_kv_dtype fp8 \\\n --speculative-config '{\"method\":\"mtp\",\"num_speculative_tokens\":3}' \\\n --enable-auto-tool-choice \\\n --tool-call-parser qwen3_coder \\\n --reasoning-parser qwen3\n```\n\nTEP4 is the recommended full-tray configuration — we recommend it over plain\nTP4 because expert parallelism is consistently faster for this model. TP4 and\nTEP4 are both validated full-tray configurations, including with MTP3.\n\nTo serve BF16, use the `Qwen/Qwen3.8-Flash-Next` checkpoint with the same\narguments.\n\n### Single-GPU deployment on GB200/GB300\n\nThe model fits on a single GPU: the FP8 checkpoint on one GB300 (288 GB\nHBM3e), and the NVFP4 variant on GB200 as well. Single-GPU TP1 suits\nevaluation and development; multi-GPU TEP remains the recommended deployment\nfor serving throughput.\n\n### Recommended FP8 configuration on 8x H200\n\nUse TEP8 with the Triton MoE backend on Hopper:\n\n```bash\nvllm serve Qwen/Qwen3.8-Flash-Next-FP8 \\\n --tensor-parallel-size 8 \\\n --enable-expert-parallel \\\n --moe-backend triton \\\n --gpu-memory-utilization 0.85 \\\n --max-num-seqs 256 \\\n --enable-prefix-caching \\\n --kv-cache-dtype fp8 \\\n --attention-config.indexer_kv_dtype fp8 \\\n --speculative-config '{\"method\":\"mtp\",\"num_speculative_tokens\":3}' \\\n --enable-auto-tool-choice \\\n --tool-call-parser qwen3_coder \\\n --reasoning-parser qwen3\n```\n\nPlain TP8 is incompatible with the FP8 checkpoint; use TEP8.\n\n### Recommended FP8 configuration on 8x H100\n\nH100's 80GB/GPU is not enough headroom for the 51B N-gram/PLE embedding table\n— it relies on PLE CPU offload, which vLLM enables by default. Use TEP8 with\nthe Triton MoE backend; as on other hardware, expert parallelism is faster\nthan plain TP, and plain TP8 is incompatible with the FP8 checkpoint's\n128-wide quantization blocks.\n\n```bash\nvllm serve Qwen/Qwen3.8-Flash-Next-FP8 \\\n --tensor-parallel-size 8 \\\n --enable-expert-parallel \\\n --moe-backend triton \\\n --gpu-memory-utilization 0.85 \\\n --max-num-seqs 256 \\\n --enable-prefix-caching \\\n --kv-cache-dtype fp8 \\\n --attention-config.indexer_kv_dtype fp8 \\\n --speculative-config '{\"method\":\"mtp\",\"num_speculative_tokens\":3}' \\\n --enable-auto-tool-choice \\\n --tool-call-parser qwen3_coder \\\n --reasoning-parser qwen3\n```\n\nBenchmark MTP with SPEED-Bench (`--dataset-name speed_bench`), not `random`.\n\n### MXFP4 on MI355X\n\nOn MI355X, the MXFP4 variant serves AMD's\n[`amd/Qwen3.8-Flash-Next-Quark-MXFP4`](https://huggingface.co/amd/Qwen3.8-Flash-Next-Quark-MXFP4)\ncheckpoint, which quantizes the routed experts to MXFP4 with AMD Quark. The\nvariant sets:\n\n- `VLLM_ROCM_USE_AITER=1`, which turns on the AITER kernels.\n- `VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS=1`, which fuses the shared\n expert into the routed-expert MoE kernel.\n- `AITER_USE_FLYDSL_MOE_SORTING=1`, which sorts MoE tokens with a FlyDSL\n kernel.\n- `--gdn-prefill-backend aiter_flydsl`, which runs Gated DeltaNet prefill\n with a FlyDSL kernel.\n\nThe default deployment is TEP8, and the TP strategy runs TP2. The checkpoint\nalso fits on a single MI355X (288 GB HBM3E), so TP1 with the same flags\nserves one replica per GPU; the builder offers one TP size per variant, so\nset `--tensor-parallel-size 1` yourself. TP2 leaves more memory for KV cache\nand gives slightly lower per-token latency; TP1 gives more throughput per\nGPU.\n\nValidated on `vllm/vllm-openai-rocm:nightly-83c078d955cd6e0c63d3b12abe669c8de936a370`\nwith the builder's TEP8 and TP2 commands, and TP1 as the TP2 command with\n`--tensor-parallel-size 1`. GSM8K is 5-shot over 1,319 questions through\nchat completions with thinking off; throughput uses random 8k-token prompts\nand 1k-token outputs.\n\n| MXFP4 on MI355X | GSM8K | Output tok/s, concurrency 8 / 32 |\n| --- | --- | --- |\n| TEP8 | 0.965 | 665 / 1831 |\n| TP2 | 0.967 | 801 / 1384 |\n| TP1 | 0.967 | 751 / 1282 |\n\n### FP8 configuration on a 4x MI355X GPU\n```bash\nexport VLLM_ROCM_USE_AITER=1\nexport VLLM_ROCM_USE_AITER_MOE=0\n\nvllm serve Qwen/Qwen3.8-Flash-Next-FP8 \\\n--tensor-parallel-size 4 \\\n--max-model-len auto \\\n--gpu-memory-utilization 0.9\n```\n\nMTP is opt-in on MI355X; to turn it on, add the option from the MTP\nspeculative decoding section below.\n\n### Full native context\n\nThe checkpoint advertises a native 262,144-token context, which vLLM uses when\n`--max-model-len` is omitted. Startup and bounded evaluation were validated with\nthis limit configured, but a single 262K-token request was not tested.\n\n### MTP speculative decoding\n\nAdd the following option:\n\n```bash\n--speculative-config '{\"method\":\"mtp\",\"num_speculative_tokens\":3}'\n```\n\n### Offload the 51B N-gram embedding table\n\nBuilds containing N-gram Embedding offload can keep the N-gram lookup\nmemory in host RAM and asynchronously prefetch the required rows. The initial\nimplementation supports both ModelRunner V1 and V2. PLE CPU offload is required\nfor DEP, and also required for TP/TEP on 80GB-class GPUs (H100) where the\nembedding table alone exceeds available headroom — optional only on GPUs with\nmore VRAM per rank (H200 and above). vLLM enables it by default; set it\nexplicitly with `--engram-config '{\"cpu_offload\":true}'`.\n\n```bash\nvllm serve Qwen/Qwen3.8-Flash-Next-FP8 \\\n --tensor-parallel-size 4 \\\n --enable-expert-parallel \\\n --max-model-len 262144 \\\n --enable-auto-tool-choice \\\n --tool-call-parser qwen3_coder \\\n --reasoning-parser qwen3\n```\n\n### Extend to one million tokens with YaRN\n\nThe checkpoint is native at 262K. For a one-million-token workload, turn on the\n**Long Context** feature above: it applies the model card's YaRN RoPE\nconfiguration under `text_config` (where this checkpoint carries\n`rope_parameters`) and raises `--max-model-len` to 1,000,000. Evaluate\nshorter-context quality before making it the default — static YaRN holds the\nscaling factor constant regardless of input length.\n\n## Verifying\n\n```python\nfrom openai import OpenAI\n\nclient = OpenAI(api_key=\"EMPTY\", base_url=\"http://localhost:8000/v1\")\nresponse = client.chat.completions.create(\n model=\"Qwen/Qwen3.8-Flash-Next-FP8\",\n messages=[{\n \"role\": \"user\",\n \"content\": \"Explain how Gated DeltaNet and Qwen Sparse Attention complement each other.\",\n }],\n max_tokens=8196,\n)\nprint(response.choices[0].message.content)\n```\n\nA correct response should explain that GDN maintains a compact recurrent summary,\nwhile QSA selectively retrieves important regions from the full token history.\n\n## Troubleshooting\n\n- **FP8 on eight GPUs:** use TEP8; plain TP8 is incompatible with this\n checkpoint. TP4 is also supported if GPU memory capacity allows.\n- **Mamba-cache capacity error at startup:** keep `--max-num-seqs 256`.\n- **Runtime OOM on large multimodal batches:** keep `--gpu-memory-utilization 0.90`.\n- **Out of memory while loading:** enable N-gram Embedding CPU offload, increase\n TP size, or reduce `--max-model-len` to reserve less KV cache. On 4x H100\n 80GB, plain TP4 OOMs at engine startup without PLE CPU offload — keep the\n default `--engram-config '{\"cpu_offload\":true}'`.\n- **XPU/TPU startup error:** these platforms are not supported by the initial\n implementation.\n- **Pipeline-parallel startup error:** N-gram Embedding does not initially support\n pipeline parallelism; use single-node TP or TEP instead.\n- **DEP startup failure:** DEP requires PLE CPU offload (on by default); DEP\n without it is not supported.\n- **MTP memory pressure:** reduce `num_speculative_tokens` below 3.\n- **PLE:** a network layer that injects N-gram Embeddings into the main model. \n\n## KV cache offloading\n\nOffloading is available upstream since vLLM v0.30.0 ([vllm-project/vllm#57145](https://github.com/vllm-project/vllm/pull/57145)).\nValidation used [Qwen/Qwen3.8-Flash-Next-FP8](https://huggingface.co/Qwen/Qwen3.8-Flash-Next-FP8) on 4×H200 (TP4),\nvLLM commit `d1b4028d7e`, the Python frontend, `--enforce-eager`,\n`--max-model-len 16384`, `--max-num-seqs 16` and `--max-num-batched-tokens 8192`.\nQwen validation used the FP8 variant with `--limit-mm-per-prompt '{\"image\":0,\"video\":0}'`.\nValidation used `TieringOffloadingSpec`: 8 GiB total CPU tier, `blocks_per_chunk: 1`,\na persistent filesystem `root_dir`, eight read/write threads and `PYTHONHASHSEED=0`.\nThree CPU-tier reloads and three filesystem reloads after restarting the server passed.\nReloaded responses matched the cold output, including the reasoning field.\nThe five-shot GSM8K diagnostic scored 31/32; this is a small sample, not the full benchmark.\nThese results cover this text-only configuration, not the builder's larger tier/chunk defaults or other variants.\n\n## References\n\n- [Qwen3.8-Flash-Next model card](https://huggingface.co/Qwen/Qwen3.8-Flash-Next)\n- [Qwen3.8-Flash-Next FP8 checkpoint](https://huggingface.co/Qwen/Qwen3.8-Flash-Next-FP8)\n- [Qwen3.8-Flash-Next: A New Architecture, Towards Ultimate Cost-Efficiency](https://qwen.ai/blog?id=qwen3.8-flash-next)\n- [Qwen3-Next recipe](../Qwen3-Next-80B-A3B-Instruct)\n", "source": { "source": "vllm", "repo": "vllm-project/vllm", "ref": "main", "sha": "564ac0de1b228568cafcbfbe8ccaccc82d6023eb", "path": "recipes/models/Qwen/Qwen3.8-Flash-Next.yaml", "edit_url": "https://github.com/vllm-project/vllm/edit/main/recipes/models/Qwen/Qwen3.8-Flash-Next.yaml", "overlays": [] } }