deepseek-ai/DeepSeek-V4-Flash

DeepSeek V4 MoE model with hybrid CSA+HCA attention, manifold-constrained hyper-connections, and three-tier reasoning (Non-think / Think High / Think Max).

Compact 284B/13B V4 sibling — single-node 1M-context serving with FP4+FP8 weights, MTP, and DSpark speculative decoding

View on HuggingFace

moe284B / 13B1,048,576 ctxvLLM 0.20.0+text

Guide

Overview

DeepSeek-V4-Flash is a 284B-total / 13B-active MoE model in the V4 preview family. It pairs a hybrid attention stack — Compressed Sparse Attention (CSA) + Heavily Compressed Attention (HCA) — with Manifold-Constrained Hyper-Connections (mHC) to reach 27% of V3.2's per-token inference FLOPs and 10% of V3.2's KV cache at 1M context. Pre-trained on 32T+ tokens; post-training is a two-stage pipeline (domain- specific expert cultivation + unified consolidation via on-policy distillation).

Checkpoint is FP4+FP8 mixed: MoE expert weights are stored in FP4 while the remaining (attention / norm / router) params stay in FP8.

An NVFP4 variant (nvidia/DeepSeek-V4-Flash-NVFP4) is also available — NVIDIA modelopt re-quantizes the MoE experts to standard NVFP4 while attention, shared experts, router head, and MTP stay FP8. Pick it from the Variant row; it runs on Blackwell GPUs with the FP4 indexer cache. Unlike the native FP8 checkpoint, the NVFP4 experts don't support the deep_gemm_mega_moe MoE kernel (FP8-only), so it runs on the default MoE backend.

Checkpoints

Four checkpoints are on the Variant row. They differ in which weights you serve and which draft module ships with them — the speculative method itself is picked on the Spec Decoding row.

Variant	Repo	Draft	Notes
FP8 (0731) (default)	`deepseek-ai/DeepSeek-V4-Flash-0731`	DSpark	Official release, new weights + DSpark
FP8 (Preview)	`deepseek-ai/DeepSeek-V4-Flash`	MTP	Preview FP4+FP8 mixed weights
NVFP4	`nvidia/DeepSeek-V4-Flash-NVFP4`	MTP	modelopt re-quant, Blackwell
DSpark	`deepseek-ai/DeepSeek-V4-Flash-DSpark`	DSpark	Preview weights + fused DSpark module

0731 is the official DeepSeek-V4-Flash release and the default here, superseding the preview with substantially stronger agentic capability — 82.7 on Terminal Bench 2.1 and 54.4 on DeepSWE, versus 61.8 and 7.3 for the preview. It beats DeepSeek-V4-Pro (Preview) on every agentic benchmark DeepSeek published despite its far smaller activated parameter count. The preview weights remain available as the FP8 (Preview) variant for reproducing earlier results.

DSpark is not new weights — it is the preview checkpoint with a speculative decoding module attached (see DeepSpec). Both fused checkpoints add a dspark_* block to config.json; the preview and NVFP4 checkpoints have no such block, which is why the DSpark method is offered only on the two variants that carry the draft. Selecting either one auto-enables Spec Decoding with method: dspark, emitting:

--speculative-config '{"method":"dspark","num_speculative_tokens":7,"draft_sample_method":"probabilistic"}'

The fused checkpoints are ~167 GB on disk versus ~160 GB for the preview — the draft module is the difference. Both require vLLM 0.25.0: DSpark drafting itself landed in 0.25.0, but ROCm support for it (MI325X / MI355X) only shipped in 0.26.0. On NVIDIA-only deployments 0.25.0 is sufficient, except the 0731 checkpoint on RTX PRO 6000 (sm_120) with DSpark, which needs vllm/vllm-openai:nightly and a FlashInfer build containing flashinfer-ai/flashinfer#4380.

Reasoning modes

The chat template exposes three reasoning-effort modes:

Non-think — fast, intuitive responses.
Think High — explicit chain-of-thought for logical analysis and planning.
Think Max — maximum reasoning effort; requires --max-model-len >= 393216 (384K tokens) to avoid truncation.

Recommended sampling: temperature = 1.0, top_p = 1.0.

On the 0731 variant the effort levels are named low / high / max, and DeepSeek recommends top_p = 0.95 for agentic scenarios (1.0 otherwise) with temperature = 1.0. Allow up to 384K output tokens at the high and max levels.

Note that neither fused checkpoint ships a Jinja chat template — the repos provide an encoding/ folder with encode_messages / parse_message_from_completion_text helpers instead. Serving through vLLM with --tokenizer-mode deepseek_v4 (the Tool Calling pill) applies the built-in DeepSeek-V4 encoding, so the OpenAI- compatible endpoint works without the helper scripts.

OpenAI Client Example

For DeepSeek-V4, keep reasoning controls in chat_template_kwargs, as it exposes a custom Think Max mode via "reasoning_effort": "max".

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
model = "deepseek-ai/DeepSeek-V4-Flash"
messages = [{"role": "user", "content": "What is 17*19? Return only the final integer."}]

# Non-think
resp = client.chat.completions.create(
    model=model,
    messages=messages,
)

# Think High
resp = client.chat.completions.create(
    model=model,
    messages=messages,
    extra_body={
        "chat_template_kwargs": {
            "thinking": True,
            "reasoning_effort": "high",
        },
    },
)

# Think Max
resp = client.chat.completions.create(
    model=model,
    messages=messages,
    extra_body={
        "chat_template_kwargs": {
            "thinking": True,
            "reasoning_effort": "max",
        },
    },
)

Recommended deployment

Non-disaggregated serving on multi-GPU supported hardware: single-node DP + EP with --data-parallel-size 4. Fills a GB200 NVL4 tray exactly; uses 4 of 8 GPUs per replica on H200/B200/B300 (leaving headroom for throughput-vs-latency tuning). On DGX Station, use the single-GPU launch below. For disaggregated prefill/decode on GB200, use the PD Cluster tab.

DGX Spark (GB10) cluster

DGX Spark requires the community Spark vLLM build for this model rather than the stock vLLM release/nightly image. The DGX Spark (GB10) hardware pill pins the Docker tab to eugr/spark-vllm-b12x:latest, whose build carries the B12X backends and Spark-specific kernels for the GB10 / SM12x family.

Use 2 nodes. A single GB10's 128 GB unified memory is below the FP8/NVFP4 checkpoint footprint, so the recipe grants DGX Spark a Spark-only Multi-Node TP layout (--tensor-parallel-size 2, one GB10 per node). The generated command also sets the B12X runtime environment and the Spark network environment; export IFACE_NAME, IB_IF and HEAD_IP on each node before launching.

To identify the values, first list the ConnectX Ethernet interfaces, RoCE devices and the IP address of the UP ConnectX interface on each Spark:

$ ibdev2netdev
roceP2p1s0f0 port 1 ==> enP2p1s0f0np0 (Up)
roceP2p1s0f1 port 1 ==> enP2p1s0f1np1 (Down)
rocep1s0f0 port 1 ==> enp1s0f0np0 (Up)
rocep1s0f1 port 1 ==> enp1s0f1np1 (Down)

$ ip addr show enp1s0f0np0
3: enp1s0f0np0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 4c:62:56:c1:83:68 brd ff:ff:ff:ff:ff:ff
    inet 100.100.100.10/24 brd 100.100.100.255 scope global noprefixroute enp1s0f0np0
      valid_lft forever preferred_lft forever

From above example values, we can get the following common values:

# Common values for all nodes
HEAD_IP="100.100.100.10"         # IP of the head node's UP ConnectX interface
IFACE_NAME="enp1s0f0np0"         # The head node's UP ConnectX Ethernet interface
IB_IF="rocep1s0f0,roceP2p1s0f0"  # Comma-separated RoCE device name list corresponding to the UP ConnectX interface

DGX Station Single-GPU

vllm serve deepseek-ai/DeepSeek-V4-Flash \
  --tensor-parallel-size 1 --pipeline-parallel-size 1 \
  --kv-cache-dtype fp8 --trust-remote-code --block-size 256 \
  --gpu-memory-utilization 0.92 \
  --compilation-config '{"cudagraph_mode":"FULL_AND_PIECEWISE","custom_ops":["all"]}' \
  --attention_config.use_fp4_indexer_cache True \
  --tokenizer-mode deepseek_v4 --tool-call-parser deepseek_v4 \
  --enable-auto-tool-choice --reasoning-parser deepseek_v4 \
  --max-cudagraph-capture-size 128 \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}'

RTX PRO 6000 8× (8×96 GB, sm_120)

Single-node TP8 + expert parallelism on the 0731 checkpoint. Serving without speculative decoding was verified on 8× RTX PRO 6000 (PCIe, no NVLink). DSpark on 0731 needs vllm/vllm-openai:nightly (or a build that includes vLLM #51538) plus a FlashInfer that instantiates the SM120 sparse-MLA decode kernel for topk=192. The Preview, NVFP4, and DSpark variant checkpoints are not verified on this profile.

sm_120 cannot use the SM100 FP4 indexer cache or deep_gemm_mega_moe. The generated command already sets --attention_config.use_fp4_indexer_cache False and --moe-backend auto. Do not use method: mtp on 0731 — that checkpoint has no MTP head; Spec Decoding defaults to DSpark.

Host-side FlashInfer install before launching 0731 with DSpark (--no-deps is required so pip does not replace vLLM's torch):

# Trace: flashinfer-ai/flashinfer#4380
# merge: 24d7dfb2639083c5a4d418881099421fc800b7bb (2026-08-08)
# wheel: flashinfer-python==0.6.18.dev20260814
# index: https://flashinfer.ai/whl/nightly/
/usr/bin/python3.12 -m pip install -U --pre flashinfer-python \
--index-url https://flashinfer.ai/whl/nightly/ --no-deps
/usr/bin/python3.12 -m pip install flashinfer-python
/usr/bin/python3.12 -m pip install -U --pre flashinfer-cubin \
--index-url https://flashinfer.ai/whl/nightly/
/usr/bin/python3.12 -m pip install -U --pre flashinfer-jit-cache \
--index-url https://flashinfer.ai/whl/nightly/cu130

vllm serve deepseek-ai/DeepSeek-V4-Flash-0731 \
--trust-remote-code \
--kv-cache-dtype fp8 \
--block-size 256 \
--enable-expert-parallel \
--tensor-parallel-size 8 \
--moe-backend auto \
--attention_config.use_fp4_indexer_cache False \
--tokenizer-mode deepseek_v4 \
--tool-call-parser deepseek_v4 \
--enable-auto-tool-choice \
--reasoning-parser deepseek_v4 \
--reasoning-config '{"reasoning_parser":"deepseek_v4","reasoning_start_str":"","reasoning_end_str":""}' \
--speculative-config '{"method":"dspark","num_speculative_tokens":7,"draft_sample_method":"probabilistic"}'

MI325X (1×256GB)

Validated end-to-end with vLLM 0.25.0 and PyTorch HIP 7.2.53211. Use this conservative TP1 eager-mode launch instead of the MI355X throughput settings:

export VLLM_ROCM_USE_AITER=1

vllm serve deepseek-ai/DeepSeek-V4-Flash \
  --host 0.0.0.0 \
  --port 8002 \
  --tensor-parallel-size 1 \
  --kv-cache-dtype fp8_e4m3 \
  --max-model-len 4096 \
  --enable-chunked-prefill \
  --max-num-batched-tokens 256 \
  --kv-cache-memory-bytes 10000000000 \
  --distributed-executor-backend mp \
  --trust-remote-code \
  --tokenizer-mode deepseek_v4 \
  --moe-backend triton_unfused \
  --enforce-eager

Chunked prefill allows 4K requests while scheduling at most 256 tokens per batch. This configuration loaded the 148.66 GiB checkpoint and passed model-list and chat probes in non-thinking and Think High modes during a 24-hour allocation. Only TP1 at 4K context is verified; TP2+ did not validate successfully. Think Max requires at least 384K context and is not available with this configuration.

MI355X (4×288GB)

If you want to further lower the latency, you can run with --tensor-parallel-size 8.

export VLLM_ROCM_USE_AITER=1

vllm serve deepseek-ai/DeepSeek-V4-Flash \
  --host localhost \
  --port 8001 \
  --dtype auto \
  --kv-cache-dtype fp8 \
  --tensor-parallel-size 4 \
  --max-num-seqs 512 \
  --max-num-batched-tokens 8192 \
  --distributed-executor-backend mp \
  --trust-remote-code \
  --gpu-memory-utilization 0.9 \
  --tokenizer-mode deepseek_v4 \
  --reasoning-parser deepseek_v4 \
  --tool-call-parser deepseek_v4 \
  --enable-auto-tool-choice \
  --compilation-config '{"mode": 3, "cudagraph_mode": "FULL_DECODE_ONLY"}'

MI355X is validated on GSM8K dataset:

Launch command

MODEL=deepseek-ai/DeepSeek-V4-Flash
lm_eval --model local-completions \
  --model_args model=$MODEL,base_url=http://0.0.0.0:8001/v1/completions,num_concurrent=128,max_retries=10,max_gen_toks=2048,timeout=60000 \
  --batch_size auto \
  --tasks gsm8k \
  --num_fewshot 8 \
  --output_path . 2>&1 | tee -a eval.log

Reported result

local-completions ({'model': 'deepseek-ai/DeepSeek-V4-Flash', 'base_url': 'http://0.0.0.0:8001/v1/completions', 'num_concurrent': 128, 'max_retries': 10, 'max_gen_toks': 2048, 'timeout': 60000}), gen_kwargs: ({}), limit: None, num_fewshot: 8, batch_size: auto
|Tasks|Version|     Filter     |n-shot|  Metric   |   |Value |   |Stderr|
|-----|------:|----------------|-----:|-----------|---|-----:|---|-----:|
|gsm8k|      3|flexible-extract|     8|exact_match|↑  |0.9439|±  |0.0063|
|     |       |strict-match    |     8|exact_match|↑  |0.9431|±  |0.0064|

H200 Single-Node PD (Mooncake)

Single-host disaggregated serving: 4 prefill GPUs + 4 decode GPUs on one 8-GPU H200 node, using MooncakeConnector over RDMA for KV cache transfer.

Prefill (GPUs 0–3, port 8000):

docker run --gpus all \
  --privileged --ipc=host -p 8000:8000 \
  --network host \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -v /mnt/shared:/mnt/shared \
  -e TILELANG_CLEANUP_TEMP_FILES=1 \
  -e VLLM_DISABLE_COMPILE_CACHE=1 \
  -e VLLM_ENGINE_READY_TIMEOUT_S=3600 \
  -e VLLM_RPC_TIMEOUT=600000 \
  -e VLLM_LOG_STATS_INTERVAL=1 \
  -e VLLM_MOONCAKE_BOOTSTRAP_PORT=8998 \
  -e CUDA_VISIBLE_DEVICES=0,1,2,3 \
  vllm/vllm-openai:v0.25.0 \
  deepseek-ai/DeepSeek-V4-Flash \
  --trust-remote-code \
  --kv-cache-dtype fp8 \
  --block-size 256 \
  --port 8000 \
  --data-parallel-size 4 \
  --enable-expert-parallel \
  --tokenizer-mode deepseek_v4 \
  --reasoning-parser deepseek_v4 \
  --max-model-len auto \
  --max-num-batched-tokens 16384 \
  --max-num-seqs 8 \
  --enforce-eager \
  --no-disable-hybrid-kv-cache-manager \
  --disable-uvicorn-access-log \
  --kv-transfer-config '{"kv_connector":"MooncakeConnector","kv_role":"kv_both","kv_load_failure_policy":"fail","kv_buffer_device":"cuda","kv_connector_extra_config":{"enforce_handshake_compat":false,"mooncake_protocol":"rdma"}}'

Decode (GPUs 4–7, port 8001):

docker run --gpus all \
  --privileged --ipc=host -p 8001:8001 \
  --network host \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -v /mnt/shared:/mnt/shared \
  -e TILELANG_CLEANUP_TEMP_FILES=1 \
  -e VLLM_DISABLE_COMPILE_CACHE=1 \
  -e VLLM_ENGINE_READY_TIMEOUT_S=3600 \
  -e VLLM_RPC_TIMEOUT=600000 \
  -e VLLM_LOG_STATS_INTERVAL=1 \
  -e VLLM_MOONCAKE_BOOTSTRAP_PORT=9889 \
  -e CUDA_VISIBLE_DEVICES=4,5,6,7 \
  vllm/vllm-openai:v0.25.0 \
  deepseek-ai/DeepSeek-V4-Flash \
  --trust-remote-code \
  --kv-cache-dtype fp8 \
  --block-size 256 \
  --port 8001 \
  --data-parallel-size 4 \
  --enable-expert-parallel \
  --tokenizer-mode deepseek_v4 \
  --reasoning-parser deepseek_v4 \
  --max-model-len auto \
  --max-num-seqs 512 \
  --max-num-batched-tokens 512 \
  --compilation-config '{"mode":0,"cudagraph_mode":"FULL_DECODE_ONLY","max_cudagraph_capture_size":512,"compile_ranges_endpoints":[512]}' \
  --no-disable-hybrid-kv-cache-manager \
  --disable-uvicorn-access-log \
  --kv-transfer-config '{"kv_connector":"MooncakeConnector","kv_role":"kv_both","kv_load_failure_policy":"fail","kv_buffer_device":"cuda","kv_connector_extra_config":{"enforce_handshake_compat":false,"mooncake_protocol":"rdma"}}'

Router:

pip install vllm-router

vllm-router --policy round_robin \
  --vllm-pd-disaggregation \
  --prefill http://localhost:8000 \
  --decode http://localhost:8001 \
  --host 127.0.0.1 \
  --port 30000 \
  --intra-node-data-parallel-size 4 \
  --kv-connector mooncake

H200 Single-Node PD (Nixl)

Single-host disaggregated serving: 4 prefill GPUs + 4 decode GPUs on one 8-GPU H200 node, using NixlConnector for KV cache transfer.

Prefill (GPUs 0–3, port 8000):

docker run --gpus all \
  --privileged --ipc=host -p 8000:8000 \
  --network host \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -v /mnt/shared:/mnt/shared \
  -e TILELANG_CLEANUP_TEMP_FILES=1 \
  -e VLLM_DISABLE_COMPILE_CACHE=1 \
  -e VLLM_ENGINE_READY_TIMEOUT_S=3600 \
  -e VLLM_RPC_TIMEOUT=600000 \
  -e VLLM_LOG_STATS_INTERVAL=1 \
  -e VLLM_NIXL_SIDE_CHANNEL_PORT=5557 \
  -e CUDA_VISIBLE_DEVICES=0,1,2,3 \
  vllm/vllm-openai:v0.25.0 \
  deepseek-ai/DeepSeek-V4-Flash \
  --trust-remote-code \
  --kv-cache-dtype fp8 \
  --block-size 256 \
  --port 8000 \
  --data-parallel-size 4 \
  --enable-expert-parallel \
  --tokenizer-mode deepseek_v4 \
  --reasoning-parser deepseek_v4 \
  --max-model-len auto \
  --max-num-batched-tokens 16384 \
  --max-num-seqs 8 \
  --enforce-eager \
  --no-disable-hybrid-kv-cache-manager \
  --disable-uvicorn-access-log \
  --kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_both"}'

Decode (GPUs 4–7, port 8001):

docker run --gpus all \
  --privileged --ipc=host -p 8001:8001 \
  --network host \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -v /mnt/shared:/mnt/shared \
  -e TILELANG_CLEANUP_TEMP_FILES=1 \
  -e VLLM_DISABLE_COMPILE_CACHE=1 \
  -e VLLM_ENGINE_READY_TIMEOUT_S=3600 \
  -e VLLM_RPC_TIMEOUT=600000 \
  -e VLLM_LOG_STATS_INTERVAL=1 \
  -e VLLM_NIXL_SIDE_CHANNEL_PORT=5558 \
  -e CUDA_VISIBLE_DEVICES=4,5,6,7 \
  vllm/vllm-openai:v0.25.0 \
  deepseek-ai/DeepSeek-V4-Flash \
  --trust-remote-code \
  --kv-cache-dtype fp8 \
  --block-size 256 \
  --port 8001 \
  --data-parallel-size 4 \
  --enable-expert-parallel \
  --tokenizer-mode deepseek_v4 \
  --reasoning-parser deepseek_v4 \
  --max-model-len auto \
  --max-num-seqs 512 \
  --max-num-batched-tokens 512 \
  --compilation-config '{"mode":0,"cudagraph_mode":"FULL_DECODE_ONLY","max_cudagraph_capture_size":512,"compile_ranges_endpoints":[512]}' \
  --no-disable-hybrid-kv-cache-manager \
  --disable-uvicorn-access-log \
  --kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_both"}'

Router:

pip install vllm-router

vllm-router --policy round_robin \
  --vllm-pd-disaggregation \
  --prefill http://localhost:8000 \
  --decode http://localhost:8001 \
  --host 127.0.0.1 \
  --port 30000 \
  --intra-node-data-parallel-size 4 \
  --kv-connector nixl