deepseek-ai/DeepSeek-V4-Flash

Overview

DeepSeek-V4-Flash is a 284B-total / 13B-active MoE model in the V4 preview family. It pairs a hybrid attention stack — Compressed Sparse Attention (CSA) + Heavily Compressed Attention (HCA) — with Manifold-Constrained Hyper-Connections (mHC) to reach 27% of V3.2's per-token inference FLOPs and 10% of V3.2's KV cache at 1M context. Pre-trained on 32T+ tokens; post-training is a two-stage pipeline (domain- specific expert cultivation + unified consolidation via on-policy distillation).

Checkpoint is FP4+FP8 mixed: MoE expert weights are stored in FP4 while the remaining (attention / norm / router) params stay in FP8.

An NVFP4 variant (nvidia/DeepSeek-V4-Flash-NVFP4) is also available — NVIDIA modelopt re-quantizes the MoE experts to standard NVFP4 while attention, shared experts, router head, and MTP stay FP8. Pick it from the Variant row; it runs on Blackwell GPUs with the FP4 indexer cache. Unlike the native FP8 checkpoint, the NVFP4 experts don't support the deep_gemm_mega_moe MoE kernel (FP8-only), so it runs on the default MoE backend.

Checkpoints

Four checkpoints are on the Variant row. They differ in which weights you serve and which draft module ships with them — the speculative method itself is picked on the Spec Decoding row.

Variant	Repo	Draft	Notes
0731 (default)	deepseek-ai/DeepSeek-V4-Flash-0731	DSpark	Official release, new weights + DSpark
FP8	deepseek-ai/DeepSeek-V4-Flash	MTP	Preview FP4+FP8 mixed weights
NVFP4	nvidia/DeepSeek-V4-Flash-NVFP4	MTP	modelopt re-quant, Blackwell
DSpark	deepseek-ai/DeepSeek-V4-Flash-DSpark	DSpark	Preview weights + fused DSpark module

0731 is the official DeepSeek-V4-Flash release and the default here, superseding the preview with substantially stronger agentic capability — 82.7 on Terminal Bench 2.1 and 54.4 on DeepSWE, versus 61.8 and 7.3 for the preview. It beats DeepSeek-V4-Pro (Preview) on every agentic benchmark DeepSeek published despite its far smaller activated parameter count. The preview weights remain available as the FP8 variant for reproducing earlier results.

DSpark is not new weights — it is the preview checkpoint with a speculative decoding module attached (see DeepSpec). Both fused checkpoints add a dspark_* block to config.json; the preview and NVFP4 checkpoints have no such block, which is why the DSpark method is offered only on the two variants that carry the draft. Selecting either one auto-enables Spec Decoding with method: dspark, emitting:

--speculative-config '{"method":"dspark","num_speculative_tokens":7,"draft_sample_method":"greedy"}'

The fused checkpoints are ~167 GB on disk versus ~160 GB for the preview — the draft module is the difference. Both require vLLM 0.25.0: DSpark drafting itself landed in 0.25.0, but ROCm support for it (MI325X / MI355X) only shipped in 0.26.0. On NVIDIA-only deployments 0.25.0 is sufficient.

Reasoning modes

The chat template exposes three reasoning-effort modes:

• Non-think — fast, intuitive responses.
• Think High — explicit chain-of-thought for logical analysis and planning.
• Think Max — maximum reasoning effort; requires --max-model-len >= 393216 (384K tokens) to avoid truncation.

Recommended sampling: temperature = 1.0, top_p = 1.0.

On the 0731 variant the effort levels are named low / high / max, and DeepSeek recommends top_p = 0.95 for agentic scenarios (1.0 otherwise) with temperature = 1.0. Allow up to 384K output tokens at the high and max levels.

Note that neither fused checkpoint ships a Jinja chat template — the repos provide an encoding/ folder with encode_messages / parse_message_from_completion_text helpers instead. Serving through vLLM with --tokenizer-mode deepseek_v4 (the Tool Calling pill) applies the built-in DeepSeek-V4 encoding, so the OpenAI- compatible endpoint works without the helper scripts.

OpenAI Client Example

For DeepSeek-V4, keep reasoning controls in chat_template_kwargs, as it exposes a custom Think Max mode via "reasoning_effort": "max".

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
model = "deepseek-ai/DeepSeek-V4-Flash"
messages = [{"role": "user", "content": "What is 17*19? Return only the final integer."}]

# Non-think
resp = client.chat.completions.create(
    model=model,
    messages=messages,
)

# Think High
resp = client.chat.completions.create(
    model=model,
    messages=messages,
    extra_body={
        "chat_template_kwargs": {
            "thinking": True,
            "reasoning_effort": "high",
        },
    },
)

# Think Max
resp = client.chat.completions.create(
    model=model,
    messages=messages,
    extra_body={
        "chat_template_kwargs": {
            "thinking": True,
            "reasoning_effort": "max",
        },
    },
)

Recommended deployment

Non-disaggregated serving on multi-GPU supported hardware: single-node DP + EP with --data-parallel-size 4. Fills a GB200 NVL4 tray exactly; uses 4 of 8 GPUs per replica on H200/B200/B300 (leaving headroom for throughput-vs-latency tuning). On DGX Station, use the single-GPU launch below. For disaggregated prefill/decode on GB200, use the PD Cluster tab.

DGX Station Single-GPU

vllm serve deepseek-ai/DeepSeek-V4-Flash \
  --tensor-parallel-size 1 --pipeline-parallel-size 1 \
  --kv-cache-dtype fp8 --trust-remote-code --block-size 256 \
  --gpu-memory-utilization 0.92 \
  --compilation-config '{"cudagraph_mode":"FULL_AND_PIECEWISE","custom_ops":["all"]}' \
  --attention_config.use_fp4_indexer_cache True \
  --tokenizer-mode deepseek_v4 --tool-call-parser deepseek_v4 \
  --enable-auto-tool-choice --reasoning-parser deepseek_v4 \
  --max-cudagraph-capture-size 128 \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}'

MI325X (1×256GB)

Validated end-to-end with vLLM 0.25.0 and PyTorch HIP 7.2.53211. Use this conservative TP1 eager-mode launch instead of the MI355X throughput settings:

export VLLM_ROCM_USE_AITER=1

vllm serve deepseek-ai/DeepSeek-V4-Flash \
  --host 0.0.0.0 \
  --port 8002 \
  --tensor-parallel-size 1 \
  --kv-cache-dtype fp8_e4m3 \
  --max-model-len 4096 \
  --enable-chunked-prefill \
  --max-num-batched-tokens 256 \
  --kv-cache-memory-bytes 10000000000 \
  --distributed-executor-backend mp \
  --trust-remote-code \
  --tokenizer-mode deepseek_v4 \
  --moe-backend triton_unfused \
  --enforce-eager

Chunked prefill allows 4K requests while scheduling at most 256 tokens per batch. This configuration loaded the 148.66 GiB checkpoint and passed model-list and chat probes in non-thinking and Think High modes during a 24-hour allocation. Only TP1 at 4K context is verified; TP2+ did not validate successfully. Think Max requires at least 384K context and is not available with this configuration.

MI355X (4×288GB)

If you want to further lower the latency, you can run with --tensor-parallel-size 8.

export VLLM_ROCM_USE_AITER=1

vllm serve deepseek-ai/DeepSeek-V4-Flash \
  --host localhost \
  --port 8001 \
  --dtype auto \
  --kv-cache-dtype fp8 \
  --tensor-parallel-size 4 \
  --max-num-seqs 512 \
  --max-num-batched-tokens 8192 \
  --distributed-executor-backend mp \
  --trust-remote-code \
  --gpu-memory-utilization 0.9 \
  --tokenizer-mode deepseek_v4 \
  --reasoning-parser deepseek_v4 \
  --tool-call-parser deepseek_v4 \
  --enable-auto-tool-choice \
  --compilation-config '{"mode": 3, "cudagraph_mode": "FULL_DECODE_ONLY"}'

MI355X is validated on GSM8K dataset:

Launch command

MODEL=deepseek-ai/DeepSeek-V4-Flash
lm_eval --model local-completions \
  --model_args model=$MODEL,base_url=http://0.0.0.0:8001/v1/completions,num_concurrent=128,max_retries=10,max_gen_toks=2048,timeout=60000 \
  --batch_size auto \
  --tasks gsm8k \
  --num_fewshot 8 \
  --output_path . 2>&1 | tee -a eval.log

Reported result

local-completions ({'model': 'deepseek-ai/DeepSeek-V4-Flash', 'base_url': 'http://0.0.0.0:8001/v1/completions', 'num_concurrent': 128, 'max_retries': 10, 'max_gen_toks': 2048, 'timeout': 60000}), gen_kwargs: ({}), limit: None, num_fewshot: 8, batch_size: auto
|Tasks|Version|     Filter     |n-shot|  Metric   |   |Value |   |Stderr|
|-----|------:|----------------|-----:|-----------|---|-----:|---|-----:|
|gsm8k|      3|flexible-extract|     8|exact_match|↑  |0.9439|±  |0.0063|
|     |       |strict-match    |     8|exact_match|↑  |0.9431|±  |0.0064|

H200 Single-Node PD (Mooncake)

Single-host disaggregated serving: 4 prefill GPUs + 4 decode GPUs on one 8-GPU H200 node, using MooncakeConnector over RDMA for KV cache transfer.

Prefill (GPUs 0–3, port 8000):

docker run --gpus all \
  --privileged --ipc=host -p 8000:8000 \
  --network host \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -v /mnt/shared:/mnt/shared \
  -e TILELANG_CLEANUP_TEMP_FILES=1 \
  -e VLLM_DISABLE_COMPILE_CACHE=1 \
  -e VLLM_ENGINE_READY_TIMEOUT_S=3600 \
  -e VLLM_RPC_TIMEOUT=600000 \
  -e VLLM_LOG_STATS_INTERVAL=1 \
  -e VLLM_MOONCAKE_BOOTSTRAP_PORT=8998 \
  -e CUDA_VISIBLE_DEVICES=0,1,2,3 \
  vllm/vllm-openai:v0.25.0 \
  deepseek-ai/DeepSeek-V4-Flash \
  --trust-remote-code \
  --kv-cache-dtype fp8 \
  --block-size 256 \
  --port 8000 \
  --data-parallel-size 4 \
  --enable-expert-parallel \
  --tokenizer-mode deepseek_v4 \
  --reasoning-parser deepseek_v4 \
  --max-model-len auto \
  --max-num-batched-tokens 16384 \
  --max-num-seqs 8 \
  --enforce-eager \
  --no-disable-hybrid-kv-cache-manager \
  --disable-uvicorn-access-log \
  --kv-transfer-config '{"kv_connector":"MooncakeConnector","kv_role":"kv_both","kv_load_failure_policy":"fail","kv_buffer_device":"cuda","kv_connector_extra_config":{"enforce_handshake_compat":false,"mooncake_protocol":"rdma"}}'

Decode (GPUs 4–7, port 8001):

docker run --gpus all \
  --privileged --ipc=host -p 8001:8001 \
  --network host \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -v /mnt/shared:/mnt/shared \
  -e TILELANG_CLEANUP_TEMP_FILES=1 \
  -e VLLM_DISABLE_COMPILE_CACHE=1 \
  -e VLLM_ENGINE_READY_TIMEOUT_S=3600 \
  -e VLLM_RPC_TIMEOUT=600000 \
  -e VLLM_LOG_STATS_INTERVAL=1 \
  -e VLLM_MOONCAKE_BOOTSTRAP_PORT=9889 \
  -e CUDA_VISIBLE_DEVICES=4,5,6,7 \
  vllm/vllm-openai:v0.25.0 \
  deepseek-ai/DeepSeek-V4-Flash \
  --trust-remote-code \
  --kv-cache-dtype fp8 \
  --block-size 256 \
  --port 8001 \
  --data-parallel-size 4 \
  --enable-expert-parallel \
  --tokenizer-mode deepseek_v4 \
  --reasoning-parser deepseek_v4 \
  --max-model-len auto \
  --max-num-seqs 512 \
  --max-num-batched-tokens 512 \
  --compilation-config '{"mode":0,"cudagraph_mode":"FULL_DECODE_ONLY","max_cudagraph_capture_size":512,"compile_ranges_endpoints":[512]}' \
  --no-disable-hybrid-kv-cache-manager \
  --disable-uvicorn-access-log \
  --kv-transfer-config '{"kv_connector":"MooncakeConnector","kv_role":"kv_both","kv_load_failure_policy":"fail","kv_buffer_device":"cuda","kv_connector_extra_config":{"enforce_handshake_compat":false,"mooncake_protocol":"rdma"}}'

Router:

pip install vllm-router

vllm-router --policy round_robin \
  --vllm-pd-disaggregation \
  --prefill http://localhost:8000 \
  --decode http://localhost:8001 \
  --host 127.0.0.1 \
  --port 30000 \
  --intra-node-data-parallel-size 4 \
  --kv-connector mooncake

H200 Single-Node PD (Nixl)

Single-host disaggregated serving: 4 prefill GPUs + 4 decode GPUs on one 8-GPU H200 node, using NixlConnector for KV cache transfer.

Prefill (GPUs 0–3, port 8000):

docker run --gpus all \
  --privileged --ipc=host -p 8000:8000 \
  --network host \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -v /mnt/shared:/mnt/shared \
  -e TILELANG_CLEANUP_TEMP_FILES=1 \
  -e VLLM_DISABLE_COMPILE_CACHE=1 \
  -e VLLM_ENGINE_READY_TIMEOUT_S=3600 \
  -e VLLM_RPC_TIMEOUT=600000 \
  -e VLLM_LOG_STATS_INTERVAL=1 \
  -e VLLM_NIXL_SIDE_CHANNEL_PORT=5557 \
  -e CUDA_VISIBLE_DEVICES=0,1,2,3 \
  vllm/vllm-openai:v0.25.0 \
  deepseek-ai/DeepSeek-V4-Flash \
  --trust-remote-code \
  --kv-cache-dtype fp8 \
  --block-size 256 \
  --port 8000 \
  --data-parallel-size 4 \
  --enable-expert-parallel \
  --tokenizer-mode deepseek_v4 \
  --reasoning-parser deepseek_v4 \
  --max-model-len auto \
  --max-num-batched-tokens 16384 \
  --max-num-seqs 8 \
  --enforce-eager \
  --no-disable-hybrid-kv-cache-manager \
  --disable-uvicorn-access-log \
  --kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_both"}'

Decode (GPUs 4–7, port 8001):

docker run --gpus all \
  --privileged --ipc=host -p 8001:8001 \
  --network host \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -v /mnt/shared:/mnt/shared \
  -e TILELANG_CLEANUP_TEMP_FILES=1 \
  -e VLLM_DISABLE_COMPILE_CACHE=1 \
  -e VLLM_ENGINE_READY_TIMEOUT_S=3600 \
  -e VLLM_RPC_TIMEOUT=600000 \
  -e VLLM_LOG_STATS_INTERVAL=1 \
  -e VLLM_NIXL_SIDE_CHANNEL_PORT=5558 \
  -e CUDA_VISIBLE_DEVICES=4,5,6,7 \
  vllm/vllm-openai:v0.25.0 \
  deepseek-ai/DeepSeek-V4-Flash \
  --trust-remote-code \
  --kv-cache-dtype fp8 \
  --block-size 256 \
  --port 8001 \
  --data-parallel-size 4 \
  --enable-expert-parallel \
  --tokenizer-mode deepseek_v4 \
  --reasoning-parser deepseek_v4 \
  --max-model-len auto \
  --max-num-seqs 512 \
  --max-num-batched-tokens 512 \
  --compilation-config '{"mode":0,"cudagraph_mode":"FULL_DECODE_ONLY","max_cudagraph_capture_size":512,"compile_ranges_endpoints":[512]}' \
  --no-disable-hybrid-kv-cache-manager \
  --disable-uvicorn-access-log \
  --kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_both"}'

Router:

pip install vllm-router

vllm-router --policy round_robin \
  --vllm-pd-disaggregation \
  --prefill http://localhost:8000 \
  --decode http://localhost:8001 \
  --host 127.0.0.1 \
  --port 30000 \
  --intra-node-data-parallel-size 4 \
  --kv-connector nixl