deepseek-ai/DeepSeek-V4-Pro

Overview

DeepSeek-V4-Pro is the flagship of the V4 preview family: a 1.6T-total / 49B-active Mixture-of-Experts model. It pairs a hybrid attention stack — Compressed Sparse Attention (CSA) + Heavily Compressed Attention (HCA) — with Manifold-Constrained Hyper-Connections (mHC) to reach 27% of V3.2's per-token inference FLOPs and 10% of V3.2's KV cache at 1M context. Pre-trained on 32T+ tokens with the Muon optimizer for faster convergence; post-training is a two-stage pipeline (domain-specific expert cultivation + unified consolidation via on-policy distillation).

Checkpoint is FP4+FP8 mixed: MoE expert weights are stored in FP4 while the remaining (attention / norm / router) params stay in FP8.

An NVFP4 variant (nvidia/DeepSeek-V4-Pro-NVFP4) is also available — NVIDIA modelopt re-quantizes the MoE experts to standard NVFP4 while attention, shared experts, router head, and MTP stay FP8. Pick it from the Variant row; it runs on Blackwell GPUs with the FP4 indexer cache. Unlike the native FP8 checkpoint, the NVFP4 experts don't support the deep_gemm_mega_moe MoE kernel (FP8-only), so it runs on the default MoE backend.

Checkpoints

Four checkpoints are on the Variant row. They differ in which weights you serve and which draft module ships with them — the speculative method itself is picked on the Spec Decoding row.

Variant	Repo	Draft	Notes
FP8 (0813) (default)	deepseek-ai/DeepSeek-V4-Pro-0813	DSpark	Official release, preview structure + DSpark
FP8 (Preview)	deepseek-ai/DeepSeek-V4-Pro	MTP	Preview FP4+FP8 mixed weights
NVFP4	nvidia/DeepSeek-V4-Pro-NVFP4	MTP	modelopt re-quant, Blackwell
DSpark	deepseek-ai/DeepSeek-V4-Pro-DSpark	DSpark	Preview weights + fused DSpark module

0813 is the official DeepSeek-V4-Pro release and the default here, superseding the preview with substantially stronger agentic capability — 87.9 on Terminal Bench 2.1, 62.7 on DeepSWE and 42.7 on HLE (60.0 with tools), versus 72.1, 12.8 and 37.7 for the preview. It is built on the preview model structure with a DSpark speculative decoding module attached. The preview weights remain available as the FP8 (Preview) variant for reproducing earlier results.

DSpark is not new weights — it is the preview checkpoint with a speculative decoding module attached (see DeepSpec). Both fused checkpoints add a dspark_* block to config.json; the preview and NVFP4 checkpoints have no such block, which is why the DSpark method is offered only on the two variants that carry the draft. Selecting either one auto-enables Spec Decoding with method: dspark, emitting:

--speculative-config '{"method":"dspark","num_speculative_tokens":7,"draft_sample_method":"probabilistic"}'

The fused checkpoints are ~893 GB on disk versus ~865 GB for the preview — the draft module is the difference. Both require vLLM 0.25.0, above this recipe's 0.20.0 baseline; the Install block bumps automatically when either is selected.

Reasoning modes

The chat template exposes three reasoning-effort modes:

• Non-think — fast, intuitive responses.
• Think High — explicit chain-of-thought for complex problem-solving and planning.
• Think Max — maximum reasoning effort; requires --max-model-len >= 393216 (384K tokens) to avoid truncation.

Recommended sampling: temperature = 1.0, top_p = 1.0.

On the 0813 variant the effort levels are named low / high / max, and DeepSeek recommends top_p = 0.95 for agentic scenarios (1.0 otherwise) with temperature = 1.0. Allow up to 384K output tokens at the high and max levels.

Note that the 0813 release ships no Jinja chat template — the repo provides an encoding/ folder with encode_messages / parse_message_from_completion_text helpers instead. Serving through vLLM with --tokenizer-mode deepseek_v4 (the Tool Calling pill) applies the built-in DeepSeek-V4 encoding, so the OpenAI- compatible endpoint works without the helper scripts.

OpenAI Client Example

For DeepSeek-V4, keep reasoning controls in chat_template_kwargs, as it exposes a custom Think Max mode via "reasoning_effort": "max".

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
model = "deepseek-ai/DeepSeek-V4-Pro-0813"
messages = [{"role": "user", "content": "What is 17*19? Return only the final integer."}]

# Non-think
resp = client.chat.completions.create(
    model=model,
    messages=messages,
)

# Think High
resp = client.chat.completions.create(
    model=model,
    messages=messages,
    extra_body={
        "chat_template_kwargs": {
            "thinking": True,
            "reasoning_effort": "high",
        },
    },
)

# Think Max
resp = client.chat.completions.create(
    model=model,
    messages=messages,
    extra_body={
        "chat_template_kwargs": {
            "thinking": True,
            "reasoning_effort": "max",
        },
    },
)

Recommended deployments

• B300 (8× GPU): single-node DP + EP with --data-parallel-size 8.
• H200 (8× GPU): DP + EP with --data-parallel-size 8. Context is capped at 800K tokens (--max-model-len 800000) to leave KV headroom with dense params replicated across ranks — applies to both single-node and multi-node H200.
• MI355X (8× GPU): validated with ROCm + AITER (VLLM_ROCM_USE_AITER=1), --gpu-memory-utilization 0.9, --max-num-seqs 128, --max-num-batched-tokens 8192, and --distributed-executor-backend mp.
• GB200 NVL4 (4× GPU per tray): the ~960 GB mixed-precision checkpoint does not fit on one tray; run multi-node DP + EP across 2 trays (8 GPUs total) with --data-parallel-size 8. Pick the "Multi-Node" tab and set nodes to 2.

MI355X (8×288GB)

export VLLM_ROCM_USE_AITER=1

vllm serve deepseek-ai/DeepSeek-V4-Pro \
  --host localhost \
  --port 8001 \
  --dtype auto \
  --kv-cache-dtype fp8 \
  --tensor-parallel-size 8 \
  --max-num-seqs 512 \
  --max-num-batched-tokens 8192 \
  --distributed-executor-backend mp \
  --trust-remote-code \
  --gpu-memory-utilization 0.9 \
  --tokenizer-mode deepseek_v4 \
  --reasoning-parser deepseek_v4 \
  --tool-call-parser deepseek_v4 \
  --enable-auto-tool-choice \
  --compilation-config '{"mode": 3, "cudagraph_mode": "FULL_DECODE_ONLY"}'

MI355X is validated on GSM8K dataset:

Launch command

MODEL=deepseek-ai/DeepSeek-V4-Pro
lm_eval --model local-completions \
  --model_args model=$MODEL,base_url=http://0.0.0.0:8001/v1/completions,num_concurrent=128,max_retries=10,max_gen_toks=2048,timeout=60000 \
  --batch_size auto \
  --tasks gsm8k \
  --num_fewshot 8 \
  --output_path . 2>&1 | tee -a eval.log

Reported result

local-completions ({'model': 'deepseek-ai/DeepSeek-V4-Pro', 'base_url': 'http://0.0.0.0:8001/v1/completions', 'num_concurrent': 128, 'max_retries': 10, 'max_gen_toks': 2048, 'timeout': 60000}), gen_kwargs: ({}), limit: None, num_fewshot: 8, batch_size: auto
|Tasks|Version|     Filter     |n-shot|  Metric   |   |Value |   |Stderr|
|-----|------:|----------------|-----:|-----------|---|-----:|---|-----:|
|gsm8k|      3|flexible-extract|     8|exact_match|↑  |0.9538|±  |0.0058|
|     |       |strict-match    |     8|exact_match|↑  |0.9545|±  |0.0057|

KV Cache Offloading

Agentic and multi-turn workloads reuse long prefixes whose KV state can exceed on-GPU capacity. The KV Offload row attaches a host-DRAM KV tier to any serving strategy — pick one of three connectors:

• Simple — SimpleCPUOffloadConnector: spills KV blocks to a per-rank region of CPU DRAM on each node. The simplest way to extend effective KV capacity for a single instance.
• Mooncake — MooncakeStoreConnector: pools CPU DRAM into a distributed shared KV store — either embedded (each rank's vLLM worker runs a Mooncake client that donates a DRAM segment) or standalone (a per-node mooncake_store_service owns the node's DRAM and contributes it to the pool, decoupling cache lifetime from the engine). A cluster-wide mooncake_master coordinates the pool; across 2+ instances, GPU workers share it for cross-instance prefix reuse. Install from the extra-install block above; see the Mooncake docs.
• LMCache — LMCacheMPConnector: a node-local KV pool served by a companion lmcache server process, launched before vllm serve. Single-node strategies only. Install from the extra-install block above; see the LMCache docs.

Agentic benchmark reproduction (InferenceX MI355X sweep)

The command above is the default serving configuration. The SemiAnalysis InferenceX MI355X benchmark lane for this checkpoint is a separate high-concurrency benchmark configuration — not the recipe default. To reproduce that sweep, start from the default command above and apply these deltas:

Applied to both arms:

• Enable VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4 (previously commented out).
• Enable VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS=1. DSv4-Pro is a mixed checkpoint (MXFP4 routed experts, FP8 shared expert) and vLLM gates the fused shared-expert path on this flag, which defaults to off, so the checked-in recipe was not running the configuration the validated manual runs used. The flag is mutually exclusive with expert parallelism inside vLLM, which is consistent with both arms running EP 1.
• Raise --gpu-memory-utilization from 0.8 to 0.86.
• Pin --max-num-batched-tokens 8192 instead of the nightly default of 16384. On the TP8 initialization check this raised GPU KV-cache capacity from 4,730,981 to 8,524,228 tokens and reduced peak activation memory from 11.44 GiB to 8.9 GiB.

Applied to the DP-attention arm only:

• Cap --max-num-seqs at CONC rather than 2*CONC. The limit is per scheduler and DP-attention runs one scheduler per rank. The pure TP8 arm keeps the existing 2*CONC headroom for AgentX subagent fan-out.
• Add --prefill-schedule-interval 8 and --long-prefill-token-threshold 16384.

With --max-num-batched-tokens 8192 now pinned for both arms, --long-prefill-token-threshold 16384 sits above the token budget and therefore never binds. It is inert rather than harmful, and the DP-attention point was measured with it present.

--compilation-config '{"mode":3,"cudagraph_mode":"FULL_AND_PIECEWISE"}' is unchanged; it was already in the checked-in recipe.

Export the AITER and INT4 quantized all-reduce environment variables before launch:

export VLLM_ROCM_USE_AITER=1
export VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4
export VLLM_ROCM_USE_AITER_MOE=1
export VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS=1
export VLLM_ENGINE_READY_TIMEOUT_S=10800
export VLLM_PREFIX_CACHE_RETENTION_INTERVAL=32768

vllm serve "$MODEL_PATH" --served-model-name "$MODEL" \
  --host 0.0.0.0 \
  --port "$VLLM_BACKEND_PORT" \
  --trust-remote-code \
  --async-scheduling \
  --distributed-executor-backend mp \
  --kv-cache-dtype fp8 \
  --max-num-batched-tokens 8192 \
  --gpu-memory-utilization 0.86 \
  --moe-backend aiter \
  --compilation-config '{"mode":3,"cudagraph_mode":"FULL_AND_PIECEWISE"}' \
  --speculative-config '{"method": "mtp", "num_speculative_tokens": 3, "rejection_sample_method": "synthetic", "synthetic_acceptance_length": 2.49}' \
  --tokenizer-mode deepseek_v4 \
  --tool-call-parser deepseek_v4 \
  --reasoning-parser deepseek_v4 \
  --enable-auto-tool-choice \
  --enable-prefix-caching \
  --no-disable-hybrid-kv-cache-manager \
  --max-num-seqs "8"