deepseek-ai/DeepSeek-V4-Pro

Overview

DeepSeek-V4-Pro is the flagship of the V4 preview family: a 1.6T-total / 49B-active Mixture-of-Experts model. It pairs a hybrid attention stack — Compressed Sparse Attention (CSA) + Heavily Compressed Attention (HCA) — with Manifold-Constrained Hyper-Connections (mHC) to reach 27% of V3.2's per-token inference FLOPs and 10% of V3.2's KV cache at 1M context. Pre-trained on 32T+ tokens with the Muon optimizer for faster convergence; post-training is a two-stage pipeline (domain-specific expert cultivation + unified consolidation via on-policy distillation).

Checkpoint is FP4+FP8 mixed: MoE expert weights are stored in FP4 while the remaining (attention / norm / router) params stay in FP8.

An NVFP4 variant (nvidia/DeepSeek-V4-Pro-NVFP4) is also available — NVIDIA modelopt re-quantizes the MoE experts to standard NVFP4 while attention, shared experts, router head, and MTP stay FP8. Pick it from the Variant row; it runs on Blackwell GPUs with the FP4 indexer cache.

Checkpoints

Four checkpoints are on the Variant row. They differ in which weights you serve and which draft module ships with them — the speculative method itself is picked on the Spec Decoding row.

Variant	Repo	Draft	Notes
FP8 (0813) (default)	deepseek-ai/DeepSeek-V4-Pro-0813	DSpark	Official release, preview structure + DSpark
FP8 (Preview)	deepseek-ai/DeepSeek-V4-Pro	MTP	Preview FP4+FP8 mixed weights
NVFP4	nvidia/DeepSeek-V4-Pro-NVFP4	MTP	modelopt re-quant, Blackwell
DSpark	deepseek-ai/DeepSeek-V4-Pro-DSpark	DSpark	Preview weights + fused DSpark module

0813 is the official DeepSeek-V4-Pro release and the default here, superseding the preview with substantially stronger agentic capability — 87.9 on Terminal Bench 2.1, 62.7 on DeepSWE and 42.7 on HLE (60.0 with tools), versus 72.1, 12.8 and 37.7 for the preview. It is built on the preview model structure with a DSpark speculative decoding module attached. The preview weights remain available as the FP8 (Preview) variant for reproducing earlier results.

DSpark is not new weights — it is the preview checkpoint with a speculative decoding module attached (see DeepSpec). Both fused checkpoints add a dspark_* block to config.json; the preview and NVFP4 checkpoints have no such block, which is why the DSpark method is offered only on the two variants that carry the draft. Selecting either one auto-enables Spec Decoding with method: dspark, emitting:

--speculative-config '{"method":"dspark","num_speculative_tokens":7,"draft_sample_method":"probabilistic"}'

The fused checkpoints are ~893 GB on disk versus ~865 GB for the preview — the draft module is the difference. Both require vLLM 0.25.0, above this recipe's 0.20.0 baseline; the Install block bumps automatically when either is selected.

Reasoning modes

The chat template exposes three reasoning-effort modes:

• Non-think — fast, intuitive responses.
• Think High — explicit chain-of-thought for complex problem-solving and planning.
• Think Max — maximum reasoning effort; requires --max-model-len >= 393216 (384K tokens) to avoid truncation.

Recommended sampling: temperature = 1.0, top_p = 1.0.

On the 0813 variant the effort levels are named low / high / max, and DeepSeek recommends top_p = 0.95 for agentic scenarios (1.0 otherwise) with temperature = 1.0. Allow up to 384K output tokens at the high and max levels.

Note that the 0813 release ships no Jinja chat template — the repo provides an encoding/ folder with encode_messages / parse_message_from_completion_text helpers instead. Serving through vLLM with --tokenizer-mode deepseek_v4 (the Tool Calling pill) applies the built-in DeepSeek-V4 encoding, so the OpenAI- compatible endpoint works without the helper scripts.

OpenAI Client Example

For DeepSeek-V4, keep reasoning controls in chat_template_kwargs, as it exposes a custom Think Max mode via "reasoning_effort": "max".

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
model = "deepseek-ai/DeepSeek-V4-Pro-0813"
messages = [{"role": "user", "content": "What is 17*19? Return only the final integer."}]

# Non-think
resp = client.chat.completions.create(
    model=model,
    messages=messages,
)

# Think High
resp = client.chat.completions.create(
    model=model,
    messages=messages,
    extra_body={
        "chat_template_kwargs": {
            "thinking": True,
            "reasoning_effort": "high",
        },
    },
)

# Think Max
resp = client.chat.completions.create(
    model=model,
    messages=messages,
    extra_body={
        "chat_template_kwargs": {
            "thinking": True,
            "reasoning_effort": "max",
        },
    },
)

Recommended deployments

• B300 (8× GPU): single-node DP + EP with --data-parallel-size 8.
• H200 (8× GPU): DP + EP with --data-parallel-size 8. Context is capped at 800K tokens (--max-model-len 800000) to leave KV headroom with dense params replicated across ranks — applies to both single-node and multi-node H200.
• MI355X (8× GPU): validated with ROCm + AITER (VLLM_ROCM_USE_AITER=1), --gpu-memory-utilization 0.9, --max-num-seqs 128, --max-num-batched-tokens 8192, and --distributed-executor-backend mp.
• GB200 NVL4 (4× GPU per tray): the ~960 GB mixed-precision checkpoint does not fit on one tray; run multi-node DP + EP across 2 trays (8 GPUs total) with --data-parallel-size 8. Pick the "Multi-Node" tab and set nodes to 2.

MI355X (8×288GB)

export VLLM_ROCM_USE_AITER=1

vllm serve deepseek-ai/DeepSeek-V4-Pro \
  --host localhost \
  --port 8001 \
  --dtype auto \
  --kv-cache-dtype fp8 \
  --tensor-parallel-size 8 \
  --max-num-seqs 512 \
  --max-num-batched-tokens 8192 \
  --distributed-executor-backend mp \
  --trust-remote-code \
  --gpu-memory-utilization 0.9 \
  --tokenizer-mode deepseek_v4 \
  --reasoning-parser deepseek_v4 \
  --tool-call-parser deepseek_v4 \
  --enable-auto-tool-choice \
  --compilation-config '{"mode": 3, "cudagraph_mode": "FULL_DECODE_ONLY"}'

MI355X is validated on GSM8K dataset:

Launch command

MODEL=deepseek-ai/DeepSeek-V4-Pro
lm_eval --model local-completions \
  --model_args model=$MODEL,base_url=http://0.0.0.0:8001/v1/completions,num_concurrent=128,max_retries=10,max_gen_toks=2048,timeout=60000 \
  --batch_size auto \
  --tasks gsm8k \
  --num_fewshot 8 \
  --output_path . 2>&1 | tee -a eval.log

Reported result

local-completions ({'model': 'deepseek-ai/DeepSeek-V4-Pro', 'base_url': 'http://0.0.0.0:8001/v1/completions', 'num_concurrent': 128, 'max_retries': 10, 'max_gen_toks': 2048, 'timeout': 60000}), gen_kwargs: ({}), limit: None, num_fewshot: 8, batch_size: auto
|Tasks|Version|     Filter     |n-shot|  Metric   |   |Value |   |Stderr|
|-----|------:|----------------|-----:|-----------|---|-----:|---|-----:|
|gsm8k|      3|flexible-extract|     8|exact_match|↑  |0.9538|±  |0.0058|
|     |       |strict-match    |     8|exact_match|↑  |0.9545|±  |0.0057|

KV Cache Offloading

Agentic and multi-turn workloads reuse long prefixes whose KV state can exceed on-GPU capacity. The KV Offload row attaches a host-DRAM KV tier to any serving strategy — pick one of three connectors:

• Simple — SimpleCPUOffloadConnector: spills KV blocks to a per-rank region of CPU DRAM on each node. The simplest way to extend effective KV capacity for a single instance.
• Mooncake — MooncakeStoreConnector: pools CPU DRAM into a distributed shared KV store — either embedded (each rank's vLLM worker runs a Mooncake client that donates a DRAM segment) or standalone (a per-node mooncake_store_service owns the node's DRAM and contributes it to the pool, decoupling cache lifetime from the engine). A cluster-wide mooncake_master coordinates the pool; across 2+ instances, GPU workers share it for cross-instance prefix reuse. Install from the extra-install block above; see the Mooncake docs.
• LMCache — LMCacheMPConnector: a node-local KV pool served by a companion lmcache server process, launched before vllm serve. Single-node strategies only. Install from the extra-install block above; see the LMCache docs.

Agentic benchmark reproduction (InferenceX MI355X sweep)

The command above is the default serving configuration. The SemiAnalysis InferenceX MI355X benchmark lane for this checkpoint is a separate high-concurrency benchmark configuration — not the recipe default. To reproduce that sweep, start from the default command above and apply these deltas:

Applied to both arms:

• Enable VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4 (previously commented out).
• Enable VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS=1. DSv4-Pro is a mixed checkpoint (MXFP4 routed experts, FP8 shared expert) and vLLM gates the fused shared-expert path on this flag, which defaults to off, so the checked-in recipe was not running the configuration the validated manual runs used. The flag is mutually exclusive with expert parallelism inside vLLM, which is consistent with both arms running EP 1.
• Raise --gpu-memory-utilization from 0.8 to 0.86.
• Pin --max-num-batched-tokens 8192 instead of the nightly default of 16384. On the TP8 initialization check this raised GPU KV-cache capacity from 4,730,981 to 8,524,228 tokens and reduced peak activation memory from 11.44 GiB to 8.9 GiB.

Applied to the DP-attention arm only:

• Cap --max-num-seqs at CONC rather than 2*CONC. The limit is per scheduler and DP-attention runs one scheduler per rank. The pure TP8 arm keeps the existing 2*CONC headroom for AgentX subagent fan-out.
• Add --prefill-schedule-interval 8 and --long-prefill-token-threshold 16384.

With --max-num-batched-tokens 8192 now pinned for both arms, --long-prefill-token-threshold 16384 sits above the token budget and therefore never binds. It is inert rather than harmful, and the DP-attention point was measured with it present.

--compilation-config '{"mode":3,"cudagraph_mode":"FULL_AND_PIECEWISE"}' is unchanged; it was already in the checked-in recipe.

Export the AITER and INT4 quantized all-reduce environment variables before launch:

export VLLM_ROCM_USE_AITER=1
export VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4
export VLLM_ROCM_USE_AITER_MOE=1
export VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS=1
export VLLM_ENGINE_READY_TIMEOUT_S=10800
export VLLM_PREFIX_CACHE_RETENTION_INTERVAL=32768

vllm serve "$MODEL_PATH" --served-model-name "$MODEL" \
  --host 0.0.0.0 \
  --port "$VLLM_BACKEND_PORT" \
  --trust-remote-code \
  --async-scheduling \
  --distributed-executor-backend mp \
  --kv-cache-dtype fp8 \
  --max-num-batched-tokens 8192 \
  --gpu-memory-utilization 0.86 \
  --moe-backend aiter \
  --compilation-config '{"mode":3,"cudagraph_mode":"FULL_AND_PIECEWISE"}' \
  --speculative-config '{"method": "mtp", "num_speculative_tokens": 3, "rejection_sample_method": "synthetic", "synthetic_acceptance_length": 2.49}' \
  --tokenizer-mode deepseek_v4 \
  --tool-call-parser deepseek_v4 \
  --reasoning-parser deepseek_v4 \
  --enable-auto-tool-choice \
  --enable-prefix-caching \
  --no-disable-hybrid-kv-cache-manager \
  --max-num-seqs "8"