{ "hf_id": "deepseek-ai/DeepSeek-V4-Pro", "meta": { "title": "DeepSeek-V4-Pro", "slug": "deepseek-v4-pro", "provider": "DeepSeek", "description": "DeepSeek V4 flagship MoE (1.6T total / 49B active) with hybrid CSA+HCA attention, manifold-constrained hyper-connections, Muon-trained on 32T+ tokens, and three-tier reasoning.", "date_added": "2026-04-24", "date_updated": "2026-10-28", "tasks": [ "text" ], "performance_headline": "Frontier 1.6T/49B reasoning MoE with native FP4+FP8 weights, MTP and DSpark speculative decoding, and 1M-token context", "related_recipes": [], "hardware": { "h200": "verified", "b200": "verified", "gb200": "verified", "b300": "verified", "gb300": "verified", "mi300x": "unsupported", "mi325x": "unsupported", "mi355x": "verified" } }, "recommended_command": { "hardware": "h200", "strategy": "single_node_tep", "variant": "default", "node_count": 1, "deploy_type": "single_node", "env": { "VLLM_USE_RUST_FRONTEND": "1" }, "docker_image": "vllm/vllm-openai:latest", "command": "vllm serve deepseek-ai/DeepSeek-V4-Pro-0813 \\\n --trust-remote-code \\\n --kv-cache-dtype fp8 \\\n --block-size 256 \\\n --enable-expert-parallel \\\n --tensor-parallel-size 8 \\\n --max-model-len 800000 \\\n --gpu-memory-utilization 0.95 \\\n --max-num-seqs 512 \\\n --disable-flashinfer-autotune \\\n --compilation-config '{\"mode\": 0, \"cudagraph_mode\": \"FULL_DECODE_ONLY\"}' \\\n --tokenizer-mode deepseek_v4 \\\n --tool-call-parser deepseek_v4 \\\n --enable-auto-tool-choice \\\n --reasoning-parser deepseek_v4 \\\n --speculative-config '{\"method\":\"dspark\",\"num_speculative_tokens\":7,\"draft_sample_method\":\"probabilistic\"}'", "argv": [ "vllm", "serve", "deepseek-ai/DeepSeek-V4-Pro-0813", "--trust-remote-code", "--kv-cache-dtype", "fp8", "--block-size", "256", "--enable-expert-parallel", "--tensor-parallel-size", "8", "--max-model-len", "800000", "--gpu-memory-utilization", "0.95", "--max-num-seqs", "512", "--disable-flashinfer-autotune", "--compilation-config", "{\"mode\": 0, \"cudagraph_mode\": \"FULL_DECODE_ONLY\"}", "--tokenizer-mode", "deepseek_v4", "--tool-call-parser", "deepseek_v4", "--enable-auto-tool-choice", "--reasoning-parser", "deepseek_v4", "--speculative-config", "{\"method\":\"dspark\",\"num_speculative_tokens\":7,\"draft_sample_method\":\"probabilistic\"}" ], "docker_command": "docker run --gpus all \\\n --privileged --ipc=host -p 8000:8000 \\\n -v ~/.cache/huggingface:/root/.cache/huggingface \\\n -e VLLM_USE_RUST_FRONTEND=1 \\\n vllm/vllm-openai:latest deepseek-ai/DeepSeek-V4-Pro-0813 \\\n --trust-remote-code \\\n --kv-cache-dtype fp8 \\\n --block-size 256 \\\n --enable-expert-parallel \\\n --tensor-parallel-size 8 \\\n --max-model-len 800000 \\\n --gpu-memory-utilization 0.95 \\\n --max-num-seqs 512 \\\n --disable-flashinfer-autotune \\\n --compilation-config '{\"mode\": 0, \"cudagraph_mode\": \"FULL_DECODE_ONLY\"}' \\\n --tokenizer-mode deepseek_v4 \\\n --tool-call-parser deepseek_v4 \\\n --enable-auto-tool-choice \\\n --reasoning-parser deepseek_v4 \\\n --speculative-config '{\"method\":\"dspark\",\"num_speculative_tokens\":7,\"draft_sample_method\":\"probabilistic\"}'", "docker_argv": [ "docker", "run", "--gpus", "all", "--privileged", "--ipc=host", "-p", "8000:8000", "-v", "~/.cache/huggingface:/root/.cache/huggingface", "-e", "VLLM_USE_RUST_FRONTEND=1", "vllm/vllm-openai:latest", "deepseek-ai/DeepSeek-V4-Pro-0813", "--trust-remote-code", "--kv-cache-dtype", "fp8", "--block-size", "256", "--enable-expert-parallel", "--tensor-parallel-size", "8", "--max-model-len", "800000", "--gpu-memory-utilization", "0.95", "--max-num-seqs", "512", "--disable-flashinfer-autotune", "--compilation-config", "{\"mode\": 0, \"cudagraph_mode\": \"FULL_DECODE_ONLY\"}", "--tokenizer-mode", "deepseek_v4", "--tool-call-parser", "deepseek_v4", "--enable-auto-tool-choice", "--reasoning-parser", "deepseek_v4", "--speculative-config", "{\"method\":\"dspark\",\"num_speculative_tokens\":7,\"draft_sample_method\":\"probabilistic\"}" ], "strategy_spec": { "name": "single_node_tep", "deploy_type": "single_node", "display_name": "Tensor + Expert Parallel", "short_label": "TEP", "orientation": "balanced", "description": "Single-node TEP. TP splits dense layers and EP splits expert layers across local GPUs. TP must be set to GPU count to avoid OOM from replicated dense layers. For MoE models only.\n", "hardware_match": { "min_gpus": 2, "max_gpus": 8, "multi_node": false, "architecture": "moe" }, "vllm_args": [ "--enable-expert-parallel" ], "parallel_flag": "--tensor-parallel-size" }, "hardware_profile": { "brand": "NVIDIA", "generation": "hopper", "compute_arch": "sm90", "display_name": "H200", "description": "NVIDIA H200 SXM 141 GB HBM3e · 8-GPU HGX node", "gpu_count": 8, "vram_gb": 1128, "multi_node": false }, "alternatives": { "single_node_tp": "/deepseek-ai/DeepSeek-V4-Pro/strategies/single_node_tp.json", "single_node_dep": "/deepseek-ai/DeepSeek-V4-Pro/strategies/single_node_dep.json", "multi_node_tep": "/deepseek-ai/DeepSeek-V4-Pro/strategies/multi_node_tep.json", "multi_node_dep": "/deepseek-ai/DeepSeek-V4-Pro/strategies/multi_node_dep.json", "pd_cluster": "/deepseek-ai/DeepSeek-V4-Pro/strategies/pd_cluster.json", "kv_store_centralized_mooncake": "/deepseek-ai/DeepSeek-V4-Pro/strategies/kv_store_centralized_mooncake.json", "kv_store_distributed_mooncake": "/deepseek-ai/DeepSeek-V4-Pro/strategies/kv_store_distributed_mooncake.json" }, "by_hardware": { "h200": "/deepseek-ai/DeepSeek-V4-Pro/hw/h200.json", "h100": "/deepseek-ai/DeepSeek-V4-Pro/hw/h100.json", "b200": "/deepseek-ai/DeepSeek-V4-Pro/hw/b200.json", "gb200": "/deepseek-ai/DeepSeek-V4-Pro/hw/gb200.json", "b300": "/deepseek-ai/DeepSeek-V4-Pro/hw/b300.json", "gb300": "/deepseek-ai/DeepSeek-V4-Pro/hw/gb300.json", "mi350x": "/deepseek-ai/DeepSeek-V4-Pro/hw/mi350x.json", "mi355x": "/deepseek-ai/DeepSeek-V4-Pro/hw/mi355x.json" } }, "model": { "model_id": "deepseek-ai/DeepSeek-V4-Pro", "min_vllm_version": "0.20.0", "default_frontend": "rust", "docker_image": { "amd": "vllm/vllm-openai-rocm:nightly" }, "architecture": "moe", "parameter_count": "1600B", "active_parameters": "49B", "context_length": 1048576, "flashinfer_autotune": true, "base_args": [ "--trust-remote-code", "--kv-cache-dtype", "fp8", "--block-size", "256" ], "install": { "pip": { "command": "uv venv\nsource .venv/bin/activate\nuv pip install -U vllm --torch-backend auto" }, "docker": { "command": "docker pull vllm/vllm-openai:latest" } } }, "features": { "tool_calling": { "description": "Enable tool calling with DeepSeek V4 chat template support.", "args": [ "--tokenizer-mode", "deepseek_v4", "--tool-call-parser", "deepseek_v4", "--enable-auto-tool-choice" ] }, "reasoning": { "description": "Enable reasoning/thinking mode with the DeepSeek V4 reasoning parser.", "args": [ "--reasoning-parser", "deepseek_v4" ] }, "spec_decoding": { "description": "Speculative decoding — pick a drafting method.", "default_mode": "mtp", "modes": { "mtp": { "label": "MTP", "description": "Built-in Multi-Token Prediction, 2 draft tokens.", "variants": [ "fp8", "nvfp4", "redhat_nvfp4" ], "args": [ "--speculative-config", "{\"method\":\"mtp\",\"num_speculative_tokens\":2}" ] }, "dspark": { "label": "DSpark", "description": "DSpark drafting, 7 draft tokens, probabilistic. Requires the 0813 or DSpark checkpoint (Variant row).", "variants": [ "default", "dspark" ], "args": [ "--speculative-config", "{\"method\":\"dspark\",\"num_speculative_tokens\":7,\"draft_sample_method\":\"probabilistic\"}" ] } } } }, "opt_in_features": [ "spec_decoding" ], "variants": { "default": { "model_id": "deepseek-ai/DeepSeek-V4-Pro-0813", "label": "FP8 (0813)", "precision": "fp8", "vram_minimum_gb": 960, "description": "Official DeepSeek-V4-Pro release (2026-08-13) superseding the preview — substantially stronger agentic performance, with the DSpark draft module attached.", "min_vllm_version": "0.25.0", "default_modes": { "spec_decoding": "dspark" }, "hardware_overrides": { "blackwell": { "extra_args": [ "--moe-backend", "deep_gemm_mega_moe" ] } }, "json": "/deepseek-ai/DeepSeek-V4-Pro-0813.json" }, "fp8": { "label": "FP8 (Preview)", "precision": "fp8", "vram_minimum_gb": 960, "description": "Native FP4+FP8 mixed checkpoint (MoE experts FP4, remaining params FP8)", "hardware_overrides": { "blackwell": { "extra_args": [ "--moe-backend", "deep_gemm_mega_moe" ] } } }, "nvfp4": { "model_id": "nvidia/DeepSeek-V4-Pro-NVFP4", "precision": "nvfp4", "vram_minimum_gb": 960, "description": "NVIDIA modelopt NVFP4 checkpoint (MoE experts NVFP4; attention, shared experts, router head, and MTP stay FP8) for Blackwell GPUs.", "hardware_overrides": { "blackwell": { "extra_args": [ "--moe-backend", "flashinfer_cutedsl" ] } }, "json": "/nvidia/DeepSeek-V4-Pro-NVFP4.json" }, "dspark": { "model_id": "deepseek-ai/DeepSeek-V4-Pro-DSpark", "label": "DSpark", "precision": "fp8", "vram_minimum_gb": 960, "description": "Preview FP4+FP8 weights with the DSpark draft module baked in — same base checkpoint as the fp8 variant plus a fused speculative decoder.", "min_vllm_version": "0.25.0", "default_modes": { "spec_decoding": "dspark" }, "hardware_overrides": { "blackwell": { "extra_args": [ "--moe-backend", "deep_gemm_mega_moe" ] } }, "json": "/deepseek-ai/DeepSeek-V4-Pro-DSpark.json" }, "redhat_nvfp4": { "model_id": "RedHatAI/DeepSeek-V4-Pro-NVFP4-FP8", "precision": "nvfp4", "label": "NVFP4+FP8 (RedHat)", "vram_minimum_gb": 1000, "description": "RedHatAI NVFP4+FP8 mixed-precision checkpoint", "json": "/RedHatAI/DeepSeek-V4-Pro-NVFP4-FP8.json" } }, "kv_offload_support": { "offloading_cpu": "verified", "offloading_fs": "verified" }, "strategy_hardware": { "single_node_dpa_tp": { "mi355x": "supported" } }, "hardware_overrides": { "hopper": { "extra_args": [ "--max-model-len", "800000", "--gpu-memory-utilization", "0.95", "--max-num-seqs", "512", "--disable-flashinfer-autotune", "--compilation-config", "{\"mode\": 0, \"cudagraph_mode\": \"FULL_DECODE_ONLY\"}" ] }, "blackwell": { "extra_args": [ "--attention_config.indexer_kv_dtype=mxfp4" ] }, "amd": { "extra_args": [ "--distributed-executor-backend", "mp", "--gpu-memory-utilization", "0.9", "--max-num-seqs", "512", "--max-num-batched-tokens", "8192", "--compilation-config", "{\"mode\": 3, \"cudagraph_mode\": \"FULL_DECODE_ONLY\"}" ], "extra_env": { "VLLM_ROCM_USE_AITER": "1" } } }, "presets": { "b200_agentx_tp": { "label": "AgentX TP8", "hardware": "b200", "strategy": "single_node_tp", "tp": 8, "variants": [ "default" ], "workload": "Agentic (InferenceX AgentX)", "description": "TP8 on one 8-GPU B200 node with DSpark (six draft tokens), FlashInfer sparse MLA and a fixed 50 GiB KV cache per GPU; no host KV offload.", "source": { "label": "InferenceX #4554", "href": "https://github.com/SemiAnalysisAI/InferenceX/pull/4554", "key": "dsv4-fp4-b200-vllm-agentic-mtp", "revision": "2c044c3" }, "image": "vllm/vllm-openai:nightly-8e2beaa3bff595577a069d49d96bde9074294ec2@sha256:3207cb99c0f59b92f7d47a74da8c6eb5467565d811a4f8d9df51df5188f59c70", "select": { "features": { "spec_decoding": "dspark" } }, "remove_args": [ "--attention_config.indexer_kv_dtype=mxfp4", "--disable-flashinfer-autotune" ], "args": [ "--no-enable-flashinfer-autotune", "--max-model-len", "1048576", "--gpu-memory-utilization", "0.90", "--max-num-batched-tokens", "16384", "--numa-bind", "--enable-cumem-allocator", "--disable-uvicorn-access-log", "--no-disable-hybrid-kv-cache-manager", "--attention-config", "{\"backend\":\"FLASHINFER_MLA_SPARSE_DSV4\",\"use_prefill_query_quantization\":true,\"indexer_kv_dtype\":\"mxfp4\"}" ], "json_args": { "--speculative-config": { "num_speculative_tokens": 6 } }, "env": { "VLLM_USE_V2_MODEL_RUNNER": "1", "VLLM_FLOAT32_MATMUL_PRECISION": "high", "TORCH_CUDA_ARCH_LIST": "10.0", "VLLM_ENGINE_READY_TIMEOUT_S": "7200", "VLLM_RPC_TIMEOUT": "600000" }, "points": { "c1": { "label": "1", "concurrency": 1, "args": [ "--max-num-seqs", "1", "--kv-cache-memory-bytes", "53687091200", "--compilation-config", "{\"cudagraph_mode\":\"FULL_AND_PIECEWISE\",\"cudagraph_capture_sizes\":[7,100,200,300,400,500]}" ], "env": { "VLLM_DSV4_MEGA_FP8_COMBINE": "1" } }, "c4": { "label": "4 (low latency)", "concurrency": 4, "image": "vllm/vllm-openai@sha256:d428631bb917a33d22308265d1b8be91522f862bf9fd133bee62dde3313664ae", "args": [ "--max-num-seqs", "4", "--kv-cache-memory-bytes", "42949672960", "--compilation-config", "{\"cudagraph_mode\":\"FULL_AND_PIECEWISE\",\"cudagraph_capture_sizes\":[7,14,21,28,100,200,300,400,500,640,800,1024,1280,1600,2048,2560,3200,4096,5120,6400,8192,10240,12800,16384],\"max_cudagraph_capture_size\":16384}", "--sparse-indexer-topk-backend", "deep_select" ], "env": { "VLLM_DSV4_MEGA_FP8_COMBINE": "1" } }, "c6": { "label": "6", "concurrency": 6, "args": [ "--max-num-seqs", "6", "--kv-cache-memory-bytes", "53687091200", "--compilation-config", "{\"cudagraph_mode\":\"FULL_AND_PIECEWISE\",\"cudagraph_capture_sizes\":[7,14,21,28,35,42,100,200,300,400,500]}" ], "env": { "VLLM_DSV4_MEGA_FP8_COMBINE": "1" } }, "c10": { "label": "10", "concurrency": 10, "image": "vllm/vllm-openai@sha256:d428631bb917a33d22308265d1b8be91522f862bf9fd133bee62dde3313664ae", "args": [ "--max-num-seqs", "10", "--kv-cache-memory-bytes", "53687091200", "--compilation-config", "{\"cudagraph_mode\":\"FULL_AND_PIECEWISE\",\"cudagraph_capture_sizes\":[7,14,21,28,35,42,49,56,63,70,100,200,300,400,500]}", "--prefix-cache-retention-interval", "32768", "--prefill-schedule-interval", "64" ], "json_args": { "--attention-config": { "indexer_tp_shard_min_rows": 4096 } } }, "c14": { "label": "14", "concurrency": 14, "args": [ "--max-num-seqs", "14", "--kv-cache-memory-bytes", "53687091200", "--compilation-config", "{\"cudagraph_mode\":\"FULL_AND_PIECEWISE\",\"cudagraph_capture_sizes\":[7,14,21,28,35,42,49,56,63,70,77,84,91,98,100,200,300,400,500]}" ], "env": { "VLLM_DSV4_MEGA_FP8_COMBINE": "1" } } }, "runbook": "The TP8 points of the B200 AgentX lane (`dsv4-fp4-b200-vllm-agentic-mtp` and its low-concurrency\ncompanion, [SemiAnalysisAI/InferenceX#4554](https://github.com/SemiAnalysisAI/InferenceX/pull/4554)) serve the\n0813 checkpoint with DSpark (six draft tokens) as one aggregated vLLM worker on one 8-GPU B200 node: TP8, no\nexpert parallelism, no host KV offload. Each GPU holds a fixed 50 GiB KV cache and `--max-num-seqs` equals the\nconcurrency.\n\nConcurrency 4 is the low-latency variant: a 40 GiB KV cache, the `deep_select` sparse indexer top-k backend and\ngraph capture up to 16384 tokens. Concurrency 10 sets `--prefix-cache-retention-interval 32768` and\n`--prefill-schedule-interval 64`, shards the indexer at 4096 rows (`indexer_tp_shard_min_rows`) and leaves\n`VLLM_DSV4_MEGA_FP8_COMBINE` unset. Concurrency 4 and 10 were measured on a second pinned image, shown on\nthose points.\n\nLeft out of the command: the lane also exported `VLLM_PREFIX_CACHE_RETENTION_INTERVAL`, which vLLM removed in\nv0.30.0 ([vllm#55353](https://github.com/vllm-project/vllm/pull/55353)) in favour of the flag above, so it had\nno effect on the image it ran on; and `--served-model-name`, which the builder replaces with the checkpoint ID.\nThe throughput points in InferenceX swap block verification for synthetic rejection at a committed\nacceptance length so every point is measured at the same acceptance rate; the command does not set it.\n", "json": "/deepseek-ai/DeepSeek-V4-Pro/presets/b200_agentx_tp.json" }, "b200_agentx_dep": { "label": "AgentX DEP", "hardware": "b200", "strategy": "single_node_dep", "variants": [ "default" ], "workload": "Agentic (InferenceX AgentX)", "description": "Data-parallel ranks with expert parallelism and a Simple CPU KV tier behind a consistent-hash vLLM Router: DEP8 for concurrency 64 to 192, TP4 x DP2 at 32 and TP2 x DP4 at 48 and 64.", "source": { "label": "InferenceX #4554", "href": "https://github.com/SemiAnalysisAI/InferenceX/pull/4554", "key": "dsv4-fp4-b200-vllm-agentic-mtp", "revision": "2c044c3" }, "image": "vllm/vllm-openai:nightly-8e2beaa3bff595577a069d49d96bde9074294ec2@sha256:3207cb99c0f59b92f7d47a74da8c6eb5467565d811a4f8d9df51df5188f59c70", "select": { "kv_offload": "simple", "features": { "spec_decoding": "dspark" } }, "remove_args": [ "--attention_config.indexer_kv_dtype=mxfp4" ], "args": [ "--max-model-len", "1048576", "--gpu-memory-utilization", "0.90", "--numa-bind", "--enable-cumem-allocator", "--disable-uvicorn-access-log", "--enable-ep-weight-filter", "--no-disable-hybrid-kv-cache-manager", "--no-enable-flashinfer-autotune", "--prefill-schedule-interval", "16", "--long-prefill-token-threshold", "512", "--moe-backend", "deep_gemm_mega_moe", "--attention-config", "{\"backend\":\"FLASHINFER_MLA_SPARSE_DSV4\",\"use_prefill_query_quantization\":true,\"indexer_kv_dtype\":\"mxfp4\"}" ], "json_args": { "--speculative-config": { "num_speculative_tokens": 6 }, "--kv-transfer-config": { "kv_connector_extra_config": { "cpu_bytes_to_use_per_rank": 241500000000, "enable_cross_layers_blocks": "true", "lazy_offload": false } } }, "env": { "VLLM_USE_V2_MODEL_RUNNER": "1", "VLLM_FLOAT32_MATMUL_PRECISION": "high", "TORCH_CUDA_ARCH_LIST": "10.0", "VLLM_ENGINE_READY_TIMEOUT_S": "7200", "VLLM_RPC_TIMEOUT": "600000", "PYTORCH_ALLOC_CONF": "expandable_segments:True", "PYTHONHASHSEED": "42" }, "points": { "dep8_c64": { "label": "64 (DEP8)", "concurrency": 64, "args": [ "--tensor-parallel-size", "1", "--data-parallel-size", "8", "--max-num-seqs", "16", "--max-num-batched-tokens", "8304", "--kv-cache-memory-bytes", "37580963840", "--compilation-config", "{\"cudagraph_mode\":\"FULL_DECODE_ONLY\",\"cudagraph_capture_sizes\":[7,14,21,28,35,42,49,56,63,70,77,84,91,98,105,112],\"mode\":0}" ], "env": { "VLLM_DSV4_MEGA_FP8_COMBINE": "1" } }, "dep8_c96": { "label": "96 (DEP8)", "concurrency": 96, "args": [ "--tensor-parallel-size", "1", "--data-parallel-size", "8", "--max-num-seqs", "24", "--max-num-batched-tokens", "8360", "--kv-cache-memory-bytes", "33285996544", "--compilation-config", "{\"cudagraph_mode\":\"FULL_DECODE_ONLY\",\"cudagraph_capture_sizes\":[7,14,21,28,35,42,49,56,63,70,77,84,91,98,105,112,119,126,133,140,147,154,161,168],\"mode\":0}" ], "env": { "VLLM_DSV4_MEGA_FP8_COMBINE": "1" } }, "dep8_c128": { "label": "128 (DEP8)", "concurrency": 128, "args": [ "--tensor-parallel-size", "1", "--data-parallel-size", "8", "--max-num-seqs", "32", "--max-num-batched-tokens", "8416", "--kv-cache-memory-bytes", "33285996544", "--compilation-config", "{\"cudagraph_mode\":\"FULL_DECODE_ONLY\",\"cudagraph_capture_sizes\":[7,14,21,28,35,42,49,56,63,70,77,84,91,98,105,112,119,126,133,140,147,154,161,168,175,182,189,196,203,210,217,224],\"mode\":0}" ], "env": { "VLLM_DSV4_MEGA_FP8_COMBINE": "1" } }, "dep8_c160": { "label": "160 (DEP8)", "concurrency": 160, "args": [ "--tensor-parallel-size", "1", "--data-parallel-size", "8", "--max-num-seqs", "40", "--max-num-batched-tokens", "8472", "--kv-cache-memory-bytes", "32212254720", "--compilation-config", "{\"cudagraph_mode\":\"FULL_DECODE_ONLY\",\"cudagraph_capture_sizes\":[7,14,21,28,35,42,49,56,63,70,77,84,91,98,105,112,119,126,133,140,147,154,161,168,175,182,189,196,203,210,217,224,231,238,245,252,259,266,273,280],\"mode\":0}" ], "env": { "VLLM_DSV4_MEGA_FP8_COMBINE": "1" } }, "dep8_c192": { "label": "192 (DEP8)", "concurrency": 192, "args": [ "--tensor-parallel-size", "1", "--data-parallel-size", "8", "--max-num-seqs", "48", "--max-num-batched-tokens", "8528", "--kv-cache-memory-bytes", "32212254720", "--compilation-config", "{\"cudagraph_mode\":\"FULL_DECODE_ONLY\",\"cudagraph_capture_sizes\":[7,14,21,28,35,42,49,56,63,70,77,84,91,98,105,112,119,126,133,140,147,154,161,168,175,182,189,196,203,210,217,224,231,238,245,252,259,266,273,280,287,294,301,308,315,322,329,336],\"mode\":0}" ], "env": { "VLLM_DSV4_MEGA_FP8_COMBINE": "1" } }, "tp4dp2_c32": { "label": "32 (TP4 x DP2)", "concurrency": 32, "args": [ "--tensor-parallel-size", "4", "--data-parallel-size", "2", "--max-num-seqs", "32", "--max-num-batched-tokens", "8416", "--kv-cache-memory-bytes", "51539607552", "--compilation-config", "{\"cudagraph_mode\":\"FULL_DECODE_ONLY\",\"cudagraph_capture_sizes\":[7,14,21,28,35,42,49,56,63,70,77,84,91,98,105,112,119,126,133,140,147,154,161,168,175,182,189,196,203,210,217,224],\"mode\":0}" ] }, "tp2dp4_c48": { "label": "48 (TP2 x DP4)", "concurrency": 48, "image": "vllm/vllm-openai@sha256:d428631bb917a33d22308265d1b8be91522f862bf9fd133bee62dde3313664ae", "args": [ "--tensor-parallel-size", "2", "--data-parallel-size", "4", "--max-num-seqs", "24", "--max-num-batched-tokens", "8360", "--kv-cache-memory-bytes", "37580963840", "--compilation-config", "{\"cudagraph_mode\":\"FULL_DECODE_ONLY\",\"cudagraph_capture_sizes\":[7,14,21,28,35,42,49,56,63,70,77,84,91,98,105,112,119,126,133,140,147,154,161,168],\"mode\":0}" ] }, "tp2dp4_c64": { "label": "64 (TP2 x DP4)", "concurrency": 64, "image": "vllm/vllm-openai@sha256:d428631bb917a33d22308265d1b8be91522f862bf9fd133bee62dde3313664ae", "args": [ "--tensor-parallel-size", "2", "--data-parallel-size", "4", "--max-num-seqs", "32", "--max-num-batched-tokens", "8416", "--kv-cache-memory-bytes", "37580963840", "--compilation-config", "{\"cudagraph_mode\":\"FULL_DECODE_ONLY\",\"cudagraph_capture_sizes\":[7,14,21,28,35,42,49,56,63,70,77,84,91,98,105,112,119,126,133,140,147,154,161,168,175,182,189,196,203,210,217,224],\"mode\":0}" ] } }, "runbook": "The data-parallel points of the B200 AgentX lane (`dsv4-fp4-b200-vllm-agentic-mtp` and its low-concurrency\ncompanion, [SemiAnalysisAI/InferenceX#4554](https://github.com/SemiAnalysisAI/InferenceX/pull/4554)) serve the\n0813 checkpoint with DSpark (six draft tokens) as one engine on one 8-GPU B200 node: DEP8 (TP1 x DP8) for\nconcurrency 64 to 192, TP4 x DP2 at 32 and TP2 x DP4 at 48 and 64, all with expert parallelism and the Simple\nCPU KV tier. Each rank gets 241.5 GB of host memory (1932 GB across the node), so size it to the machine.\n`--max-num-seqs` is per DP rank, and `--max-num-batched-tokens` is an 8192-token prefill budget plus seven\nverification slots per sequence. The TP4 x DP2 and TP2 x DP4 points leave `VLLM_DSV4_MEGA_FP8_COMBINE` unset.\nTP2 x DP4 was measured on a second pinned image, shown on those points.\n\nThe lane fronts the server with vLLM Router 0.1.14 (`--policy consistent_hash`, `--intra-node-data-parallel-size`\nequal to the server's DP size, as on the router tab), plus `--request-timeout-secs 14400` so long agentic\nrequests outlive the default and `--disable-retries` so a failed request is not replayed. Clients send a stable\nper-conversation `X-Session-ID` so the router keeps a conversation on the rank that holds its prefix.\n\nLeft out of the command: `VLLM_PREFIX_CACHE_RETENTION_INTERVAL` (removed in vLLM v0.30.0,\n[vllm#55353](https://github.com/vllm-project/vllm/pull/55353), so it had no effect), `--served-model-name`\n(the builder replaces it with the checkpoint ID) and the router's Prometheus port. The throughput points in\nInferenceX swap block verification for synthetic rejection at a committed acceptance length so every point is\nmeasured at the same acceptance rate; the command does not set it.\n", "json": "/deepseek-ai/DeepSeek-V4-Pro/presets/b200_agentx_dep.json" }, "mi355x_agentx_tp": { "label": "AgentX TP8 + monokernel", "hardware": "mi355x", "strategy": "single_node_tp", "variants": [ "default" ], "workload": "Agentic coding (InferenceX AgentX)", "description": "Source reproduction: DSpark K6 and monokernel on eight MI355X GPUs; draft communication precision requires resolution.", "source": { "label": "InferenceX #4526", "href": "https://github.com/SemiAnalysisAI/InferenceX/pull/4526", "key": "dsv4-fp4-mi355x-vllm-agentic-mtp", "revision": "5ec3c995988e5d3632c1ff53e8ae39da3e799892" }, "image": { "vllm": "vllm/vllm-openai-rocm:nightly-e4a5b5537344b103ed10de7764378ef09604ef3d", "model_revision": "72e1d3230f6c080a530b0a1d46f8eb4602340597", "pip": [ "Pillow", "fastapi", "uvicorn" ] }, "tp": 8, "select": { "frontend": "python", "features": { "tool_calling": true, "reasoning": true, "spec_decoding": "dspark" }, "kv_offload": "off" }, "args": [ "--revision", "72e1d3230f6c080a530b0a1d46f8eb4602340597", "--tokenizer-revision", "72e1d3230f6c080a530b0a1d46f8eb4602340597", "--async-scheduling", "--data-parallel-size", "1", "--gpu-memory-utilization", "0.86", "--moe-backend", "aiter", "--monokernel", "--enable-prefix-caching", "--no-disable-hybrid-kv-cache-manager" ], "env": { "VLLM_ENGINE_READY_TIMEOUT_S": "10800", "VLLM_USE_V2_MODEL_RUNNER": "1", "VLLM_ROCM_USE_AITER": "1", "VLLM_ROCM_QUICK_REDUCE_QUANTIZATION": "INT4", "VLLM_ROCM_USE_AITER_MOE": "1", "VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS": "0", "OMP_NUM_THREADS": "1" }, "json_args": { "--compilation-config": { "mode": 3, "cudagraph_mode": "FULL_AND_PIECEWISE" }, "--speculative-config": { "num_speculative_tokens": 6 } }, "points": { "c1": { "label": "1", "concurrency": 1, "args": [ "--max-num-seqs", "2" ] }, "c4": { "label": "4", "concurrency": 4, "args": [ "--max-num-seqs", "8" ] }, "c8": { "label": "8", "concurrency": 8, "args": [ "--max-num-seqs", "16" ] }, "c16": { "label": "16", "concurrency": 16, "args": [ "--max-num-seqs", "32" ] }, "c32": { "label": "32", "concurrency": 32, "args": [ "--max-num-seqs", "64" ] }, "c40": { "label": "40", "concurrency": 40, "args": [ "--max-num-seqs", "80" ] }, "c48": { "label": "48", "concurrency": 48, "args": [ "--max-num-seqs", "96" ] }, "c56": { "label": "56", "concurrency": 56, "args": [ "--max-num-seqs", "112" ] }, "c64": { "label": "64", "concurrency": 64, "args": [ "--max-num-seqs", "128" ] } }, "runbook": "This opt-in profile serves [DeepSeek-V4-Pro-0813](https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro-0813)\non one 8× MI355X node. The monokernel is enabled only for the TP8 arm;\n[vLLM #61482](https://github.com/vllm-project/vllm/pull/61482) added this\nDSpark-compatible path. These presets do not change the generic defaults.\n\n**Qualification blocked:** the source enables nondefault INT4 QuickReduce\nfor draft collectives too. For example, 32 active K6 requests produce at\nleast 192 draft rows, making the BF16 attention-output reduction eligible\nfor INT4 communication in the pinned image. This conflicts with\n[InferenceX's draft-precision rule](https://github.com/SemiAnalysisAI/InferenceX/blob/67de4b34ee1419b2a31c0e7200964a6e78015be4/CONTRIBUTING.md#draft-model-precision).\nThese are faithful source profiles, pending a source correction and\nqualifying validation; they are not a production recommendation. Real\ntarget verification and passing GSM8K do not resolve that precision issue.\n\nInferenceX installs `Pillow fastapi uvicorn` before engine startup. When\nusing the pinned image interactively, install these prerequisites inside\nthe container, then run the generated Python command in that container:\n\n```bash\nuv pip install Pillow fastapi uvicorn\n```\n\nThe generated Docker launch does not perform this setup step; provide the\ndependencies in its container before using that launch form.\n\nThe source pins the target and tokenizer revisions, but supplies no separate\nDSpark draft revision. DSpark loads the native draft from the same model ID\nthrough the image's default mixed FP8/MXFP4 handling; no draft dtype or\nquantization override is added. INT4 QuickReduce is shared by eligible\nstandard-path target and draft collectives, rather than being target-only.\nThe KV cache is FP8. The image is recorded as a tag, without a source-pinned\ndigest. Keep inherited environment settings consistent with the preset.\n\nThe generated serving commands use real target verification. For the\n**InferenceX throughput sweep only**, merge the following keys into their\nspeculative config, retaining DSpark K6 and probabilistic drafting:\n\n```json\n{\"rejection_sample_method\":\"synthetic\",\"synthetic_acceptance_length\":3.77}\n```\n\n`3.77` comes from the committed\n[DeepSeek-V4-Pro thinking-on K6 golden curve](https://github.com/SemiAnalysisAI/InferenceX/blob/5ec3c995988e5d3632c1ff53e8ae39da3e799892/inferencex-e2e/infx/golden_al_distribution/dsv4-pro-0813-dspark.yaml).\nSynthetic acceptance is a measurement control; do not use it for production\nor accuracy evaluation. The generic Bench command is a smoke benchmark,\nnot an AgentX replay; reproduce AgentX with the linked InferenceX harness.\n\nThe TP8 scheduler caps are twice the listed client concurrency for AgentX\nfan-out. No host-memory offload is enabled. The source's obsolete\n`VLLM_PREFIX_CACHE_RETENTION_INTERVAL=32768` has no effect in this image;\nthe presets retain the effective CLI default of `0` (semantic checkpoints).\n\nEvidence: [sweep 40260921924, attempt 3](https://github.com/SemiAnalysisAI/InferenceX/actions/runs/40260921924/attempts/3)\nfor source head `882db2060239f4d849f8b1c93f3b8b192ad3637e` covers all twelve TP8/DPA\nthroughput points and two real-verification GSM8K evals. The current-head\n[run 40626241537](https://github.com/SemiAnalysisAI/InferenceX/actions/runs/40626241537)\nonly reuses that sweep; its performance/eval matrices are skipped.\nIts actual CI merge `f301906060dcfd3b63ac6f77a735698f731f65b1` and the cited\nhead produce identical native recipe expansion for this lane.\nThis documentation change does not rerun GPU inference or establish private\nagentic accuracy evaluation.\n", "json": "/deepseek-ai/DeepSeek-V4-Pro/presets/mi355x_agentx_tp.json" }, "mi355x_agentx_dpa": { "label": "AgentX DPA8 + TP experts", "hardware": "mi355x", "strategy": "single_node_dpa_tp", "variants": [ "default" ], "workload": "Agentic coding (InferenceX AgentX)", "description": "TP1 attention / DP8 with tensor-sharded experts, DSpark K6 and session-affine routing; monokernel disabled.", "source": { "label": "InferenceX #4526", "href": "https://github.com/SemiAnalysisAI/InferenceX/pull/4526", "key": "dsv4-fp4-mi355x-vllm-agentic-mtp", "revision": "5ec3c995988e5d3632c1ff53e8ae39da3e799892" }, "image": { "vllm": "vllm/vllm-openai-rocm:nightly-e4a5b5537344b103ed10de7764378ef09604ef3d", "model_revision": "72e1d3230f6c080a530b0a1d46f8eb4602340597", "pip": [ "Pillow", "fastapi", "uvicorn", "vllm-router==0.1.14" ] }, "select": { "frontend": "python", "features": { "tool_calling": true, "reasoning": true, "spec_decoding": "dspark" }, "kv_offload": "off" }, "args": [ "--revision", "72e1d3230f6c080a530b0a1d46f8eb4602340597", "--tokenizer-revision", "72e1d3230f6c080a530b0a1d46f8eb4602340597", "--async-scheduling", "--gpu-memory-utilization", "0.86", "--moe-backend", "aiter", "--enable-prefix-caching", "--no-disable-hybrid-kv-cache-manager", "--prefill-schedule-interval", "8", "--long-prefill-token-threshold", "16384" ], "env": { "VLLM_ENGINE_READY_TIMEOUT_S": "10800", "VLLM_USE_V2_MODEL_RUNNER": "1", "VLLM_ROCM_USE_AITER": "1", "VLLM_ROCM_QUICK_REDUCE_QUANTIZATION": "INT4", "VLLM_ROCM_USE_AITER_MOE": "1", "VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS": "0", "OMP_NUM_THREADS": "1" }, "json_args": { "--compilation-config": { "mode": 3, "cudagraph_mode": "FULL_AND_PIECEWISE" }, "--speculative-config": { "num_speculative_tokens": 6 } }, "points": { "c64": { "label": "64", "concurrency": 64, "args": [ "--max-num-seqs", "64" ] }, "c128": { "label": "128", "concurrency": 128, "args": [ "--max-num-seqs", "128" ] }, "c192": { "label": "192", "concurrency": 192, "args": [ "--max-num-seqs", "192" ] } }, "runbook": "This is the DP-attention arm of the\n[TP8 profile](#preset-mi355x_agentx_tp), using the same checkpoint, image,\nruntime prerequisites and real-versus-synthetic verification distinction.\nThe source qualification blocker described there remains unresolved.\nRuntime flags are TP1/DP8 with no expert-parallel flag: experts remain\ntensor-sharded across the eight ranks. The source's matrix label TP8/EP1\ndescribes the aggregate topology, not the attention TP argument.\nMonokernel remains disabled (the image default); TP8-only monokernel\neligibility does not extend to this arm.\n\nEach DP scheduler caps sequences at the listed client concurrency.\nThe measured prefill interval is `8`; the `16384` long-prefill threshold\nexceeds the `8192` token budget and does not bind.\n\nFor exact source routing, replace the generic router companion with this\npinned setup in another shell on the engine's host/container network:\n\n```bash\nuv pip install vllm-router==0.1.14\nvllm-router \\\n --host 0.0.0.0 --port 8000 \\\n --worker-urls http://localhost:8100 \\\n --intra-node-data-parallel-size 8 \\\n --policy consistent_hash \\\n --request-timeout-secs 14400 \\\n --disable-retries\n```\n\nClients connect to the router on port `8000`, retaining an `X-Session-ID`\nacross turns of each conversation. InferenceX maps correlation IDs to\nthis header; `AIPERF_HTTP_X_SESSION_ID_FROM_CORRELATION_ID=1` is a benchmark\nclient setting, not an engine environment variable.\n", "json": "/deepseek-ai/DeepSeek-V4-Pro/presets/mi355x_agentx_dpa.json" } }, "guide": "## Overview\n\nDeepSeek-V4-Pro is the flagship of the V4 preview family: a 1.6T-total / 49B-active\nMixture-of-Experts model. It pairs a **hybrid attention stack** — Compressed Sparse\nAttention (CSA) + Heavily Compressed Attention (HCA) — with **Manifold-Constrained\nHyper-Connections (mHC)** to reach 27% of V3.2's per-token inference FLOPs and 10% of\nV3.2's KV cache at 1M context. Pre-trained on 32T+ tokens with the **Muon optimizer**\nfor faster convergence; post-training is a two-stage pipeline (domain-specific expert\ncultivation + unified consolidation via on-policy distillation).\n\nCheckpoint is **FP4+FP8 mixed**: MoE expert weights are stored in FP4 while the\nremaining (attention / norm / router) params stay in FP8.\n\nAn **NVFP4 variant** (`nvidia/DeepSeek-V4-Pro-NVFP4`) is also available — NVIDIA\nmodelopt re-quantizes the MoE experts to standard NVFP4 while attention, shared\nexperts, router head, and MTP stay FP8. Pick it from the **Variant** row; it runs on\nBlackwell GPUs with the FP4 indexer cache.\n\n## Checkpoints\n\nFour checkpoints are on the **Variant** row. They differ in *which weights you serve*\nand *which draft module ships with them* — the speculative method itself is picked on\nthe **Spec Decoding** row.\n\n| Variant | Repo | Draft | Notes |\n| :-- | :-- | :-- | :-- |\n| **FP8 (0813)** (default) | `deepseek-ai/DeepSeek-V4-Pro-0813` | DSpark | Official release, preview structure + DSpark |\n| **FP8 (Preview)** | `deepseek-ai/DeepSeek-V4-Pro` | MTP | Preview FP4+FP8 mixed weights |\n| **NVFP4** | `nvidia/DeepSeek-V4-Pro-NVFP4` | MTP | modelopt re-quant, Blackwell |\n| **DSpark** | `deepseek-ai/DeepSeek-V4-Pro-DSpark` | DSpark | Preview weights + fused DSpark module |\n\n**0813 is the official DeepSeek-V4-Pro release** and the default here, superseding\nthe preview with substantially stronger agentic capability — 87.9 on Terminal Bench\n2.1, 62.7 on DeepSWE and 42.7 on HLE (60.0 with tools), versus 72.1, 12.8 and 37.7\nfor the preview. It is built on the preview model structure with a DSpark speculative\ndecoding module attached. The preview weights remain available as the\n**FP8 (Preview)** variant for reproducing earlier results.\n\n**DSpark** is *not* new weights — it is the preview checkpoint with a speculative\ndecoding module attached (see [DeepSpec](https://github.com/deepseek-ai/DeepSpec)).\nBoth fused checkpoints add a `dspark_*` block to `config.json`; the preview and NVFP4\ncheckpoints have no such block, which is why the DSpark method is offered only on the\ntwo variants that carry the draft. Selecting either one auto-enables **Spec Decoding**\nwith `method: dspark`, emitting:\n\n```\n--speculative-config '{\"method\":\"dspark\",\"num_speculative_tokens\":7,\"draft_sample_method\":\"probabilistic\"}'\n```\n\nThe fused checkpoints are ~893 GB on disk versus ~865 GB for the preview — the draft\nmodule is the difference. Both require **vLLM 0.25.0**, above this recipe's 0.20.0\nbaseline; the Install block bumps automatically when either is selected.\n\n## Reasoning modes\n\nThe chat template exposes three reasoning-effort modes:\n\n- **Non-think** — fast, intuitive responses.\n- **Think High** — explicit chain-of-thought for complex problem-solving and planning.\n- **Think Max** — maximum reasoning effort; requires `--max-model-len >= 393216`\n (384K tokens) to avoid truncation.\n\nRecommended sampling: `temperature = 1.0`, `top_p = 1.0`.\n\nOn the **0813** variant the effort levels are named `low` / `high` / `max`, and\nDeepSeek recommends `top_p = 0.95` for agentic scenarios (`1.0` otherwise) with\n`temperature = 1.0`. Allow up to **384K output tokens** at the `high` and `max`\nlevels.\n\nNote that the 0813 release ships no Jinja chat template — the repo provides an\n`encoding/` folder with `encode_messages` / `parse_message_from_completion_text`\nhelpers instead. Serving through vLLM with `--tokenizer-mode deepseek_v4` (the\n**Tool Calling** pill) applies the built-in DeepSeek-V4 encoding, so the OpenAI-\ncompatible endpoint works without the helper scripts.\n\n### OpenAI Client Example\n\nFor DeepSeek-V4, keep reasoning controls in `chat_template_kwargs`, as it exposes a\ncustom **Think Max** mode via `\"reasoning_effort\": \"max\"`.\n\n```python\nfrom openai import OpenAI\n\nclient = OpenAI(base_url=\"http://localhost:8000/v1\", api_key=\"EMPTY\")\nmodel = \"deepseek-ai/DeepSeek-V4-Pro-0813\"\nmessages = [{\"role\": \"user\", \"content\": \"What is 17*19? Return only the final integer.\"}]\n\n# Non-think\nresp = client.chat.completions.create(\n model=model,\n messages=messages,\n)\n\n# Think High\nresp = client.chat.completions.create(\n model=model,\n messages=messages,\n extra_body={\n \"chat_template_kwargs\": {\n \"thinking\": True,\n \"reasoning_effort\": \"high\",\n },\n },\n)\n\n# Think Max\nresp = client.chat.completions.create(\n model=model,\n messages=messages,\n extra_body={\n \"chat_template_kwargs\": {\n \"thinking\": True,\n \"reasoning_effort\": \"max\",\n },\n },\n)\n```\n\n## Recommended deployments\n\n- **B300 (8× GPU)**: single-node DP + EP with `--data-parallel-size 8`.\n- **H200 (8× GPU)**: DP + EP with `--data-parallel-size 8`. Context is capped at\n 800K tokens (`--max-model-len 800000`) to leave KV headroom with dense params\n replicated across ranks — applies to both single-node and multi-node H200.\n- **MI355X (8× GPU)**: validated with ROCm + AITER (`VLLM_ROCM_USE_AITER=1`),\n `--gpu-memory-utilization 0.9`, `--max-num-seqs 128`,\n `--max-num-batched-tokens 8192`, and `--distributed-executor-backend mp`.\n- **GB200 NVL4 (4× GPU per tray)**: the ~960 GB mixed-precision checkpoint does not\n fit on one tray; run multi-node DP + EP across **2 trays** (8 GPUs total) with\n `--data-parallel-size 8`. Pick the \"Multi-Node\" tab and set nodes to 2.\n\n### MI355X (8×288GB)\n\n```bash\nexport VLLM_ROCM_USE_AITER=1\n\nvllm serve deepseek-ai/DeepSeek-V4-Pro \\\n --host localhost \\\n --port 8001 \\\n --dtype auto \\\n --kv-cache-dtype fp8 \\\n --tensor-parallel-size 8 \\\n --max-num-seqs 512 \\\n --max-num-batched-tokens 8192 \\\n --distributed-executor-backend mp \\\n --trust-remote-code \\\n --gpu-memory-utilization 0.9 \\\n --tokenizer-mode deepseek_v4 \\\n --reasoning-parser deepseek_v4 \\\n --tool-call-parser deepseek_v4 \\\n --enable-auto-tool-choice \\\n --compilation-config '{\"mode\": 3, \"cudagraph_mode\": \"FULL_DECODE_ONLY\"}'\n```\n\n#### MI355X is validated on GSM8K dataset:\n\n##### Launch command\n\n```bash\nMODEL=deepseek-ai/DeepSeek-V4-Pro\nlm_eval --model local-completions \\\n --model_args model=$MODEL,base_url=http://0.0.0.0:8001/v1/completions,num_concurrent=128,max_retries=10,max_gen_toks=2048,timeout=60000 \\\n --batch_size auto \\\n --tasks gsm8k \\\n --num_fewshot 8 \\\n --output_path . 2>&1 | tee -a eval.log\n```\n\n##### Reported result\n\n```text\nlocal-completions ({'model': 'deepseek-ai/DeepSeek-V4-Pro', 'base_url': 'http://0.0.0.0:8001/v1/completions', 'num_concurrent': 128, 'max_retries': 10, 'max_gen_toks': 2048, 'timeout': 60000}), gen_kwargs: ({}), limit: None, num_fewshot: 8, batch_size: auto\n|Tasks|Version| Filter |n-shot| Metric | |Value | |Stderr|\n|-----|------:|----------------|-----:|-----------|---|-----:|---|-----:|\n|gsm8k| 3|flexible-extract| 8|exact_match|↑ |0.9538|± |0.0058|\n| | |strict-match | 8|exact_match|↑ |0.9545|± |0.0057|\n```\n\n## KV Cache Offloading\n\nAgentic and multi-turn workloads reuse long prefixes whose KV state can exceed on-GPU\ncapacity. The **KV Offload** row attaches a host-DRAM KV tier to any serving strategy —\npick one of three connectors:\n\n- **Simple** — `SimpleCPUOffloadConnector`: spills KV blocks to a per-rank region of CPU\n DRAM on each node. The simplest way to extend effective KV capacity for a single instance.\n- **Mooncake** — `MooncakeStoreConnector`: pools CPU DRAM into a distributed shared KV\n store — either embedded (each rank's vLLM worker runs a Mooncake client that donates a DRAM segment)\n or standalone (a per-node `mooncake_store_service` owns the node's DRAM and contributes\n it to the pool, decoupling cache lifetime from the engine). A cluster-wide\n `mooncake_master` coordinates the pool; across 2+ instances, GPU workers share it for\n cross-instance prefix reuse. Install from the extra-install block above; see the [Mooncake docs](https://kvcache-ai.github.io/Mooncake/).\n- **LMCache** — `LMCacheMPConnector`: a node-local KV pool served by a\n companion `lmcache server` process, launched before `vllm serve`.\n Single-node strategies only. Install from the extra-install block above; see the [LMCache docs](https://docs.lmcache.ai/).\n\n## Agentic serving on GB200/GB300 (DSpark)\n\nFor agentic serving, use the following common settings with\n[DeepSeek-V4-Pro-0813](https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro-0813/tree/72e1d3230f6c080a530b0a1d46f8eb4602340597)\nand its native DSpark draft. These are separate from the command-builder defaults.\n\n### Common settings\n\nSet `VLLM_USE_V2_MODEL_RUNNER=1` and use these engine arguments:\n\n```text\n--tokenizer-mode deepseek_v4 --trust-remote-code\n--kv-cache-dtype fp8 --block-size 256\n--no-disable-hybrid-kv-cache-manager --disable-flashinfer-autotune\n--attention-config '{\"backend\":\"FLASHINFER_MLA_SPARSE_DSV4\",\"use_prefill_query_quantization\":true,\"indexer_kv_dtype\":\"mxfp4\"}'\n--speculative-config '{\"method\":\"dspark\",\"num_speculative_tokens\":6,\"draft_sample_method\":\"probabilistic\",\"rejection_sample_method\":\"block\"}'\n```\n\nThe example uses `K=6`, where `K` is `num_speculative_tokens`. Tune K and the\nprefill/decode (PD) ratio for each target concurrency to balance throughput,\nper-user latency and time to first token. Use the same K on both PD roles.\n\nFor aggregate TP serving, use\n`--disable-custom-all-reduce`, `VLLM_ALLREDUCE_USE_FLASHINFER=1`,\n`VLLM_FLASHINFER_ALLREDUCE_BACKEND=auto`; do not select an EP MoE backend.\nFor DEP (TP1 with data and expert parallelism), both pools instead use\n`--enable-expert-parallel --enable-ep-weight-filter --enable-cumem-allocator`,\n`--moe-backend deep_gemm_amxf4_mega_moe` and `VLLM_DSV4_MEGA_FP8_COMBINE=1`.\nThis MoE configuration supports the native mixed FP4/FP8 checkpoint, unlike\nthe FP8-only `deep_gemm_mega_moe` setting above.\n\nFor aggregate and decode, set `cudagraph_mode=FULL_DECODE_ONLY` and `mode=0`\nin `--compilation-config`, with explicit capture sizes `n*(K+1)` for `n=1..S`.\n`S` is `max-num-seqs` per DP rank, or per engine for pure TP. Set the maximum\ncapture size to `S*(K+1)`; aggregate serving can also include `n*K` sizes.\nRecompute the capture sizes when K or S changes. Size `max-num-batched-tokens`\nto cover verification tokens plus any prefill work scheduled on that engine.\n\n### Scheduling, NUMA binding, and streaming\n\n**DEP decode scheduling:** the decode example below uses:\n\n```text\n--prefill-schedule-interval 16 --long-prefill-token-threshold 512\n```\n\nThe interval controls how often DP ranks admit new prefill requests. Increasing\nit can balance work across ranks but delays admission. The threshold limits\nlong-prefill chunk size when requests share the token budget.\nCompare these overrides with leaving both flags unset, measuring throughput,\ntime to first token, and inter-token latency at your target load. Retune when\nchanging the workload, pool sizes, hardware, or engine version.\n\nThe prefill example uses `--long-prefill-token-threshold 1024` and leaves\n`--prefill-schedule-interval` unset.\n\n**DEP NUMA binding:** on every prefill and decode node, enable binding and\nprovide one NUMA node ID per visible GPU. For the four-GPU layout where GPUs\n0/1 are local to NUMA node 0 and GPUs 2/3 to NUMA node 1, use:\n\n```text\n--numa-bind --numa-bind-nodes 0 0 1 1\n```\n\nMatch the list to the local GPU order and topology on each node, including\nGB300 deployments. See the [vLLM NUMA binding guide](https://docs.vllm.ai/en/latest/configuration/optimization/#numa-binding-for-multi-socket-gpu-nodes)\nfor automatic detection and explicit mappings.\n\n**Aggregate streaming:** add `--stream-interval 10` to the aggregate TP8\nworker at concurrency 1, 4, or 8. This batches streamed output into larger\nupdates to reduce host overhead; use a smaller interval when smoother token\ndelivery matters more. The DEP configurations leave this override unset.\n\n### Running with Dynamo\n\nThe example below runs a DEP8 prefill pool and a DEP8 decode pool, each across\ntwo four-GPU GB200 nodes, plus a host for shared services and the frontend.\nRun the Python commands in `vllm/vllm-openai:nightly-dev-arm64-cu13.0.1-20ebb28`\nwith the model, recipe files, GPUs, and RDMA devices accessible;\nuse host networking so the advertised addresses are reachable across nodes.\nInstall the matching Dynamo packages in each frontend/worker container:\n\n```bash\npython3 -m pip install --extra-index-url https://pypi.nvidia.com \\\n ai-dynamo==1.3.0.dev20260720 ai-dynamo-runtime==1.3.0.dev20260720\n```\n\nSave the following as `dynamo-env.sh` and source it in each shell. Replace the\nexample addresses with routable addresses on your cluster. `MODEL_PATH` must\ncontain the pinned 0813 checkpoint on every worker.\n\n```bash\nexport INFRA_IP=10.0.0.10\nexport ETCD_ENDPOINTS=\"http://${INFRA_IP}:2379\"\nexport NATS_SERVER=\"nats://${INFRA_IP}:4222\"\nexport DYN_NAMESPACE=dspark\nexport MODEL_PATH=/models/DeepSeek-V4-Pro-0813\nexport SERVED_MODEL=deepseek-ai/DeepSeek-V4-Pro-0813\nexport PREFILL_HEAD_IP=10.0.0.11\nexport DECODE_HEAD_IP=10.0.0.21\n```\n\n**Shared services.** Provide `etcd`, `nats-server`, and `mooncake_master` on the\nservice host's `PATH`. See the official [Dynamo service setup](https://github.com/ai-dynamo/dynamo/blob/59dd504105db498b11e5715854a2512d57b7a1e0/dev/docker-compose.yml)\nand [Mooncake installation](https://kvcache-ai.github.io/Mooncake/) for installation\noptions. Start these services once on the private cluster network, before workers:\n\n```bash\nsource ./dynamo-env.sh\netcd --data-dir ./etcd-data --listen-client-urls http://0.0.0.0:2379 \\\n --advertise-client-urls \"$ETCD_ENDPOINTS\" >etcd.log 2>&1 &\ncat >nats.conf <<'EOF'\nport: 4222\nmax_payload: 33554432\njetstream { store_dir: \"./nats-data\" }\nEOF\nnats-server -c ./nats.conf >nats.log 2>&1 &\nmooncake_master --port=8700 --enable_http_metadata_server=true \\\n --http_metadata_server_port=8701 --eviction_high_watermark_ratio=0.9 \\\n --default_kv_lease_ttl=10000 --rpc_thread_num=16 \\\n --enable_metric_reporting=true --metrics_port=8702 >mooncake.log 2>&1 &\n```\n\nWait for the services to listen before continuing. All hosts must reach the\nshared endpoints; workers must also reach each other's DP rendezvous and NIXL\naddresses. Existing managed services can supply the same endpoints.\n\n**Frontend.** In a separate shell, start the HTTP endpoint for the disaggregated\ndeployment:\n\n```bash\nsource ./dynamo-env.sh\nexport DYN_ROUTER_ACTIVE_REQUEST_EXPIRY_SECS=3600\nexport DYN_TCP_CHANNEL_BUFFER=128\nexport DYN_TCP_REQUEST_TIMEOUT=60\npython3 -m dynamo.frontend --http-port 8000 --router-mode random \\\n --router-session-affinity-ttl-secs 900\n```\n\n- Aggregate: `--router-mode kv`, `DYN_ROUTER_TEMPERATURE=0` and\n `DYN_TOKENIZER=fastokens`.\n- Disaggregated: `--router-mode random` with session affinity, or `kv` with\n the same tokenizer settings. Send a stable conversation\n ID as `x-dynamo-session-id` and set `--router-session-affinity-ttl-secs` to\n cover the conversation lifetime.\n- Both DEP roles use `MultiConnector` with `NixlConnector` for PD transfer and\n `MooncakeStoreConnector` for shared prefix storage. Enable store lookup only\n on prefill, and set `VLLM_CONNECTOR_PREFETCH_DEPTH=0` there. Role-specific\n `--kv-transfer-config` values:\n\nPrefill — save as `prefill-kv.json` in the worker's working directory:\n\n```json\n{\"kv_connector\":\"MultiConnector\",\"kv_role\":\"kv_both\",\"kv_connector_extra_config\":{\"connectors\":[{\"kv_connector\":\"NixlConnector\",\"kv_role\":\"kv_both\",\"kv_load_failure_policy\":\"fail\",\"kv_buffer_device\":\"cuda\",\"kv_connector_extra_config\":{\"enforce_handshake_compat\":false,\"enable_cross_layers_blocks\":false}},{\"kv_connector\":\"MooncakeStoreConnector\",\"kv_role\":\"kv_both\",\"kv_connector_extra_config\":{\"load_async\":true,\"lookup_async\":true,\"enable_cross_layers_blocks\":false,\"enable_offload\":false}}]}}\n```\n\nDecode — save as `decode-kv.json` in the worker's working directory:\n\n```json\n{\"kv_connector\":\"MultiConnector\",\"kv_role\":\"kv_consumer\",\"kv_connector_extra_config\":{\"connectors\":[{\"kv_connector\":\"NixlConnector\",\"kv_role\":\"kv_consumer\",\"kv_load_failure_policy\":\"fail\",\"kv_buffer_device\":\"cuda\",\"kv_connector_extra_config\":{\"enforce_handshake_compat\":false,\"enable_cross_layers_blocks\":false}},{\"kv_connector\":\"MooncakeStoreConnector\",\"kv_role\":\"kv_consumer\",\"kv_connector_extra_config\":{\"load_async\":true,\"lookup_async\":false,\"enable_lookup\":false,\"enable_cross_layers_blocks\":false,\"enable_offload\":false}}]}}\n```\n\n**Worker environment.** Run the following in Bash on each worker node. Set\n`NODE_IP` to that node's routable address and `NODE_RANK` to 0 or 1 within its\nown pool. For example, prefill nodes `10.0.0.11` / `10.0.0.12` use ranks 0 / 1;\ndecode nodes `10.0.0.21` / `10.0.0.22` also use ranks 0 / 1. All four GPUs are\nvisible to one launch process per node. Adjust the NUMA and NIC mappings to\nmatch the hardware. The embedded Mooncake store uses 140 GB per local rank\nin this example; ensure enough host memory or resize it for your deployment.\n\n```bash\nsource ./dynamo-env.sh\nexport NODE_IP=10.0.0.11 # Change on each worker node.\nNODE_RANK=0 # Use 1 on the second node of either pool.\nexport CUDA_VISIBLE_DEVICES=0,1,2,3\nexport VLLM_HOST_IP=\"$NODE_IP\"\nexport VLLM_NIXL_SIDE_CHANNEL_HOST=\"$NODE_IP\"\nexport VLLM_NIXL_SIDE_CHANNEL_PORT=20096\nexport DYN_SYSTEM_PORT=8081\nexport VLLM_USE_V2_MODEL_RUNNER=1 VLLM_DSV4_MEGA_FP8_COMBINE=1\nexport PYTHONHASHSEED=0\nexport MOONCAKE_MASTER=\"${INFRA_IP}:8700\"\nexport MOONCAKE_TE_META_DATA_SERVER=\"http://${INFRA_IP}:8701/metadata\"\nexport MOONCAKE_LOCAL_HOSTNAME=\"$NODE_IP\"\nexport MOONCAKE_CONFIG_PATH=\"$PWD/mooncake.json\"\ncat >\"$MOONCAKE_CONFIG_PATH\" <