{ "hf_id": "deepseek-ai/DeepSeek-V3.2", "meta": { "title": "DeepSeek-V3.2", "slug": "deepseek-v3.2", "provider": "DeepSeek", "description": "DeepSeek V3.2 MoE model with MLA attention, sparse attention, and scalable RL for strong reasoning and agent capabilities.", "date_added": "2025-12-04", "date_updated": "2026-09-14", "tasks": [ "text" ], "performance_headline": "GPT-5-level reasoning with efficient MoE inference", "related_recipes": [], "hardware": { "h100": "verified", "h200": "verified" } }, "recommended_command": { "hardware": "h200", "strategy": "single_node_tp", "variant": "default", "node_count": 1, "deploy_type": "single_node", "env": { "VLLM_USE_RUST_FRONTEND": "1" }, "docker_image": "vllm/vllm-openai:latest", "command": "vllm serve deepseek-ai/DeepSeek-V3.2 \\\n --trust-remote-code \\\n --disable-flashinfer-autotune \\\n --tensor-parallel-size 8 \\\n --tokenizer-mode deepseek_v32 \\\n --tool-call-parser deepseek_v32 \\\n --enable-auto-tool-choice \\\n --reasoning-parser deepseek_v3", "argv": [ "vllm", "serve", "deepseek-ai/DeepSeek-V3.2", "--trust-remote-code", "--disable-flashinfer-autotune", "--tensor-parallel-size", "8", "--tokenizer-mode", "deepseek_v32", "--tool-call-parser", "deepseek_v32", "--enable-auto-tool-choice", "--reasoning-parser", "deepseek_v3" ], "docker_command": "docker run --gpus all \\\n --privileged --ipc=host -p 8000:8000 \\\n -v ~/.cache/huggingface:/root/.cache/huggingface \\\n -e VLLM_USE_RUST_FRONTEND=1 \\\n vllm/vllm-openai:latest deepseek-ai/DeepSeek-V3.2 \\\n --trust-remote-code \\\n --disable-flashinfer-autotune \\\n --tensor-parallel-size 8 \\\n --tokenizer-mode deepseek_v32 \\\n --tool-call-parser deepseek_v32 \\\n --enable-auto-tool-choice \\\n --reasoning-parser deepseek_v3", "docker_argv": [ "docker", "run", "--gpus", "all", "--privileged", "--ipc=host", "-p", "8000:8000", "-v", "~/.cache/huggingface:/root/.cache/huggingface", "-e", "VLLM_USE_RUST_FRONTEND=1", "vllm/vllm-openai:latest", "deepseek-ai/DeepSeek-V3.2", "--trust-remote-code", "--disable-flashinfer-autotune", "--tensor-parallel-size", "8", "--tokenizer-mode", "deepseek_v32", "--tool-call-parser", "deepseek_v32", "--enable-auto-tool-choice", "--reasoning-parser", "deepseek_v3" ], "strategy_spec": { "name": "single_node_tp", "deploy_type": "single_node", "display_name": "Tensor Parallel", "short_label": "TP", "orientation": "latency", "description": "Single-node tensor parallel. Splits the model across all local GPUs. TP size is set to the GPU count at deploy time. The simplest multi-GPU strategy — works for all model architectures.\n", "hardware_match": { "min_gpus": 1, "max_gpus": 8, "multi_node": false }, "vllm_args": [], "parallel_flag": "--tensor-parallel-size" }, "hardware_profile": { "brand": "NVIDIA", "generation": "hopper", "compute_arch": "sm90", "display_name": "H200", "description": "NVIDIA H200 SXM 141 GB HBM3e · 8-GPU HGX node", "gpu_count": 8, "vram_gb": 1128, "multi_node": false }, "alternatives": { "single_node_tep": "/deepseek-ai/DeepSeek-V3.2/strategies/single_node_tep.json", "single_node_dep": "/deepseek-ai/DeepSeek-V3.2/strategies/single_node_dep.json", "multi_node_tp": "/deepseek-ai/DeepSeek-V3.2/strategies/multi_node_tp.json", "multi_node_tp_pp": "/deepseek-ai/DeepSeek-V3.2/strategies/multi_node_tp_pp.json", "multi_node_dep": "/deepseek-ai/DeepSeek-V3.2/strategies/multi_node_dep.json", "multi_node_tep": "/deepseek-ai/DeepSeek-V3.2/strategies/multi_node_tep.json", "pd_cluster": "/deepseek-ai/DeepSeek-V3.2/strategies/pd_cluster.json", "kv_store_centralized_mooncake": "/deepseek-ai/DeepSeek-V3.2/strategies/kv_store_centralized_mooncake.json", "kv_store_distributed_mooncake": "/deepseek-ai/DeepSeek-V3.2/strategies/kv_store_distributed_mooncake.json" }, "by_hardware": { "h200": "/deepseek-ai/DeepSeek-V3.2/hw/h200.json", "h100": "/deepseek-ai/DeepSeek-V3.2/hw/h100.json", "b200": "/deepseek-ai/DeepSeek-V3.2/hw/b200.json", "gb200": "/deepseek-ai/DeepSeek-V3.2/hw/gb200.json", "b300": "/deepseek-ai/DeepSeek-V3.2/hw/b300.json", "gb300": "/deepseek-ai/DeepSeek-V3.2/hw/gb300.json", "mi300x": "/deepseek-ai/DeepSeek-V3.2/hw/mi300x.json", "mi325x": "/deepseek-ai/DeepSeek-V3.2/hw/mi325x.json", "mi350x": "/deepseek-ai/DeepSeek-V3.2/hw/mi350x.json", "mi355x": "/deepseek-ai/DeepSeek-V3.2/hw/mi355x.json" } }, "model": { "model_id": "deepseek-ai/DeepSeek-V3.2", "min_vllm_version": "0.18.0", "default_frontend": "rust", "architecture": "moe", "parameter_count": "671B", "active_parameters": "37B", "context_length": 163840, "supports_dcp": true, "base_args": [ "--trust-remote-code", "--disable-flashinfer-autotune" ], "base_env": {}, "install": { "pip": { "command": "uv venv\nsource .venv/bin/activate\nuv pip install -U vllm --torch-backend auto" }, "docker": { "command": "docker pull vllm/vllm-openai:latest" }, "extras": [ { "note": "DeepGEMM pinned build for MQA logits (FP8 MoE kernels)", "command": "uv pip install git+https://github.com/deepseek-ai/DeepGEMM.git@v2.1.1.post3 --no-build-isolation" }, { "note": "Set VLLM_USE_DEEP_GEMM=0 to skip DeepGEMM for the MoE path (recommended on H20)", "command": "export VLLM_USE_DEEP_GEMM=0", "optional": true } ] } }, "features": { "tool_calling": { "description": "Enable tool calling with DeepSeek V3.2 chat template support.", "args": [ "--tokenizer-mode", "deepseek_v32", "--tool-call-parser", "deepseek_v32", "--enable-auto-tool-choice" ] }, "reasoning": { "description": "Enable reasoning/thinking mode with the DeepSeek V3 reasoning parser.", "args": [ "--reasoning-parser", "deepseek_v3" ] }, "spec_decoding": { "description": "Multi-Token Prediction speculative decoding with 3 speculative tokens.", "args": [ "--speculative-config", "{\"method\":\"mtp\",\"num_speculative_tokens\":3}" ] } }, "opt_in_features": [ "spec_decoding" ], "variants": { "default": { "precision": "fp8", "vram_minimum_gb": 805, "description": "Native FP8 checkpoint (F8_E4M3)" }, "nvfp4": { "model_id": "nvidia/DeepSeek-V3.2-NVFP4", "precision": "nvfp4", "vram_minimum_gb": 403, "description": "NVIDIA FP4 quantized variant with FP8 KV cache for reduced VRAM usage.", "extra_args": [ "--kv-cache-dtype", "fp8" ], "json": "/nvidia/DeepSeek-V3.2-NVFP4.json" }, "redhat_nvfp4": { "model_id": "RedHatAI/DeepSeek-V3.2-NVFP4-FP8-BLOCK", "precision": "nvfp4", "label": "NVFP4+FP8 Block (RedHat)", "vram_minimum_gb": 430, "description": "RedHatAI NVFP4+FP8 block mixed-precision checkpoint", "json": "/RedHatAI/DeepSeek-V3.2-NVFP4-FP8-BLOCK.json" } }, "hardware_overrides": {}, "guide": "## Overview\n\nDeepSeek-V3.2 is a Mixture-of-Experts model that balances computational efficiency with\nstrong reasoning and agent capabilities through three technical innovations: DeepSeek Sparse\nAttention (DSA) for efficient long-context processing, a scalable reinforcement learning\nframework achieving GPT-5-level performance, and a large-scale agentic task synthesis pipeline\nfor robust tool-use generalization.\n\n## Prerequisites\n\n- **Hardware**: Minimum 8x H100/H200 80GB GPUs (BF16) or 3x H200 (NVFP4 variant).\n- **vLLM**: Version 0.18.0 or later (nightly recommended).\n- **Python**: 3.10+\n- **CUDA**: 12.x or later (CUDA 13.x may require extra env vars; see Troubleshooting).\n- **Disk**: ~1.3 TB for BF16 weights; ~350 GB for NVFP4 variant.\n- **DeepGEMM** (recommended):\n ```bash\n uv pip install git+https://github.com/deepseek-ai/DeepGEMM.git@v2.1.1.post3 --no-build-isolation\n ```\n Note: Set `VLLM_USE_DEEP_GEMM=0` to disable MoE DeepGEMM if you experience issues\n (e.g., on H20 GPUs) or want to skip the long warmup.\n\n## Client Usage\n\nLaunch the server:\n```bash\nvllm serve deepseek-ai/DeepSeek-V3.2 \\\n --tensor-parallel-size 8 \\\n --trust-remote-code \\\n --disable-flashinfer-autotune \\\n --tokenizer-mode deepseek_v32 \\\n --tool-call-parser deepseek_v32 \\\n --enable-auto-tool-choice \\\n --reasoning-parser deepseek_v3\n```\n\nUse the OpenAI Python SDK to interact with the server:\n```python\nfrom openai import OpenAI\n\nclient = OpenAI(\n api_key=\"your-api-key\",\n base_url=\"http://localhost:8000/v1\",\n)\n\n# Standard chat\nresponse = client.chat.completions.create(\n model=\"deepseek-ai/DeepSeek-V3.2\",\n messages=[{\"role\": \"user\", \"content\": \"Hello!\"}],\n)\n\n# Thinking / reasoning mode\nresponse = client.chat.completions.create(\n model=\"deepseek-ai/DeepSeek-V3.2\",\n messages=[{\"role\": \"user\", \"content\": \"Solve this step by step...\"}],\n extra_body={\"chat_template_kwargs\": {\"thinking\": True}},\n)\n```\n\n## Troubleshooting\n\n**`ptxas fatal: Value 'sm_110a' is not defined for option 'gpu-name'`**\nThis can occur on CUDA 13.x. Fix by exporting:\n```bash\nexport TRITON_PTXAS_PATH=/usr/local/cuda/bin/ptxas\nexport PATH=/usr/local/cuda/bin:$PATH\nexport LD_LIBRARY_PATH=/usr/local/cuda/lib64:${LD_LIBRARY_PATH:-}\n```\n\n**TP=8 performance on Hopper/Blackwell**\nAvoid `-tp 8` with FlashMLA-Sparse. Due to kernel restrictions, TP=8 yields only 16 heads\nper rank but is padded to 64, causing overhead. Prefer TP=2 (Hopper) or TP=1 (Blackwell)\nwith DP/EP mode: `vllm serve deepseek-ai/DeepSeek-V3.2 -dp 8 --enable-expert-parallel`.\n\n**DeepGEMM warmup too slow**\nSet `VLLM_USE_DEEP_GEMM=0` to disable MoE DeepGEMM and skip the long warmup.\n\n## References\n\n- [DeepSeek-V3.2 on Hugging Face](https://huggingface.co/deepseek-ai/DeepSeek-V3.2)\n- [DeepSeek API Documentation](https://api-docs.deepseek.com/)\n- [vLLM Data Parallel Deployment Guide](https://docs.vllm.ai/en/latest/serving/data_parallel_deployment.html)\n", "source": { "source": "vllm", "repo": "vllm-project/vllm", "ref": "main", "sha": "564ac0de1b228568cafcbfbe8ccaccc82d6023eb", "path": "recipes/models/deepseek-ai/DeepSeek-V3.2.yaml", "edit_url": "https://github.com/vllm-project/vllm/edit/main/recipes/models/deepseek-ai/DeepSeek-V3.2.yaml", "overlays": [] } }