# deepseek-ai/DeepSeek-R1 — vLLM serving recipe

> DeepSeek-R1 is a 671B-parameter MoE reasoning model built on the DeepSeek-V3 architecture, trained with large-scale reinforcement learning for strong chain-of-thought capabilities.

Configuration: hardware `h200` (8× H200) · strategy `single_node_tp` (Tensor Parallel) · variant `default` (FP8) · features `tool_calling`, `reasoning` · frontend `rust`

- Interactive page: https://recipes.vllm.ai/deepseek-ai/DeepSeek-R1
- This configuration as JSON: https://recipes.vllm.ai/deepseek-ai/DeepSeek-R1.json?hardware=h200

The commands below were rendered for exactly this configuration and take precedence over any example in the guide.

## Other configurations

Change the query string (the same parameters as the page URL) and fetch this URL again. Unknown or incompatible values fall back to the recipe default, so check the Configuration line.

- `hardware`: `h100`, `h200`, `b200`, `gb200`, `b300`, `gb300`, `mi300x`, `mi325x`, `mi350x`, `mi355x`
- `variant`: `default`, `r1_0528`, `nvfp4`, `mxfp4`, `redhat_nvfp4`
- `strategy`: `single_node_tp`, `single_node_tep`, `single_node_dep`, `single_node_dpa_tp`, `multi_node_tp`, `multi_node_tp_pp`, `multi_node_dep`, `multi_node_tep`, `pd_cluster` (multi-node strategies also need `nodes=<n>`)
- `features`: comma-separated list of `tool_calling`, `reasoning` (default: `tool_calling`, `reasoning`)

## Install

Requires vLLM >= 0.12.0.

```bash
uv venv
source .venv/bin/activate
uv pip install -U vllm --torch-backend auto
```

Docker image: `vllm/vllm-openai:latest`

## Serve

```bash
export VLLM_USE_RUST_FRONTEND=1
vllm serve deepseek-ai/DeepSeek-R1 \
  --trust-remote-code \
  --tensor-parallel-size 8 \
  --enable-expert-parallel \
  --enable-auto-tool-choice \
  --tool-call-parser deepseek_v3 \
  --chat-template examples/tool_chat_template_deepseekr1.jinja \
  --reasoning-parser deepseek_r1
```

Docker equivalent:

```bash
docker run --gpus all \
  --privileged --ipc=host -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -e VLLM_USE_RUST_FRONTEND=1 \
  vllm/vllm-openai:latest deepseek-ai/DeepSeek-R1 \
  --trust-remote-code \
  --tensor-parallel-size 8 \
  --enable-expert-parallel \
  --enable-auto-tool-choice \
  --tool-call-parser deepseek_v3 \
  --chat-template examples/tool_chat_template_deepseekr1.jinja \
  --reasoning-parser deepseek_r1
```

## Guide

> Background reference. Commands in it are examples and may target another configuration — use the Serve section above.

### Overview

DeepSeek-R1 is a 671B-parameter Mixture-of-Experts reasoning model (37B activated per
token) that shares its architecture with DeepSeek-V3, so the same launch recipes apply
to both. DeepSeek publishes a refreshed checkpoint as `DeepSeek-R1-0528`, NVIDIA
publishes an FP4 quantized variant (`nvidia/DeepSeek-R1-FP4`) that runs on Blackwell
GPUs with fewer devices, and AMD publishes an MXFP4 quantized variant
(`amd/DeepSeek-R1-0528-MXFP4-MTP-MoEFP4`) that runs on Instinct GPUs.

### Prerequisites

- **Hardware (FP8)**: 8x H200 (CUDA) or 8x MI300X / MI325X / MI355X (ROCm)
- **Hardware (NVFP4)**: 4x B200 GPUs
- **Hardware (MXFP4)**: 8x MI350X / MI355X (ROCm)
- **vLLM**: Current stable release

#### CUDA
```bash
uv venv
source .venv/bin/activate
uv pip install -U vllm --torch-backend auto
```

#### ROCm (MI300X, MI325X, MI355X)
Requires Python 3.12, ROCm 7.2.1, and glibc >= 2.35.
```bash
uv venv --python 3.12
source .venv/bin/activate
uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/
```

### Client Usage

#### 8xH200 / 8xMI300X (FP8)

Tensor Parallel + Expert Parallel (TP8+EP) — CUDA:
```bash
vllm serve deepseek-ai/DeepSeek-R1-0528 \
  --trust-remote-code \
  --tensor-parallel-size 8 \
  --enable-expert-parallel
```

Tensor Parallel + Expert Parallel (TP8+EP) — ROCm:
```bash
export VLLM_ROCM_USE_AITER=1

vllm serve deepseek-ai/DeepSeek-R1 \
  --trust-remote-code \
  --tensor-parallel-size 8 \
  --enable-expert-parallel
```

Data Parallel + Expert Parallel (DP8+EP) — CUDA:
```bash
vllm serve deepseek-ai/DeepSeek-R1-0528 \
  --trust-remote-code \
  --data-parallel-size 8 \
  --enable-expert-parallel
```

#### 8xMI355X (FP8)

Tensor Parallel (TP8) — ROCm. Validated on MI355X only; INT4 quick reduce and
fused shared experts are not enabled on other AMD GPUs or other strategies:
```bash
export VLLM_ROCM_USE_AITER=1
export VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4
export VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS=1

vllm serve deepseek-ai/DeepSeek-R1 \
  --trust-remote-code \
  --tensor-parallel-size 8
```

#### 8xMI350X / 8xMI355X (DPA+TP)

Each GPU runs its own attention replica with a private KV cache, while the MoE
experts stay tensor-sharded across all eight GPUs. Aggregate KV-cache
capacity is roughly 8x that of a single TP8 replica, useful for workloads 
benefiting from larger KV capacities. 

Engine (all eight ranks in one process, listening on 8100):
```bash
export VLLM_ROCM_USE_AITER=1
export VLLM_ENGINE_READY_TIMEOUT_S=1800

vllm serve deepseek-ai/DeepSeek-R1 \
  --trust-remote-code \
  --port 8100 \
  --data-parallel-size 8 \
  --tensor-parallel-size 1
```

Router (client-facing on 8000, spreads requests across the eight DP ranks):
```bash
uv pip install vllm-router

vllm-router \
  --host 0.0.0.0 \
  --port 8000 \
  --policy consistent_hash \
  --intra-node-data-parallel-size 8 \
  --worker-urls http://localhost:8100 \
  --worker-startup-timeout-secs 1800
```

Notes:

- `--intra-node-data-parallel-size` must match the engine's
  `--data-parallel-size`. A mismatch silently under-uses ranks.
- `consistent_hash` is the policy that makes this worthwhile for chat: it pins
  a conversation to the rank that already holds its prefix, so the larger
  aggregate cache actually gets hit. `round_robin` scatters turns of the same
  conversation across ranks and throws the prefix away.
- Start the engine first, then the router; `--worker-startup-timeout-secs 1800`
  covers the engine's weight load either way.
- Send client traffic to the router on port 8000, never to 8100 directly.

#### 4xB200 (FP4)

On vLLM v0.28.0 and later, the FlashInfer MoE kernels are selected automatically
for both FP4 and FP8 on Blackwell — no environment variables are needed. To pin
the backend explicitly, pass `--moe-backend flashinfer_trtllm`.

Tensor Parallel + Expert Parallel (TP4+EP):
```bash
CUDA_VISIBLE_DEVICES=0,1,2,3 vllm serve nvidia/DeepSeek-R1-FP4 \
  --trust-remote-code \
  --tensor-parallel-size 4 \
  --enable-expert-parallel
```

Data Parallel + Expert Parallel (DP4+EP):
```bash
CUDA_VISIBLE_DEVICES=0,1,2,3 vllm serve nvidia/DeepSeek-R1-FP4 \
  --trust-remote-code \
  --data-parallel-size 4 \
  --enable-expert-parallel
```

#### 8xMI350X / 8xMI355X (MXFP4)

Tensor Parallel + FP8 KV cache (TP8):
```bash
export VLLM_ROCM_USE_AITER=1
export VLLM_ROCM_USE_SKINNY_GEMM=0

vllm serve amd/DeepSeek-R1-0528-MXFP4-MTP-MoEFP4 \
  --trust-remote-code \
  --tensor-parallel-size 8 \
  --kv-cache-dtype fp8
```

### Benchmarking

For benchmarking, prefix caching is disabled by default in vLLM — no extra server
flag is needed.

```bash
# FP8 benchmark
vllm bench serve \
  --model deepseek-ai/DeepSeek-R1-0528 \
  --dataset-name random \
  --random-input-len 8000 \
  --random-output-len 1000 \
  --request-rate 10000 \
  --num-prompts 16 \
  --ignore-eos
```

```bash
# NVFP4 benchmark
vllm bench serve \
  --model nvidia/DeepSeek-R1-FP4 \
  --dataset-name random \
  --random-input-len 8000 \
  --random-output-len 1000 \
  --request-rate 10000 \
  --num-prompts 16 \
  --ignore-eos
```

```bash
# MXFP4 benchmark (MI350X / MI355X)
vllm bench serve \
  --model amd/DeepSeek-R1-0528-MXFP4-MTP-MoEFP4 \
  --dataset-name random \
  --random-input-len 8000 \
  --random-output-len 1000 \
  --request-rate 10000 \
  --num-prompts 16 \
  --ignore-eos
```

Test different workloads by adjusting input/output lengths:
- Prompt-heavy: 8000 input / 1000 output
- Decode-heavy: 1000 input / 8000 output
- Balanced: 1000 input / 1000 output

### Troubleshooting

- **Disaggregated Serving with Wide EP (Experimental GB200)**: See
  [vLLM issue #33583](https://github.com/vllm-project/vllm/issues/33583),
  [the vLLM blog post](https://blog.vllm.ai/2026/02/03/dsr1-gb200-part1.html), and
  [this reference fork](https://github.com/minosfuture/vllm/tree/pd_gb200_0114/runs/DS-R1/fp4)
  for GB200 disaggregated serving recipes.

### References

- [DeepSeek-R1 on Hugging Face](https://huggingface.co/deepseek-ai/DeepSeek-R1)
- [DeepSeek-R1-0528 on Hugging Face](https://huggingface.co/deepseek-ai/DeepSeek-R1-0528)
- [NVIDIA DeepSeek-R1-FP4](https://huggingface.co/nvidia/DeepSeek-R1-FP4)
- [AMD DeepSeek-R1-0528-MXFP4-MTP-MoEFP4](https://huggingface.co/amd/DeepSeek-R1-0528-MXFP4-MTP-MoEFP4)
- [vLLM Expert Parallelism docs](https://docs.vllm.ai/en/latest/serving/expert_parallel_deployment.html)
- [vLLM Data Parallel deployment docs](https://docs.vllm.ai/en/latest/serving/data_parallel_deployment.html)