# deepseek-ai/DeepSeek-V4-Flash — vLLM serving recipe

> DeepSeek V4 MoE model with hybrid CSA+HCA attention, manifold-constrained hyper-connections, and three-tier reasoning (Non-think / Think High / Think Max).

Configuration: hardware `h200` (8× H200) · strategy `single_node_tep` (Tensor + Expert Parallel) · variant `default` (FP8 (0731)) · features `tool_calling`, `reasoning`, `spec_decoding` · frontend `rust`

- Interactive page: https://recipes.vllm.ai/deepseek-ai/DeepSeek-V4-Flash
- This configuration as JSON: https://recipes.vllm.ai/deepseek-ai/DeepSeek-V4-Flash.json?hardware=h200

The commands below were rendered for exactly this configuration and take precedence over any example in the guide.

## Other configurations

Change the query string (the same parameters as the page URL) and fetch this URL again. Unknown or incompatible values fall back to the recipe default, so check the Configuration line.

- `hardware`: `h100`, `h200`, `b200`, `gb200`, `b300`, `gb300`, `dgx_station_gb300`, `dgx_spark_gb10`, `rtx_pro_6000_8x`, `mi300x`, `mi325x`, `mi355x`, `ascend_950dt_8x`, `ascend_950pr_8x`
- `variant`: `default`, `fp8`, `nvfp4`, `redhat_nvfp4`, `dspark`, `ascend_w8a8`
- `strategy`: `single_node_tp`, `single_node_tep`, `single_node_dep`, `multi_node_dep`, `multi_node_tp`, `pd_cluster` (multi-node strategies also need `nodes=<n>`)
- `features`: comma-separated list of `tool_calling`, `reasoning`, `spec_decoding` (default: `tool_calling`, `reasoning`, `spec_decoding`)

## Install

Requires vLLM >= 0.20.0.

```bash
uv venv
source .venv/bin/activate
uv pip install -U vllm --torch-backend auto
```

Docker image: `vllm/vllm-openai:v0.28.0`

## Serve

```bash
export VLLM_USE_RUST_FRONTEND=1
vllm serve deepseek-ai/DeepSeek-V4-Flash-0731 \
  --trust-remote-code \
  --kv-cache-dtype fp8 \
  --block-size 256 \
  --enable-expert-parallel \
  --tensor-parallel-size 8 \
  --tokenizer-mode deepseek_v4 \
  --tool-call-parser deepseek_v4 \
  --enable-auto-tool-choice \
  --reasoning-parser deepseek_v4 \
  --reasoning-config '{"reasoning_parser":"deepseek_v4","reasoning_start_str":"","reasoning_end_str":""}' \
  --speculative-config '{"method":"dspark","num_speculative_tokens":7,"draft_sample_method":"probabilistic"}'
```

Docker equivalent:

```bash
docker run --gpus all \
  --privileged --ipc=host -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -e VLLM_USE_RUST_FRONTEND=1 \
  vllm/vllm-openai:v0.28.0 deepseek-ai/DeepSeek-V4-Flash-0731 \
  --trust-remote-code \
  --kv-cache-dtype fp8 \
  --block-size 256 \
  --enable-expert-parallel \
  --tensor-parallel-size 8 \
  --tokenizer-mode deepseek_v4 \
  --tool-call-parser deepseek_v4 \
  --enable-auto-tool-choice \
  --reasoning-parser deepseek_v4 \
  --reasoning-config '{"reasoning_parser":"deepseek_v4","reasoning_start_str":"","reasoning_end_str":""}' \
  --speculative-config '{"method":"dspark","num_speculative_tokens":7,"draft_sample_method":"probabilistic"}'
```

## Guide

> Background reference. Commands in it are examples and may target another configuration — use the Serve section above.

### Overview

DeepSeek-V4-Flash is a 284B-total / 13B-active MoE model in the V4 preview family.
It pairs a **hybrid attention stack** — Compressed Sparse Attention (CSA) + Heavily
Compressed Attention (HCA) — with **Manifold-Constrained Hyper-Connections (mHC)** to
reach 27% of V3.2's per-token inference FLOPs and 10% of V3.2's KV cache at 1M
context. Pre-trained on 32T+ tokens; post-training is a two-stage pipeline (domain-
specific expert cultivation + unified consolidation via on-policy distillation).

Checkpoint is **FP4+FP8 mixed**: MoE expert weights are stored in FP4 while the
remaining (attention / norm / router) params stay in FP8.

An **NVFP4 variant** (`nvidia/DeepSeek-V4-Flash-NVFP4`) is also available — NVIDIA
modelopt re-quantizes the MoE experts to standard NVFP4 while attention, shared
experts, router head, and MTP stay FP8. Pick it from the **Variant** row; it runs on
Blackwell GPUs with the FP4 indexer cache. Unlike the native FP8 checkpoint, the
NVFP4 experts don't support the `deep_gemm_mega_moe` MoE kernel (FP8-only), so it
runs on the default MoE backend.

### Checkpoints

Four checkpoints are on the **Variant** row. They differ in *which weights you serve*
and *which draft module ships with them* — the speculative method itself is picked on
the **Spec Decoding** row.

| Variant | Repo | Draft | Notes |
| :-- | :-- | :-- | :-- |
| **FP8 (0731)** (default) | `deepseek-ai/DeepSeek-V4-Flash-0731` | DSpark | Official release, new weights + DSpark |
| **FP8 (Preview)** | `deepseek-ai/DeepSeek-V4-Flash` | MTP | Preview FP4+FP8 mixed weights |
| **NVFP4** | `nvidia/DeepSeek-V4-Flash-NVFP4` | MTP | modelopt re-quant, Blackwell |
| **DSpark** | `deepseek-ai/DeepSeek-V4-Flash-DSpark` | DSpark | Preview weights + fused DSpark module |

**0731 is the official DeepSeek-V4-Flash release** and the default here, superseding
the preview with substantially stronger agentic capability — 82.7 on Terminal Bench
2.1 and 54.4 on DeepSWE, versus 61.8 and 7.3 for the preview. It beats
DeepSeek-V4-Pro (Preview) on every agentic benchmark DeepSeek published despite its
far smaller activated parameter count. The preview weights remain available as the
**FP8 (Preview)** variant for reproducing earlier results.

**DSpark** is *not* new weights — it is the preview checkpoint with a speculative
decoding module attached (see [DeepSpec](https://github.com/deepseek-ai/DeepSpec)).
Both fused checkpoints add a `dspark_*` block to `config.json`; the preview and NVFP4
checkpoints have no such block, which is why the DSpark method is offered only on the
two variants that carry the draft. Selecting either one auto-enables **Spec Decoding**
with `method: dspark`, emitting:

```
--speculative-config '{"method":"dspark","num_speculative_tokens":7,"draft_sample_method":"probabilistic"}'
```

The fused checkpoints are ~167 GB on disk versus ~160 GB for the preview — the draft
module is the difference. Both require **vLLM 0.25.0**: DSpark drafting itself landed
in 0.25.0, but ROCm support for it (MI325X / MI355X) only shipped in 0.26.0. On
NVIDIA-only deployments 0.25.0 is sufficient, except the 0731 checkpoint
on RTX PRO 6000 (sm_120) with DSpark, which needs `vllm/vllm-openai:nightly`
and a FlashInfer build containing flashinfer-ai/flashinfer#4380.

### Reasoning modes

The chat template exposes three reasoning-effort modes:

- **Non-think** — fast, intuitive responses.
- **Think High** — explicit chain-of-thought for logical analysis and planning.
- **Think Max** — maximum reasoning effort; requires `--max-model-len >= 393216`
  (384K tokens) to avoid truncation.

Recommended sampling: `temperature = 1.0`, `top_p = 1.0`.

On the **0731** variant the effort levels are named `low` / `high` / `max`, and
DeepSeek recommends `top_p = 0.95` for agentic scenarios (`1.0` otherwise) with
`temperature = 1.0`. Allow up to **384K output tokens** at the `high` and `max`
levels.

Note that neither fused checkpoint ships a Jinja chat template — the repos provide an
`encoding/` folder with `encode_messages` / `parse_message_from_completion_text`
helpers instead. Serving through vLLM with `--tokenizer-mode deepseek_v4` (the
**Tool Calling** pill) applies the built-in DeepSeek-V4 encoding, so the OpenAI-
compatible endpoint works without the helper scripts.

#### OpenAI Client Example

For DeepSeek-V4, keep reasoning controls in `chat_template_kwargs`, as it exposes a
custom **Think Max** mode via `"reasoning_effort": "max"`.

```python
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
model = "deepseek-ai/DeepSeek-V4-Flash"
messages = [{"role": "user", "content": "What is 17*19? Return only the final integer."}]

# Non-think
resp = client.chat.completions.create(
    model=model,
    messages=messages,
)

# Think High
resp = client.chat.completions.create(
    model=model,
    messages=messages,
    extra_body={
        "chat_template_kwargs": {
            "thinking": True,
            "reasoning_effort": "high",
        },
    },
)

# Think Max
resp = client.chat.completions.create(
    model=model,
    messages=messages,
    extra_body={
        "chat_template_kwargs": {
            "thinking": True,
            "reasoning_effort": "max",
        },
    },
)
```

### Recommended deployment

Non-disaggregated serving on multi-GPU supported hardware: single-node DP + EP with
`--data-parallel-size 4`. Fills a GB200 NVL4 tray exactly; uses 4 of 8 GPUs per
replica on H200/B200/B300 (leaving headroom for throughput-vs-latency tuning).
On DGX Station, use the single-GPU launch below. For disaggregated prefill/decode
on GB200, use the PD Cluster tab.

#### H100 (8×80 GB)

The FP8 (0731) and DSpark checkpoints are selectable on H100. A reported
8×H100 TP8 + EP deployment of 0731 served requests with vLLM 0.26.0, while
the same deployment failed during startup on 0.27.0, both with and without
DSpark. See [vLLM #52065](https://github.com/vllm-project/vllm/issues/52065).

For the separate DSpark checkpoint, a 4×H100 TP4 report on vLLM 0.25.0
reached inference after removing `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True`
and setting `--gpu-memory-utilization` below 0.9. A 2×H100 NVL report also
ran the checkpoint with DSpark k=5 on a custom vLLM build. These reports
establish Hopper compatibility, but do not verify this recipe's generated
H100 topology or current image. Check startup and inference before production.
See [vLLM #49101](https://github.com/vllm-project/vllm/issues/49101) and
[the H100 NVL report](https://github.com/vllm-project/vllm/pull/46995#issuecomment-4835994467).

#### DGX Spark (GB10) cluster

DGX Spark requires the community Spark vLLM build for this model rather than the
stock vLLM release/nightly image. The **DGX Spark (GB10)** hardware pill pins the
Docker tab to `eugr/spark-vllm-b12x:latest`, whose build carries the B12X
backends and Spark-specific kernels for the GB10 / SM12x family.

Use **2 nodes**. A single GB10's 128 GB unified memory is below the FP8/NVFP4
checkpoint footprint, so the recipe grants DGX Spark a Spark-only **Multi-Node TP**
layout (`--tensor-parallel-size 2`, one GB10 per node). The generated command
also sets the B12X runtime environment and the Spark network environment; export
`IFACE_NAME`, `IB_IF` and `HEAD_IP` on each node before launching.

To identify the values, first list the ConnectX Ethernet interfaces, RoCE
devices and the IP address of the UP ConnectX interface on each Spark:

```bash
$ ibdev2netdev
roceP2p1s0f0 port 1 ==> enP2p1s0f0np0 (Up)
roceP2p1s0f1 port 1 ==> enP2p1s0f1np1 (Down)
rocep1s0f0 port 1 ==> enp1s0f0np0 (Up)
rocep1s0f1 port 1 ==> enp1s0f1np1 (Down)

$ ip addr show enp1s0f0np0
3: enp1s0f0np0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 4c:62:56:c1:83:68 brd ff:ff:ff:ff:ff:ff
    inet 100.100.100.10/24 brd 100.100.100.255 scope global noprefixroute enp1s0f0np0
      valid_lft forever preferred_lft forever
```

From above example values, we can get the following common values:

```bash
# Common values for all nodes
HEAD_IP="100.100.100.10"         # IP of the head node's UP ConnectX interface
IFACE_NAME="enp1s0f0np0"         # The head node's UP ConnectX Ethernet interface
IB_IF="rocep1s0f0,roceP2p1s0f0"  # Comma-separated RoCE device name list corresponding to the UP ConnectX interface
```

#### DGX Station Single-GPU

```bash
vllm serve deepseek-ai/DeepSeek-V4-Flash \
  --tensor-parallel-size 1 --pipeline-parallel-size 1 \
  --kv-cache-dtype fp8 --trust-remote-code --block-size 256 \
  --gpu-memory-utilization 0.92 \
  --compilation-config '{"cudagraph_mode":"FULL_AND_PIECEWISE","custom_ops":["all"]}' \
  --attention_config.indexer_kv_dtype mxfp4 \
  --tokenizer-mode deepseek_v4 --tool-call-parser deepseek_v4 \
  --enable-auto-tool-choice --reasoning-parser deepseek_v4 \
  --max-cudagraph-capture-size 128 \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}'
```

#### RTX PRO 6000 8× (8×96 GB, sm_120)

Single-node TP8 + expert parallelism on the **0731** checkpoint. Serving
without speculative decoding was verified on 8× RTX PRO 6000 (PCIe, no
NVLink). DSpark on **0731** needs `vllm/vllm-openai:nightly` (or a build
that includes vLLM #51538) plus a FlashInfer that instantiates the SM120
sparse-MLA decode kernel for `topk=192`. The Preview, NVFP4, and DSpark
variant checkpoints are not verified on this profile.

sm_120 cannot use the SM100 FP4 indexer cache or `deep_gemm_mega_moe`.
The generated command already sets `--attention_config.indexer_kv_dtype
fp8` and `--moe-backend auto`. Do not use `method: mtp` on 0731 — that
checkpoint has no MTP head; Spec Decoding defaults to DSpark.

Host-side FlashInfer install before launching 0731 with DSpark (`--no-deps`
is required so pip does not replace vLLM's torch):

```bash
# Trace: flashinfer-ai/flashinfer#4380
# merge: 24d7dfb2639083c5a4d418881099421fc800b7bb (2026-08-08)
# wheel: flashinfer-python==0.6.18.dev20260814
# index: https://flashinfer.ai/whl/nightly/
/usr/bin/python3.12 -m pip install -U --pre flashinfer-python \
--index-url https://flashinfer.ai/whl/nightly/ --no-deps
/usr/bin/python3.12 -m pip install flashinfer-python
/usr/bin/python3.12 -m pip install -U --pre flashinfer-cubin \
--index-url https://flashinfer.ai/whl/nightly/
/usr/bin/python3.12 -m pip install -U --pre flashinfer-jit-cache \
--index-url https://flashinfer.ai/whl/nightly/cu130

vllm serve deepseek-ai/DeepSeek-V4-Flash-0731 \
--trust-remote-code \
--kv-cache-dtype fp8 \
--block-size 256 \
--enable-expert-parallel \
--tensor-parallel-size 8 \
--moe-backend auto \
--attention_config.indexer_kv_dtype fp8 \
--tokenizer-mode deepseek_v4 \
--tool-call-parser deepseek_v4 \
--enable-auto-tool-choice \
--reasoning-parser deepseek_v4 \
--reasoning-config '{"reasoning_parser":"deepseek_v4","reasoning_start_str":"","reasoning_end_str":""}' \
--speculative-config '{"method":"dspark","num_speculative_tokens":7,"draft_sample_method":"probabilistic"}'
```

#### MI300X (8×192GB)

Verified single-node serving on MI300X (8×192 GB, CDNA 3) with the same AMD
launch as the MI325X/MI355X sections below — the builder lowers
`--gpu-memory-utilization` to `0.85` on MI300X (the 192 GB cards leave less
headroom at `0.9`), and DSpark speculative decoding is validated on it.

#### MI325X (1×256GB)

Validated end-to-end with vLLM 0.25.0 and PyTorch HIP 7.2.53211. Use this
conservative TP1 eager-mode launch instead of the MI355X throughput settings:

```bash
export VLLM_ROCM_USE_AITER=1

vllm serve deepseek-ai/DeepSeek-V4-Flash \
  --host 0.0.0.0 \
  --port 8002 \
  --tensor-parallel-size 1 \
  --kv-cache-dtype fp8_e4m3 \
  --max-model-len 4096 \
  --enable-chunked-prefill \
  --max-num-batched-tokens 256 \
  --kv-cache-memory-bytes 10000000000 \
  --distributed-executor-backend mp \
  --trust-remote-code \
  --tokenizer-mode deepseek_v4 \
  --moe-backend triton_unfused \
  --enforce-eager
```

Chunked prefill allows 4K requests while scheduling at most 256 tokens per
batch. This configuration loaded the 148.66 GiB checkpoint and passed model-list
and chat probes in non-thinking and Think High modes during a 24-hour allocation.
Only TP1 at 4K context is verified; TP2+ did not validate successfully. Think Max
requires at least 384K context and is not available with this configuration.

#### MI355X (4×288GB)

If you want to further lower the latency, you can run with `--tensor-parallel-size 8`.

```bash
export VLLM_ROCM_USE_AITER=1

vllm serve deepseek-ai/DeepSeek-V4-Flash \
  --host localhost \
  --port 8001 \
  --dtype auto \
  --kv-cache-dtype fp8 \
  --tensor-parallel-size 4 \
  --max-num-seqs 512 \
  --max-num-batched-tokens 8192 \
  --distributed-executor-backend mp \
  --trust-remote-code \
  --gpu-memory-utilization 0.9 \
  --tokenizer-mode deepseek_v4 \
  --reasoning-parser deepseek_v4 \
  --tool-call-parser deepseek_v4 \
  --enable-auto-tool-choice \
  --compilation-config '{"mode": 3, "cudagraph_mode": "FULL_DECODE_ONLY"}'
```

##### MI355X is validated on GSM8K dataset:

###### Launch command

```bash
MODEL=deepseek-ai/DeepSeek-V4-Flash
lm_eval --model local-completions \
  --model_args model=$MODEL,base_url=http://0.0.0.0:8001/v1/completions,num_concurrent=128,max_retries=10,max_gen_toks=2048,timeout=60000 \
  --batch_size auto \
  --tasks gsm8k \
  --num_fewshot 8 \
  --output_path . 2>&1 | tee -a eval.log
```

###### Reported result

```text
local-completions ({'model': 'deepseek-ai/DeepSeek-V4-Flash', 'base_url': 'http://0.0.0.0:8001/v1/completions', 'num_concurrent': 128, 'max_retries': 10, 'max_gen_toks': 2048, 'timeout': 60000}), gen_kwargs: ({}), limit: None, num_fewshot: 8, batch_size: auto
|Tasks|Version|     Filter     |n-shot|  Metric   |   |Value |   |Stderr|
|-----|------:|----------------|-----:|-----------|---|-----:|---|-----:|
|gsm8k|      3|flexible-extract|     8|exact_match|↑  |0.9439|±  |0.0063|
|     |       |strict-match    |     8|exact_match|↑  |0.9431|±  |0.0064|
```

#### H200 Single-Node PD (Mooncake)

Single-host disaggregated serving: 4 prefill GPUs + 4 decode GPUs on one 8-GPU H200 node,
using MooncakeConnector over RDMA for KV cache transfer.

**Prefill** (GPUs 0–3, port 8000):

```bash
docker run --gpus all \
  --privileged --ipc=host -p 8000:8000 \
  --network host \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -v /mnt/shared:/mnt/shared \
  -e TILELANG_CLEANUP_TEMP_FILES=1 \
  -e VLLM_DISABLE_COMPILE_CACHE=1 \
  -e VLLM_ENGINE_READY_TIMEOUT_S=3600 \
  -e VLLM_LOG_STATS_INTERVAL=1 \
  -e VLLM_MOONCAKE_BOOTSTRAP_PORT=8998 \
  -e CUDA_VISIBLE_DEVICES=0,1,2,3 \
  vllm/vllm-openai:v0.28.0 \
  deepseek-ai/DeepSeek-V4-Flash \
  --trust-remote-code \
  --kv-cache-dtype fp8 \
  --block-size 256 \
  --port 8000 \
  --data-parallel-size 4 \
  --enable-expert-parallel \
  --tokenizer-mode deepseek_v4 \
  --reasoning-parser deepseek_v4 \
  --max-model-len auto \
  --max-num-batched-tokens 16384 \
  --max-num-seqs 8 \
  --no-disable-hybrid-kv-cache-manager \
  --disable-uvicorn-access-log \
  --kv-transfer-config '{"kv_connector":"MooncakeConnector","kv_role":"kv_both","kv_load_failure_policy":"fail","kv_buffer_device":"cuda","kv_connector_extra_config":{"enforce_handshake_compat":false,"mooncake_protocol":"rdma"}}'
```

**Decode** (GPUs 4–7, port 8001):

```bash
docker run --gpus all \
  --privileged --ipc=host -p 8001:8001 \
  --network host \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -v /mnt/shared:/mnt/shared \
  -e TILELANG_CLEANUP_TEMP_FILES=1 \
  -e VLLM_DISABLE_COMPILE_CACHE=1 \
  -e VLLM_ENGINE_READY_TIMEOUT_S=3600 \
  -e VLLM_LOG_STATS_INTERVAL=1 \
  -e VLLM_MOONCAKE_BOOTSTRAP_PORT=9889 \
  -e CUDA_VISIBLE_DEVICES=4,5,6,7 \
  vllm/vllm-openai:v0.28.0 \
  deepseek-ai/DeepSeek-V4-Flash \
  --trust-remote-code \
  --kv-cache-dtype fp8 \
  --block-size 256 \
  --port 8001 \
  --data-parallel-size 4 \
  --enable-expert-parallel \
  --tokenizer-mode deepseek_v4 \
  --reasoning-parser deepseek_v4 \
  --max-model-len auto \
  --max-num-seqs 512 \
  --max-num-batched-tokens 512 \
  --compilation-config '{"mode":0,"cudagraph_mode":"FULL_DECODE_ONLY","max_cudagraph_capture_size":512,"compile_ranges_endpoints":[512]}' \
  --no-disable-hybrid-kv-cache-manager \
  --disable-uvicorn-access-log \
  --kv-transfer-config '{"kv_connector":"MooncakeConnector","kv_role":"kv_both","kv_load_failure_policy":"fail","kv_buffer_device":"cuda","kv_connector_extra_config":{"enforce_handshake_compat":false,"mooncake_protocol":"rdma"}}'
```

**[Router](https://github.com/vllm-project/router)**:

```bash
pip install vllm-router

vllm-router --policy round_robin \
  --vllm-pd-disaggregation \
  --prefill http://localhost:8000 \
  --decode http://localhost:8001 \
  --host 127.0.0.1 \
  --port 30000 \
  --intra-node-data-parallel-size 4 \
  --kv-connector mooncake
```

#### H200 Single-Node PD (Nixl)

Single-host disaggregated serving: 4 prefill GPUs + 4 decode GPUs on one 8-GPU H200 node,
using NixlConnector for KV cache transfer.

**Prefill** (GPUs 0–3, port 8000):

```bash
docker run --gpus all \
  --privileged --ipc=host -p 8000:8000 \
  --network host \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -v /mnt/shared:/mnt/shared \
  -e TILELANG_CLEANUP_TEMP_FILES=1 \
  -e VLLM_DISABLE_COMPILE_CACHE=1 \
  -e VLLM_ENGINE_READY_TIMEOUT_S=3600 \
  -e VLLM_LOG_STATS_INTERVAL=1 \
  -e VLLM_NIXL_SIDE_CHANNEL_PORT=5557 \
  -e CUDA_VISIBLE_DEVICES=0,1,2,3 \
  vllm/vllm-openai:v0.28.0 \
  deepseek-ai/DeepSeek-V4-Flash \
  --trust-remote-code \
  --kv-cache-dtype fp8 \
  --block-size 256 \
  --port 8000 \
  --data-parallel-size 4 \
  --enable-expert-parallel \
  --tokenizer-mode deepseek_v4 \
  --reasoning-parser deepseek_v4 \
  --max-model-len auto \
  --max-num-batched-tokens 16384 \
  --max-num-seqs 8 \
  --no-disable-hybrid-kv-cache-manager \
  --disable-uvicorn-access-log \
  --kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_both"}'
```

**Decode** (GPUs 4–7, port 8001):

```bash
docker run --gpus all \
  --privileged --ipc=host -p 8001:8001 \
  --network host \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -v /mnt/shared:/mnt/shared \
  -e TILELANG_CLEANUP_TEMP_FILES=1 \
  -e VLLM_DISABLE_COMPILE_CACHE=1 \
  -e VLLM_ENGINE_READY_TIMEOUT_S=3600 \
  -e VLLM_LOG_STATS_INTERVAL=1 \
  -e VLLM_NIXL_SIDE_CHANNEL_PORT=5558 \
  -e CUDA_VISIBLE_DEVICES=4,5,6,7 \
  vllm/vllm-openai:v0.28.0 \
  deepseek-ai/DeepSeek-V4-Flash \
  --trust-remote-code \
  --kv-cache-dtype fp8 \
  --block-size 256 \
  --port 8001 \
  --data-parallel-size 4 \
  --enable-expert-parallel \
  --tokenizer-mode deepseek_v4 \
  --reasoning-parser deepseek_v4 \
  --max-model-len auto \
  --max-num-seqs 512 \
  --max-num-batched-tokens 512 \
  --compilation-config '{"mode":0,"cudagraph_mode":"FULL_DECODE_ONLY","max_cudagraph_capture_size":512,"compile_ranges_endpoints":[512]}' \
  --no-disable-hybrid-kv-cache-manager \
  --disable-uvicorn-access-log \
  --kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_both"}'
```

**[Router](https://github.com/vllm-project/router)**:

```bash
pip install vllm-router

vllm-router --policy round_robin \
  --vllm-pd-disaggregation \
  --prefill http://localhost:8000 \
  --decode http://localhost:8001 \
  --host 127.0.0.1 \
  --port 30000 \
  --intra-node-data-parallel-size 4 \
  --kv-connector nixl
```