# deepseek-ai/DeepSeek-V4-Flash-Vision-Exp — vLLM serving recipe

> DeepSeek's first experimental multimodal V4 model — the V4-Flash MoE backbone plus a 32-layer vision tower, 1M context, and a fused DSpark draft module.

Configuration: hardware `gb200` (4× GB200 NVL4) · strategy `single_node_tep` (Tensor + Expert Parallel) · variant `default` (FP8) · features `tool_calling`, `reasoning`, `spec_decoding`

- Interactive page: https://recipes.vllm.ai/deepseek-ai/DeepSeek-V4-Flash-Vision-Exp
- This configuration as JSON: https://recipes.vllm.ai/deepseek-ai/DeepSeek-V4-Flash-Vision-Exp.json?hardware=gb200

The commands below were rendered for exactly this configuration and take precedence over any example in the guide.

## Other configurations

Change the query string (the same parameters as the page URL) and fetch this URL again. Unknown or incompatible values fall back to the recipe default, so check the Configuration line.

- `hardware`: `h100`, `h200`, `b200`, `gb200`, `b300`, `gb300`, `mi300x`, `mi325x`, `mi350x`, `mi355x`, `ascend_950dt_8x`, `ascend_950pr_8x`
- `variant`: `default`
- `strategy`: `single_node_tp`, `single_node_tep`, `single_node_dep`, `multi_node_tep`, `multi_node_dep` (multi-node strategies also need `nodes=<n>`)
- `features`: comma-separated list of `tool_calling`, `reasoning`, `spec_decoding` (default: `tool_calling`, `reasoning`, `spec_decoding`)

## Install

Requires vLLM >= 0.30.0 (not released yet — nightly wheel).

Docker image: `vllm/vllm-openai:deepseekv4-flash-vision`

## Serve

```bash
vllm serve deepseek-ai/DeepSeek-V4-Flash-Vision-Exp \
  --kv-cache-dtype fp8 \
  --block-size 256 \
  --enable-expert-parallel \
  --tensor-parallel-size 4 \
  --tool-call-parser deepseek_v4 \
  --enable-auto-tool-choice \
  --reasoning-parser deepseek_v4 \
  --reasoning-config '{"reasoning_parser":"deepseek_v4","reasoning_start_str":"","reasoning_end_str":""}' \
  --speculative-config '{"method":"dspark","model":"deepseek-ai/DeepSeek-V4-Flash-Vision-Exp","num_speculative_tokens":3,"draft_sample_method":"probabilistic","enable_adaptive_verification":true}'
```

Docker equivalent:

```bash
docker run --gpus all \
  --privileged --ipc=host -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai:deepseekv4-flash-vision deepseek-ai/DeepSeek-V4-Flash-Vision-Exp \
  --kv-cache-dtype fp8 \
  --block-size 256 \
  --enable-expert-parallel \
  --tensor-parallel-size 4 \
  --tool-call-parser deepseek_v4 \
  --enable-auto-tool-choice \
  --reasoning-parser deepseek_v4 \
  --reasoning-config '{"reasoning_parser":"deepseek_v4","reasoning_start_str":"","reasoning_end_str":""}' \
  --speculative-config '{"method":"dspark","model":"deepseek-ai/DeepSeek-V4-Flash-Vision-Exp","num_speculative_tokens":3,"draft_sample_method":"probabilistic","enable_adaptive_verification":true}'
```

## Guide

> Background reference. Commands in it are examples and may target another configuration — use the Serve section above.

### Overview

DeepSeek-V4-Flash-Vision-Exp is DeepSeek's first multimodal model in the V4 family.
It keeps the DeepSeek-V4-Flash language backbone — 43 layers, 256 routed experts with
6 active per token, Compressed Sparse Attention plus manifold-constrained
hyper-connections, a 1,048,576-token context window — and adds a 32-layer / 1024-dim
ViT with a two-layer aligner (~0.5B parameters on top of the 284B backbone). Weights
are **FP4+FP8 mixed**: MoE experts in FP4, the remaining attention / norm / router
params in FP8, vision tower in BF16. The checkpoint is 48 shards / ~168 GB and carries
the fused **DSpark** draft module.

DeepSeek reports it beats DeepSeek-V4-Flash-0731 on multimodal agent benchmarks
(ApexBench 36.5 vs 26.2, Agents' Last Exam 27.3 vs 25.2) while holding text-agent
parity (Terminal Bench 2.1 at 83.9, DeepSWE at 59.3).

### Prerequisites

Vision support is not in a stable vLLM release yet — it lives in
[vllm-project/vllm#54566](https://github.com/vllm-project/vllm/pull/54566). Until that
lands, use the pinned `vllm/vllm-openai:deepseekv4-flash-vision` image; the official
wheel routes this checkpoint to the text-only class and fails on the vision tensors.

- **Weights:** ~168 GB on disk (48 shards). Budget ~202 GB of VRAM before KV cache.
- **Hardware:** NVIDIA (verified on GB200) and AMD MI350X (gfx950);
  XPU still has no vision implementation.
- **Tokenizer / chat template:** nothing to pass. The repo ships no Jinja template
  (prompt encoding lives in `encoding/`); vLLM resolves `--tokenizer-mode` to
  `deepseek_v4` automatically, which is what turns OpenAI content blocks into
  `<｜deepseek_image｜>` placeholders.

### Launching the server

#### Verified configuration — one GB200 NVL4 tray (TP4 + EP)

```bash
export VLLM_FLASHINFER_AUTOTUNE_SKIP_OPS="trtllm_fp4_block_scale_moe,flashinfer::trtllm_fp4_block_scale_moe"

vllm serve deepseek-ai/DeepSeek-V4-Flash-Vision-Exp \
  --tensor-parallel-size 4 \
  --enable-expert-parallel \
  --kv-cache-dtype fp8 \
  --block-size 256 \
  --max-model-len 32768 \
  --speculative-config '{"method":"dspark","model":"deepseek-ai/DeepSeek-V4-Flash-Vision-Exp","num_speculative_tokens":3,"draft_sample_method":"probabilistic","enable_adaptive_verification":true}' \
  --allowed-local-media-path /mnt/lustre
```

Three things there are environment-specific, which is why the builder above leaves
them out:

- `VLLM_FLASHINFER_AUTOTUNE_SKIP_OPS` excludes the FP4 block-scale MoE op from
  FlashInfer's startup autotuning. Export it if autotuning that op fails or costs more
  startup time than it earns back on your FlashInfer build; it is a tuning knob, not a
  requirement. The Advanced row's `--disable-flashinfer-autotune` turns autotuning
  off wholesale instead.
- `--max-model-len 32768` bounds KV cache for the benchmark. The checkpoint advertises
  1M tokens; that is **not** what was measured, and 1M-token KV needs far more memory
  than the weights do. Raise it deliberately, or use the Advanced row's
  `--max-model-len auto`.
- `--allowed-local-media-path` is only needed for `file://` image URLs. Drop it if
  clients send `http(s)://` or base64 data URLs.

#### 8-GPU nodes (H200 / B200 / B300)

Same **Tensor + Expert Parallel** strategy; the builder emits
`--tensor-parallel-size 8 --enable-expert-parallel`. The language backbone is the one
the [DeepSeek-V4-Flash](/deepseek-ai/DeepSeek-V4-Flash) recipe already runs there, but
the vision path has no published run on those GPUs — expect to tune
`--gpu-memory-utilization` and `--max-num-seqs`. The Blackwell
`--attention_config.indexer_kv_dtype mxfp4` / `--moe-backend deep_gemm_mega_moe` tweaks
the text recipe applies are deliberately not set here: the verified vision run did not
use them.

#### AMD MI350X (gfx950)

ROCm vision support is merged (#55107 on the #54566 base); a
`vllm/vllm-openai-rocm:nightly` built on/after 2026-09-11 runs the checkpoint
natively. Three env vars are required:

- `VLLM_ROCM_USE_AITER=1` — the sparse-attention indexer's ROCm path runs only on
  AITER; without it the engine aborts at KV-cache profiling.
- `VLLM_USE_BREAKABLE_CUDAGRAPH=1` — the multimodal wrapper is not torch-compiled, so
  the default `FULL_AND_PIECEWISE` capture cannot attach without breakable graphs.
- `VLLM_USE_V2_MODEL_RUNNER=1` — required for DSpark (the V1 ROCm runner refuses it);
  harmless for plain serving.

Verified single-node launch:

```bash
docker run -d --name dsv4-vision \
  --ipc host --network host --privileged --shm-size 128g \
  --device /dev/kfd --device /dev/dri --group-add video \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -e VLLM_ROCM_USE_AITER=1 \
  -e VLLM_USE_BREAKABLE_CUDAGRAPH=1 \
  -e VLLM_USE_V2_MODEL_RUNNER=1 \
  vllm/vllm-openai-rocm:nightly \
  deepseek-ai/DeepSeek-V4-Flash-Vision-Exp \
  --tensor-parallel-size 2 \
  --kv-cache-dtype fp8 --block-size 256 \
  --gpu-memory-utilization 0.9 \
  --max-model-len 32768
```

Observed backends: AITER Flash-Attention for the ViT, DeepSeek `fp8_ds_mla` KV cache,
FP8 Lightning-Indexer cache, `DEEPSEEK_SPARSE_SWA` (block 256), and the
`ROCM_FLASHMLA_SPARSE_DSV4` MLA backend. TP2 loads 75.2 GiB/card (1.76M-token KV pool
at 131K context); TP4 loads 38.8 GiB/card.

**Stable v0.30.0 and single-card TP1** (verified 2026-10-02 on one MI350X). The stable
`vllm/vllm-openai-rocm:v0.30.0` image (ROCm 7.2.3) already contains #54566 and #55107 and
runs the checkpoint natively — text, vision and DSpark — with the same three env vars. On
v0.30.0 only, also pass `--compilation-config '{"cudagraph_mode":"FULL_DECODE_ONLY"}'`:
v0.30.0's ROCm platform still defaults DeepSeek V4 on gfx950 + ModelRunner V2 to eager
(the #52644 workaround, removed by #57229 after v0.30.0 was branched), so without it decode
runs without CUDA graphs. Batch 1 at TP1: 12 tok/s eager vs 94 tok/s with
`FULL_DECODE_ONLY`, and 137–164 tok/s with DSpark (k=3, adaptive verification off per the
AMD override). Against eager in the same run, a 25-item known-answer check scored 25/25 and
decode-vs-prefill logprobs agreed within eager's range. Nightlies after 2026-09-17 and
v0.31+ don't need the override. TP1 fits one 288 GB card (147.9 GiB of weights).


### Reasoning modes

Same three-tier control as the rest of the V4 family, driven from
`chat_template_kwargs` rather than a flag — `encoding/encoding_dsv4.py` names the
effort levels `low` / `high` / `max`. DeepSeek evaluated this model at
`reasoning_effort: "max"` with `temperature = 1.0, top_p = 0.95`.

```python
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
model = "deepseek-ai/DeepSeek-V4-Flash-Vision-Exp"

resp = client.chat.completions.create(
    model=model,
    messages=[{"role": "user", "content": "What is 17*19? Return only the final integer."}],
    temperature=1.0,
    top_p=0.95,
    extra_body={
        "chat_template_kwargs": {"thinking": True, "reasoning_effort": "max"},
    },
)
print(resp.choices[0].message.reasoning)
print(resp.choices[0].message.content)
```

### Client usage — image input

Standard OpenAI multi-part content. Multiple images per request are allowed with no
cap, so the practical limit is `--max-model-len`.

```python
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")

resp = client.chat.completions.create(
    model="deepseek-ai/DeepSeek-V4-Flash-Vision-Exp",
    messages=[{
        "role": "user",
        "content": [
            {"type": "image_url", "image_url": {"url": "https://example.com/chart.png"}},
            {"type": "text", "text": "Read the y-axis label and the highest bar's value."},
        ],
    }],
    max_tokens=512,
)
print(resp.choices[0].message.content)
```

```bash
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "deepseek-ai/DeepSeek-V4-Flash-Vision-Exp",
    "messages": [{
      "role": "user",
      "content": [
        {"type": "image_url", "image_url": {"url": "https://example.com/chart.png"}},
        {"type": "text", "text": "Describe the image."}
      ]
    }],
    "max_tokens": 512
  }'
```

Image preprocessing is fixed by the checkpoint config (the processor takes no kwargs):
each image costs at most 387 prompt tokens (`vision_max_n_token: 384` plus the
compressor-alignment pad), images below `vision_min_pixels: 147456` are upscaled, and
aspect ratios beyond 8:1 are cropped. An image's exact token count depends on where it
sits in the prompt, since the compressor pads each image to a 4-token block boundary.

### Speculative decoding

DSpark drafts from the module fused into the checkpoint, so `--speculative-config`
names the target repo as its own `model` — there is no separate draft repo to download.
Measured over the full OCRBench run on GB200 (mixed text+image traffic): **2.99**
tokens/forward mean acceptance length, **66.3%** overall acceptance
(83.9% / 66.5% / 50.7% by draft position). Image tokens in the prompt do not degrade
acceptance — the drafter reads image content through the target's hidden states.

Stay at `num_speculative_tokens: 3` unless you verify acceptance yourself; the
checkpoint's trained DSpark block width is 5, but 3 is the depth with published
numbers.

### Benchmarking

OCRBench, full 1000 samples, on the verified GB200 configuration: **835/1000
(83.5%)**, 0 request errors, 312.6 s (~3.2 samples/s). Strongest categories are Key
Information Extraction (92.5%), Doc-oriented VQA (90.0%) and Scene Text VQA (89.5%);
weakest are handwritten math (49.0%) and handwriting recognition (58.0%).

For throughput:

```bash
vllm bench serve \
  --model deepseek-ai/DeepSeek-V4-Flash-Vision-Exp \
  --dataset-name random \
  --random-input-len 4096 \
  --random-output-len 1024 \
  --num-prompts 200
```

### Limitations

- **Pre-release.** Flags and defaults can change before #54566 lands; re-check before
  deploying, and keep the image pinned.
- **XPU has no vision path** — it raises `NotImplementedError` at model construction.
- **GB200 (NVIDIA) and MI350X (AMD) are the hardware with a verified vision run.**
  Every other pill is the repo's fail-open default, not a tested claim.
- **PD disaggregation is not offered**, and the CPU/filesystem KV-offload pills stay
  off — neither has a verified run to record. The Mooncake KV-store pills follow the
  repo's fail-open default and are offered, but untested here.
- **1M context is advertised, not measured.** The reference run used 32K.

### References

- [Model card](https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-Vision-Exp)
- [vLLM support PR #54566](https://github.com/vllm-project/vllm/pull/54566)
- [vLLM ROCm vision PR #55107](https://github.com/vllm-project/vllm/pull/55107)
- [DeepSeek-V4-Flash recipe](/deepseek-ai/DeepSeek-V4-Flash) — the text-only sibling
- [DeepSpec (DSpark)](https://github.com/deepseek-ai/DeepSpec)