deepseek-ai/DeepSeek-V4.1-Flash

Overview

DeepSeek-V4.1 is a vision-language Mixture-of-Experts model: 552B backbone parameters plus 196B of Engram memory, 8B active per prompt token and 16B per output token, 40 transformer layers at hidden size 5120, with a 32-layer ViT and aligner in front of the text stack. Six things distinguish it from V4:

• Causal encoder-decoder. The 40 layers split into a 20-layer encoder and a 20-layer decoder, and the decoder's global KV is projected from the encoder's final hidden state instead of being computed layer by layer. Prefill therefore only runs the encoder half, which is where the 8B-prefill / 16B-decode split comes from.
• Two-tier sparse attention. Every layer attends over a 128-token sliding window. Layers that carry a compression ratio add compressed KV latents reaching further back, pooled by a learned softmax gate. A small side attention — the indexer, inherited from V3.2-Exp — scores those latents and keeps the best 512 per query, pre-filtered by a candidate stage that selects 2048 blocks of 8. Only four layers (2, 8, 14, 20) actually compress their own KV; the rest read that cache. V4.1 uses only compression ratios 1 and 2, where V4 used 4 and 128.
• Engram n-gram memory. Layers 1 and 14 each own a hash table of ~384M rows x 256 dims, looked up by 2-, 3- and 4-gram hashes of the input and written into the residual stream through a learned gate. These two tables alone are 196.6B parameters (~183 GiB) — plan capacity for them, they dominate everything except the experts.
• Hyper-Connections. The residual stream is carried as 4 parallel copies; each sublayer derives its own pre/post/combine coefficients from the stream, with the combine matrix made doubly stochastic by 20 Sinkhorn iterations.
• DSpark draft head. Three stages (128 routed experts each, 3 activated) draft a block of 5 tokens, reading the attention input of layers 37-39, with a Markov bias head and a confidence head on the last stage.
• Mixed MXFP4/MXFP8 checkpoint. Routed expert weights are MXFP4; everything else is MXFP8 block-quantized, UE8M0 scales throughout. Embedding and LM head are BF16.

Routing is 6 of 384 experts per token plus one shared expert, scored with sqrtsoftplus and a noaux_tc bias — with a separate routing bias for tokens inside an image span, so vision and text tokens do not compete for the same experts.

Context length

1,048,576 tokens, reached by YaRN with factor 16 over a 65,536-token training window. Compressed KV rotates at its own RoPE theta (160,000) because one latent stands for several tokens, so its positions are further apart than the raw stream's.

Images

Images enter the prompt as <|deepseek_image|> spans; every position in the span carries the image token id, and the per-position role (start / newline / end / image) comes from the processor. The vision tower is a 32-layer ViT at hidden size 1024, patch 14, with a 3x downsampling aligner and a cap of 1024 tokens per image (minimum 295,936 pixels). There is no limit on images per prompt. Merged embeddings enter the text model as inputs_embeds, before the hyper-connection stream expansion, while raw token ids still flow through so the router can apply the image routing bias.

Tick Encoder parallel to run the ViT data-parallel (--mm-encoder-tp-mode data) instead of tensor-parallel: at 32 layers / hidden 1024 the encoder is small enough that TP communication costs more than it saves, which can significantly reduce TTFT for multi-image requests. It is mutually exclusive with Text only.

Reasoning and tool calling

Two thinking modes and a numeric reasoning budget rather than discrete tiers. Send them as chat_template_kwargs:

key	values
thinking / enable_thinking	boolean; if you send both they must agree
reasoning_effort	low (25), high (50), xhigh (75), max (100), or an integer 1-100

The top-level OpenAI reasoning_effort field also works, where "none" turns thinking off. "minimal" and "medium" are rejected — they are not part of this model's set.

Note that the DeepSeek API maps its low / high / max tiers to 50 / 75 / 100, while the open-source prompt encoder (and therefore vLLM) maps low / high / xhigh / max to 25 / 50 / 75 / 100. A request tuned at high on the API runs at effort 50 here; pass an integer if you need parity.

With both keys unset, thinking is ON at effort 50. The do-nothing config is the most verbose one, so a request with a small max_tokens spends its budget on the trace and returns empty content with finish_reason=length. That reads as a broken model and is not. Either pin thinking: false or give the budget room.

In thinking mode the budget is rendered into the prompt as a Reasoning Effort: N prefix on the first turn only.

resp = client.chat.completions.create(
    model="deepseek-ai/DeepSeek-V4.1-Flash",
    messages=[{"role": "user", "content": "What is 17*19?"}],
    extra_body={"chat_template_kwargs": {"thinking": True, "reasoning_effort": 25}},
)

DeepSeek's published numbers use temperature=1.0, top_p=0.95 and effort 100 with max_tokens of at least 256K. The tech report notes that efforts of 60-80 recover most of that accuracy at well under half the tokens, so reserve max for hard tasks.

Tool calls are wrapped in DSML tag blocks rather than JSON fences, and tool output comes back in <tool_result> tags.

Speculative decoding

DSpark is the only speculative method for this checkpoint. V4.1 dropped the MTP module that V3 and V4 trained alongside the backbone. Speculative decoding enables a 5-token block. On NVIDIA it also turns on adaptive verification (see the feature description). On AMD, vLLM currently refuses adaptive verification. The generated command sets enable_adaptive_verification:false and verifies the full block. Two checks in maybe_create_adaptive_verification_manager fail on ROCm: the indexer helper supports_device_cpu_query_lens_mismatch() is False, and DeepseekV41ROCMAiterSparseSWABackend reports AttentionCGSupport.UNIFORM_BATCH rather than ALWAYS. Remove the AMD override only after a serve with the true flag starts. Acceptance depends on the workload, so measure it on your own traffic before sizing a deployment around it. The drafter's experts add roughly 14B parameters to the load.

Serving text-only

Tick Text only to add --language-model-only, which skips the vision encoder entirely. Worth it whenever the workload is text: it drops the ViT and aligner from the load and frees that VRAM for KV cache. It is mutually exclusive with encoder_parallel. The verified GB200 runs (TP4 and 1P1D) were text-only.

Prefill/Decode disaggregation

The Prefill/Decode Disaggregation strategy is the verified 1P1D layout on GB200 NVL4: one tray (4 GPUs) per role, TP4 in each pool, KV handed over through NIXL, fronted by vllm-router --vllm-pd-disaggregation. Both pools disable FlashInfer autotune plus JIT and CuTeDSL warmup via --kernel-config, skip the DeepGEMM warmup (VLLM_DEEP_GEMM_WARMUP=skip), and cap --max-num-seqs at 32. On 8-GPU nodes the same layout becomes TP8 per role.

With Speculative decoding on, DSpark runs in both pools so the transferred KV stays compatible.

Memory

The checkpoint is roughly 511 GB on disk (476 GiB), which breaks down as:

Component	Entries	Stored
Routed + DSpark experts (MXFP4)	557.2B	259.5 GiB
Engram tables (FP8)	196.6B	183.1 GiB
Attention, dense projections, routers (FP8)	7.4B	6.9 GiB
Embedding + LM head (BF16), norms (FP32)	2.0B	3.9 GiB
UE8M0 block scales	23.6B	21.9 GiB

DeepSeek's 552B backbone figure covers the routed experts, attention and embeddings; the Engram tables, the ~14B DSpark drafter and the block scales sit outside it.

vram_minimum_gb: 614 is that total times the schema's 1.2 headroom factor. It fits one GB200 NVL4 tray (768 GB) at TP4, or one 8-GPU H200 node (1128 GB) with room for KV cache.

KV cache is a small share of that budget. The compressed latents are shared across layers and trained to be stored in FP4, and DeepSeek puts the resulting global KV at 890 bytes per token, about a quarter of V4-Flash; by that accounting a full 1M-token prompt holds under 1 GB of global KV, plus a fixed 128-token sliding window per layer. Weights and batch size, not cache, set the capacity limit — still, measure before assuming a given context/batch fits.

Prerequisites

• An image (vLLM 0.30.0+); no pip wheel serves this architecture, so the Install block only offers Docker. On NVIDIA that is vllm/vllm-openai:nightly: the architecture landed on main in vllm-project/vllm#56228, so any nightly from 2026-09-10 on serves it, and the launch-day deepseekv41-flash-0909 tag is no longer needed. On AMD it is vllm/vllm-openai-rocm:nightly, which also carries vllm-project/vllm#56503 — the change that moves the mHC delayed pre block off the eager Torch reference and onto AITER and postdates the 0909 tag.
• The Rust OpenAI frontend is selected by default in the command builder. Switch to Python if you encounter unsupported features or compatibility issues.
• Expect a long first load: VLLM_ENGINE_READY_TIMEOUT_S=3600 is set for that reason.
• On AMD, the generated command sets VLLM_USE_BREAKABLE_CUDAGRAPH=1. DeepSeek-V4.1-Flash does not support torch.compile, and the ROCm sparse SWA backend only supports uniform-batch CUDA graphs. Without breakable CUDA graphs, default FULL_AND_PIECEWISE dies at capture.
• On AMD, the generated command passes --moe-backend aiter rather than naming a kernel. Naming aiter_triton_mxfp4_bf16 pins the Triton W4A16 _moe_gemm_a16w4 kernel; the plain name lets vLLM select the Composable Kernel a8w4 experts, used in the MI355X benchmark below. It also sets AITER_TRITON_LOG_LEVEL=ERROR to drop AITER's per-call "Gluon unavailable" warning, which is constant on gfx950 and dominated the server log.
• On MI355X, Tensor Parallel inherits vLLM's scheduler limits. Tune --max-num-seqs for your traffic and memory budget if needed. The benchmark configuration below records an explicit limit for reproducibility.
• On AMD, ticking Speculative decoding also sets enable_adaptive_verification:false. vLLM currently refuses the true flag: DeepseekV41IndexerBackend.supports_device_cpu_query_lens_mismatch() is False, and DeepseekV41ROCMAiterSparseSWABackend reports UNIFORM_BATCH rather than ALWAYS. DSpark still drafts 5 tokens per round.

Verifying

Serve, then send one text request and one image request — the vision tower is a separate path and a text-only smoke test will not exercise it:

curl http://localhost:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"deepseek-ai/DeepSeek-V4.1-Flash",
       "messages":[{"role":"user","content":"What is 17*19? Return only the integer."}]}'

A correct answer is 323.

Performance tuning

Start with vLLM's defaults and tune one parameter at a time for your workload:

• --max-num-seqs: raise the running-sequence limit if it causes queueing and memory permits; lower it if larger batches hurt latency or memory use.
• --max-num-batched-tokens: balance prefill throughput against decode latency and memory use by adjusting the token budget per scheduling iteration.
• --max-cudagraph-capture-size: cover expected decode batches, including DSpark draft tokens, while accounting for graph memory and startup cost. The benchmark below rounds 128 * (1 + 5) = 768 up to 1024.
• --max-model-len: size the context limit for your prompt plus output lengths.

MI355X benchmark server

This runnable command matches the server arguments and execution environment in InferenceX #3058 at commit 559ef756. It uses four MI355X GPUs and the pinned image. Set ROCR_VISIBLE_DEVICES and PORT for your host, and HF_HOME to an existing writable Hugging Face cache if needed. Throughput mode uses synthetic acceptance for benchmark measurement only. Run export EVAL_ONLY=true before this command for real block rejection, as used by the InferenceX accuracy eval; use that mode for ordinary serving too.

docker run --rm -i --network host --ipc host \
  --device /dev/kfd --device /dev/dri --group-add video \
  --security-opt seccomp=unconfined --workdir /tmp \
  -v "${HF_HOME:-$HOME/.cache/huggingface}:/root/.cache/huggingface" \
  -e ROCR_VISIBLE_DEVICES="${ROCR_VISIBLE_DEVICES:-0,1,2,3}" \
  -e PORT="${PORT:-8000}" -e EVAL_ONLY="${EVAL_ONLY:-false}" \
  --entrypoint /bin/bash \
  vllm/vllm-openai-rocm:nightly-eed1f3d0c6043bd494424a22443ee198dd56f657 -s <<'BASH'
set -eo pipefail
export HIP_VISIBLE_DEVICES="$ROCR_VISIBLE_DEVICES"
export VLLM_ROCM_USE_AITER=1 VLLM_ROCM_USE_AITER_MOE=1
export AITER_TRITON_LOG_LEVEL=ERROR VLLM_USE_BREAKABLE_CUDAGRAPH=1
export OMP_NUM_THREADS=1 VLLM_ENGINE_READY_TIMEOUT_S=3600
export VLLM_USE_RUST_FRONTEND=1 PYTHONUNBUFFERED=1 GPU_COUNT=4
MODEL=deepseek-ai/DeepSeek-V4.1-Flash
hf download "$MODEL"
if [[ "$EVAL_ONLY" == true ]]; then
  SPEC_CONFIG='{"method":"dspark","num_speculative_tokens":5,"draft_sample_method":"probabilistic","rejection_sample_method":"block","enable_adaptive_verification":false}'
else
  SPEC_CONFIG='{"method":"dspark","num_speculative_tokens":5,"draft_sample_method":"probabilistic","rejection_sample_method":"synthetic","synthetic_acceptance_length":3.51,"enable_adaptive_verification":false}'
fi
exec vllm serve "$MODEL" --served-model-name "$MODEL" \
  --host 0.0.0.0 --port "$PORT" --tensor-parallel-size 4 \
  --language-model-only \
  --tokenizer-mode deepseek_v41 \
  --tool-call-parser deepseek_v41 --enable-auto-tool-choice \
  --reasoning-parser deepseek_v41 \
  --moe-backend aiter \
  --gpu-memory-utilization 0.9 \
  --speculative-config "$SPEC_CONFIG" \
  --max-model-len 1048576 \
  --max-num-seqs 128 \
  --max-cudagraph-capture-size 1024 \
  --max-num-batched-tokens 16384 \
  --disable-uvicorn-access-log
BASH

The AgentX sweep uses task concurrencies 1, 2, 4, 8, 16 and 32, GPU-resident KV, and WEKA_LOADER_OVERRIDE=semianalysis_cc_traces_weka_062126. Five-token DSpark uses golden acceptance length 3.51. For replay setup and results, see InferenceX #3058 and its pinned master config. The command above starts the server; the InferenceX harness supplies the replay client. Validation evidence applies to that pinned benchmark configuration, not every future :nightly image or the general command's inherited scheduler limits.

MI355X Tensor Parallel

On MI355X, Tensor Parallel defaults to TP2 with Engram CPU offload (--engram-config '{"cpu_offload":true}'). The Engram tables cost 47.2 GiB of device memory per rank at TP4 and 94.4 GiB at TP2; on a 288 GiB card the TP2 figure does not fit beside half the checkpoint, so the tables move to pinned host memory. TP2 halves the GPUs per server and so doubles the servers per node. For TP4, pass --tensor-parallel-size 4 and --engram-config '{"cpu_offload":false}' to keep the tables resident.

The offload resolves on gfx950 only on builds carrying vllm-project/vllm#57491, which widened two is_cuda() gates to is_cuda_alike(). Those builds also default cpu_offload on through VLLM_PLE_CPU_OFFLOAD, so set it explicitly rather than inheriting the default.

ROCm additionally needs --no-swa-bounded-replay, which the shared AMD override supplies; see the comment there for why.

MI325X Tensor Parallel

On MI325X, Tensor Parallel takes the recipe default of TP4, with the Engram tables offloaded to pinned host memory (--engram-config '{"cpu_offload":true}'). A 256 GB card leaves roughly 81 GiB per rank of resident weights at TP4 and roughly 145 GiB at TP2; TP2 fits but leaves the smallest KV pool of any arm in this recipe, so TP4 is the default here rather than the TP2 that MI355X uses on its larger 288 GiB card. For TP2, pass --tensor-parallel-size 2.

MI325X is gfx942, which differs from gfx950 in two ways that matter. It has no FP4 MFMA, so the MXFP4 experts run through a Triton kernel and device memory is better spent on KV than on resident Engram tables. It also segfaults in every worker during piecewise graph capture, so this recipe sets VLLM_USE_BREAKABLE_CUDAGRAPH=0 and captures only full decode graphs (--compilation-config '{"cudagraph_mode":"FULL_DECODE_ONLY"}'); prefill runs eagerly.

The offload resolves on gfx942 only on builds carrying vllm-project/vllm#57491, the same gate MI355X needs. Those builds also default cpu_offload on through VLLM_PLE_CPU_OFFLOAD, so set it explicitly rather than inheriting the default.

ROCm additionally needs --no-swa-bounded-replay, which the shared AMD override supplies; see the comment there for why.

H200 Tensor Parallel

On H200, Tensor Parallel uses the TP4 default with Engram CPU offload (--engram-config '{"cpu_offload":true}'). The Engram tables move to pinned host DRAM and are read through UVA, so the output is unchanged. On 141 GB cards that offload is what makes TP4 practical: it moves 23.6 GiB per rank off the GPU, leaving 81.2 GiB of resident weights and 38.5 GiB of KV per GPU (about 20M tokens). No scheduler caps are needed at this replica size.

Blackwell TP and DEP

On B200, B300, GB200 and GB300, Tensor Parallel uses TP2 with FLASHINFER_MLA_SPARSE_DSV41 and Engram CPU offload (--engram-config '{"cpu_offload":true}'), which moves the Engram tables to pinned host DRAM and reads them through UVA; the output is unchanged. Data + Expert Parallel uses the hardware's GPU count: DEP4 on GB200/GB300 trays, DEP8 on B200/B300 nodes. It enables --enable-expert-parallel, FLASHMLA_MEGA_ATTN_DSV41, --kernel-config '{"moe_backend":"deep_gemm_mega_moe"}', and --engram-config '{"embedding_across_dp":true}'.

Both configurations set indexer_kv_dtype to mxfp4, indexer_sparse_logits to true, --kv-cache-dtype fp8, and --max-num-seqs 128. Reasoning uses --reasoning-parser deepseek_v41.

Each Blackwell SKU carries an exact-GPU override that takes precedence over the shared Blackwell defaults. B300 keeps its existing CUDA Graph capture configuration (--max-cudagraph-capture-size 8190), --max-num-batched-tokens 8192 and --max-num-seqs 256; B200, GB200 and GB300 add Engram CPU offload and otherwise inherit the shared Blackwell settings. Scheduler limits are workload-dependent at TP2, so size --max-num-batched-tokens and --max-cudagraph-capture-size for your own serving profile. For high interactivity on any of them, use --tensor-parallel-size 4.

KV cache offloading

Offloading requires a build containing vllm-project/vllm#57145. Validation used deepseek-ai/DeepSeek-V4.1-Flash on 4×H200 (TP4), vLLM commit d1b4028d7e, the Python frontend, --enforce-eager, --max-model-len 16384, --max-num-seqs 16 and --max-num-batched-tokens 8192. DeepSeek validation used --gpu-memory-utilization 0.95. Validation used TieringOffloadingSpec: 8 GiB total CPU tier, blocks_per_chunk: 1, a persistent filesystem root_dir, eight read/write threads and PYTHONHASHSEED=0. Three CPU-tier reloads and three filesystem reloads after restarting the server passed. Reloaded responses matched the cold output, including the reasoning field. The five-shot GSM8K diagnostic scored 31/32; this is a small sample, not the full benchmark. These results cover this text-only configuration, not the builder's larger tier/chunk defaults or other variants.

References

• Model card
• Technical report
• DeepSeek-V4-Flash recipe — the V4 sibling this diverges from
• DeepSeek-V3.2-Exp recipe — where the indexer first appeared