deepseek-ai/DeepSeek-V4.1-Flash

Overview

DeepSeek-V4.1 is a vision-language Mixture-of-Experts model: 552B backbone parameters plus 196B of Engram memory, 8B active per prompt token and 16B per output token, 40 transformer layers at hidden size 5120, with a 32-layer ViT and aligner in front of the text stack. Six things distinguish it from V4:

• Causal encoder-decoder. The 40 layers split into a 20-layer encoder and a 20-layer decoder, and the decoder's global KV is projected from the encoder's final hidden state instead of being computed layer by layer. Prefill therefore only runs the encoder half, which is where the 8B-prefill / 16B-decode split comes from.
• Two-tier sparse attention. Every layer attends over a 128-token sliding window. Layers that carry a compression ratio add compressed KV latents reaching further back, pooled by a learned softmax gate. A small side attention — the indexer, inherited from V3.2-Exp — scores those latents and keeps the best 512 per query, pre-filtered by a candidate stage that selects 2048 blocks of 8. Only four layers (2, 8, 14, 20) actually compress their own KV; the rest read that cache. V4.1 uses only compression ratios 1 and 2, where V4 used 4 and 128.
• Engram n-gram memory. Layers 1 and 14 each own a hash table of ~384M rows x 256 dims, looked up by 2-, 3- and 4-gram hashes of the input and written into the residual stream through a learned gate. These two tables alone are 196.6B parameters (~183 GiB) — plan capacity for them, they dominate everything except the experts.
• Hyper-Connections. The residual stream is carried as 4 parallel copies; each sublayer derives its own pre/post/combine coefficients from the stream, with the combine matrix made doubly stochastic by 20 Sinkhorn iterations.
• DSpark draft head. Three stages (128 routed experts each, 3 activated) draft a block of 5 tokens, reading the attention input of layers 37-39, with a Markov bias head and a confidence head on the last stage.
• Mixed MXFP4/MXFP8 checkpoint. Routed expert weights are MXFP4; everything else is MXFP8 block-quantized, UE8M0 scales throughout. Embedding and LM head are BF16.

Routing is 6 of 384 experts per token plus one shared expert, scored with sqrtsoftplus and a noaux_tc bias — with a separate routing bias for tokens inside an image span, so vision and text tokens do not compete for the same experts.

Checkpoints

1017 is the default here. It is a minor update with the same architecture, config.json, tokenizer and prompt encoding as the launch checkpoint — only the weights change. DeepSeek reports gains across reasoning, coding, agentic and visual benchmarks (e.g. Terminal-Bench 4.0 31.2 → 42.3, HLE w/o tools 36.8 → 42.6) along with shorter outputs. Both serve with the same command; the Variant row only swaps the model id. Like the launch repo it ships no Jinja chat template, which the --tokenizer-mode deepseek_v41 flag already covers. Pick FP8 (0909) to reproduce earlier results.

The verified hardware badges and the measured numbers in the sections below (benchmarks, KV offloading validation) were taken on the 0909 launch checkpoint; the 1017 weights share its config and footprint but have not been separately re-run.

Context length

1,048,576 tokens, reached by YaRN with factor 16 over a 65,536-token training window. Compressed KV rotates at its own RoPE theta (160,000) because one latent stands for several tokens, so its positions are further apart than the raw stream's.

Images

Images enter the prompt as <|deepseek_image|> spans; every position in the span carries the image token id, and the per-position role (start / newline / end / image) comes from the processor. The vision tower is a 32-layer ViT at hidden size 1024, patch 14, with a 3x downsampling aligner and a cap of 1024 tokens per image (minimum 295,936 pixels). There is no limit on images per prompt. Merged embeddings enter the text model as inputs_embeds, before the hyper-connection stream expansion, while raw token ids still flow through so the router can apply the image routing bias.

Tick Encoder parallel to run the ViT data-parallel (--mm-encoder-tp-mode data) instead of tensor-parallel: at 32 layers / hidden 1024 the encoder is small enough that TP communication costs more than it saves, which can significantly reduce TTFT for multi-image requests. It is mutually exclusive with Text only.

Reasoning and tool calling

Two thinking modes and a numeric reasoning budget rather than discrete tiers. Send them as chat_template_kwargs:

The top-level OpenAI reasoning_effort field also works, where "none" turns thinking off. "minimal" and "medium" are rejected — they are not part of this model's set.

Note that the DeepSeek API maps its low / high / max tiers to 50 / 75 / 100, while the open-source prompt encoder (and therefore vLLM) maps low / high / xhigh / max to 25 / 50 / 75 / 100. A request tuned at high on the API runs at effort 50 here; pass an integer if you need parity.

With both keys unset, thinking is ON at effort 50. The do-nothing config is the most verbose one, so a request with a small max_tokens spends its budget on the trace and returns empty content with finish_reason=length. That reads as a broken model and is not. Either pin thinking: false or give the budget room.

In thinking mode the budget is rendered into the prompt as a Reasoning Effort: N prefix on the first turn only.

resp = client.chat.completions.create(
    model="deepseek-ai/DeepSeek-V4.1-Flash-1017",
    messages=[{"role": "user", "content": "What is 17*19?"}],
    extra_body={"chat_template_kwargs": {"thinking": True, "reasoning_effort": 25}},
)

DeepSeek's published numbers use temperature=1.0, top_p=0.95 and effort 100 with max_tokens of at least 256K. The tech report notes that efforts of 60-80 recover most of that accuracy at well under half the tokens, so reserve max for hard tasks.

Tool calls are wrapped in DSML tag blocks rather than JSON fences, and tool output comes back in <tool_result> tags.

Speculative decoding

DSpark is the only speculative method for this checkpoint. V4.1 dropped the MTP module that V3 and V4 trained alongside the backbone. Speculative decoding enables a 5-token block. On NVIDIA it also turns on adaptive verification (see the feature description). On AMD, vLLM currently refuses adaptive verification. The generated command sets enable_adaptive_verification:false and verifies the full block. Two checks in maybe_create_adaptive_verification_manager fail on ROCm: the indexer helper supports_device_cpu_query_lens_mismatch() is False, and DeepseekV41ROCMAiterSparseSWABackend reports AttentionCGSupport.UNIFORM_BATCH rather than ALWAYS. Remove the AMD override only after a serve with the true flag starts. Acceptance depends on the workload, so measure it on your own traffic before sizing a deployment around it. The drafter's experts add roughly 14B parameters to the load.

Serving text-only

Tick Text only to add --language-model-only, which skips the vision encoder entirely. Worth it whenever the workload is text: it drops the ViT and aligner from the load and frees that VRAM for KV cache. It is mutually exclusive with encoder_parallel. The verified GB200 runs (TP4 and 1P1D) were text-only.

Prefill/Decode disaggregation

The Prefill/Decode Disaggregation strategy is the verified 1P1D layout on GB200 NVL4: one tray (4 GPUs) per role, TP4 in each pool, KV handed over through NIXL, fronted by vllm-router --vllm-pd-disaggregation. Both pools disable FlashInfer autotune plus JIT and CuTeDSL warmup via --kernel-config, skip the DeepGEMM warmup (VLLM_DEEP_GEMM_WARMUP=skip), and cap --max-num-seqs at 32. On 8-GPU nodes the same layout becomes TP8 per role.

With Speculative decoding on, DSpark runs in both pools so the transferred KV stays compatible.

GB300

GB300 runs both pools as DEP, per SemiAnalysisAI/InferenceX#3941: one tray of DEP4 prefill, four trays of DEP16 decode. Keep them on DEP; MegaMoE needs expert parallelism.

To scale prefill, InferenceX ran two DEP2 engines per tray. Drop the --data-parallel-{start-rank,address,rpc-port} flags from the prefill command and run it twice:

CUDA_VISIBLE_DEVICES=0,1 VLLM_NIXL_SIDE_CHANNEL_PORT=5557 vllm serve ... --port 8001 \
  --data-parallel-size 2 --data-parallel-size-local 2
CUDA_VISIBLE_DEVICES=2,3 VLLM_NIXL_SIDE_CHANNEL_PORT=5559 vllm serve ... --port 8011 \
  --data-parallel-size 2 --data-parallel-size-local 2
# vllm-router: list both ports, with --intra-node-data-parallel-size 1

Memory

The checkpoint is roughly 511 GB on disk (476 GiB), which breaks down as:

DeepSeek's 552B backbone figure covers the routed experts, attention and embeddings; the Engram tables, the ~14B DSpark drafter and the block scales sit outside it.

vram_minimum_gb: 614 is that total times the schema's 1.2 headroom factor. It fits one GB200 NVL4 tray (768 GB) at TP4, or one 8-GPU H200 node (1128 GB) with room for KV cache.

KV cache is a small share of that budget. The compressed latents are shared across layers and trained to be stored in FP4, and DeepSeek puts the resulting global KV at 890 bytes per token, about a quarter of V4-Flash; by that accounting a full 1M-token prompt holds under 1 GB of global KV, plus a fixed 128-token sliding window per layer. Weights and batch size, not cache, set the capacity limit — still, measure before assuming a given context/batch fits.

Prerequisites

• vLLM 0.31.0+, from the pip wheel or the release image. The architecture shipped in 0.30.0 (vllm-project/vllm#56228); 0.31.0 adds the KV-offload fix (vllm-project/vllm#57145) and the ROCm Engram host-memory gate (vllm-project/vllm#57491). The launch-day deepseekv41-flash-0909 image is no longer needed.
• On AMD the Docker tab uses vllm/vllm-openai-rocm:nightly-rocm100, the ROCm 10.0 nightly the AMD rows were verified on. MI355X and MI325X need it: their commands pass --monokernel, and MI355X also passes the ROCm MXFP4 indexer. Both are on main but not in 0.31.0, so the Install block offers only Docker on those two SKUs. MI300X and MI350X also run from the ROCm pip wheel.
• The Rust OpenAI frontend is selected by default in the command builder. Switch to Python if you encounter unsupported features or compatibility issues.
• Expect a long first load: VLLM_ENGINE_READY_TIMEOUT_S=3600 is set for that reason.
• On AMD, the generated command sets VLLM_USE_BREAKABLE_CUDAGRAPH=1. DeepSeek-V4.1-Flash does not support torch.compile, and the ROCm sparse SWA backend only supports uniform-batch CUDA graphs. Without breakable CUDA graphs, default FULL_AND_PIECEWISE dies at capture.
• On AMD, the generated command passes --moe-backend aiter rather than naming a kernel. Naming aiter_triton_mxfp4_bf16 pins the Triton W4A16 _moe_gemm_a16w4 kernel; the plain name lets vLLM select the Composable Kernel a8w4 experts, used in the MI355X benchmark below. It also sets AITER_TRITON_LOG_LEVEL=ERROR to drop AITER's per-call "Gluon unavailable" warning, which is constant on gfx950 and dominated the server log.
• On MI355X, Tensor Parallel inherits vLLM's scheduler limits. Tune --max-num-seqs for your traffic and memory budget if needed. The benchmark configuration below records an explicit limit for reproducibility.
• On AMD, ticking Speculative decoding also sets enable_adaptive_verification:false. vLLM currently refuses the true flag: DeepseekV41IndexerBackend.supports_device_cpu_query_lens_mismatch() is False, and DeepseekV41ROCMAiterSparseSWABackend reports UNIFORM_BATCH rather than ALWAYS. DSpark still drafts 5 tokens per round.

Verifying

Serve, then send one text request and one image request — the vision tower is a separate path and a text-only smoke test will not exercise it:

curl http://localhost:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"deepseek-ai/DeepSeek-V4.1-Flash-1017",
       "messages":[{"role":"user","content":"What is 17*19? Return only the integer."}]}'

A correct answer is 323.

Performance tuning

Start with vLLM's defaults and tune one parameter at a time for your workload:

• --max-num-seqs: raise the running-sequence limit if it causes queueing and memory permits; lower it if larger batches hurt latency or memory use.
• --max-num-batched-tokens: balance prefill throughput against decode latency and memory use by adjusting the token budget per scheduling iteration.
• --max-cudagraph-capture-size: cover expected decode batches, including DSpark draft tokens, while accounting for graph memory and startup cost. The benchmark below rounds 128 * (1 + 5) = 768 up to 1024.
• --max-model-len: size the context limit for your prompt plus output lengths.

MI355X benchmark server

This runnable command matches the server arguments and execution environment in InferenceX #3058 at commit 559ef756. It uses four MI355X GPUs and the pinned image. Set ROCR_VISIBLE_DEVICES and PORT for your host, and HF_HOME to an existing writable Hugging Face cache if needed. Throughput mode uses synthetic acceptance for benchmark measurement only. Run export EVAL_ONLY=true before this command for real block rejection, as used by the InferenceX accuracy eval; use that mode for ordinary serving too.

docker run --rm -i --network host --ipc host \
  --device /dev/kfd --device /dev/dri --group-add video \
  --security-opt seccomp=unconfined --workdir /tmp \
  -v "${HF_HOME:-$HOME/.cache/huggingface}:/root/.cache/huggingface" \
  -e ROCR_VISIBLE_DEVICES="${ROCR_VISIBLE_DEVICES:-0,1,2,3}" \
  -e PORT="${PORT:-8000}" -e EVAL_ONLY="${EVAL_ONLY:-false}" \
  --entrypoint /bin/bash \
  vllm/vllm-openai-rocm:nightly-eed1f3d0c6043bd494424a22443ee198dd56f657 -s <<'BASH'
set -eo pipefail
export HIP_VISIBLE_DEVICES="$ROCR_VISIBLE_DEVICES"
export VLLM_ROCM_USE_AITER=1 VLLM_ROCM_USE_AITER_MOE=1
export AITER_TRITON_LOG_LEVEL=ERROR VLLM_USE_BREAKABLE_CUDAGRAPH=1
export OMP_NUM_THREADS=1 VLLM_ENGINE_READY_TIMEOUT_S=3600
export VLLM_USE_RUST_FRONTEND=1 PYTHONUNBUFFERED=1 GPU_COUNT=4
MODEL=deepseek-ai/DeepSeek-V4.1-Flash
hf download "$MODEL"
if [[ "$EVAL_ONLY" == true ]]; then
  SPEC_CONFIG='{"method":"dspark","num_speculative_tokens":5,"draft_sample_method":"probabilistic","rejection_sample_method":"block","enable_adaptive_verification":false}'
else
  SPEC_CONFIG='{"method":"dspark","num_speculative_tokens":5,"draft_sample_method":"probabilistic","rejection_sample_method":"synthetic","synthetic_acceptance_length":3.51,"enable_adaptive_verification":false}'
fi
exec vllm serve "$MODEL" --served-model-name "$MODEL" \
  --host 0.0.0.0 --port "$PORT" --tensor-parallel-size 4 \
  --language-model-only \
  --tokenizer-mode deepseek_v41 \
  --tool-call-parser deepseek_v41 --enable-auto-tool-choice \
  --reasoning-parser deepseek_v41 \
  --moe-backend aiter \
  --gpu-memory-utilization 0.9 \
  --speculative-config "$SPEC_CONFIG" \
  --max-model-len 1048576 \
  --max-num-seqs 128 \
  --max-cudagraph-capture-size 1024 \
  --max-num-batched-tokens 16384 \
  --disable-uvicorn-access-log
BASH

The validation sweep uses task concurrencies 1, 2, 4, 8, 16 and 32, GPU-resident KV, and WEKA_LOADER_OVERRIDE=semianalysis_cc_traces_weka_062126. Five-token DSpark uses golden acceptance length 3.51. For replay setup and results, see InferenceX #3058 and its pinned master config. The command above starts the server; the InferenceX harness supplies the replay client. Validation evidence applies to that pinned benchmark configuration, not every future :nightly image or the general command's inherited scheduler limits.

MI355X Tensor Parallel

On MI355X, Tensor Parallel defaults to TP2 with Engram CPU offload (--engram-config '{"cpu_offload":true}'). The Engram tables cost 47.2 GiB of device memory per rank at TP4 and 94.4 GiB at TP2; on a 288 GiB card the TP2 figure does not fit beside half the checkpoint, so the tables move to pinned host memory. TP2 halves the GPUs per server and so doubles the servers per node. For TP4, pass --tensor-parallel-size 4 and --engram-config '{"cpu_offload":false}' to keep the tables resident.

The offload resolves on gfx950 only on builds carrying vllm-project/vllm#57491, which widened two is_cuda() gates to is_cuda_alike(). Those builds also default cpu_offload on through VLLM_PLE_CPU_OFFLOAD, so set it explicitly rather than inheriting the default.

--monokernel selects the mono decode layer from vllm-project/vllm#60397, which vllm-project/vllm#61082 extended to all 40 backbone layers and moved from the VLLM_ROCM_MONO_DECODE environment variable to this flag. Two persistent launches cover a layer for decode steps of up to 48 rows, which with five DSpark drafts is concurrency 8; larger steps fall back to the regular path, so the gain is largest at low concurrency and fades as concurrency rises.

Decoder SWA bounded replay (vllm-project/vllm#59039) is on by default on ROCm, so the recipe no longer passes --no-swa-bounded-replay. Use a nightly from 2026-10-13 or later; earlier ROCm builds lack the replay window clamp and fault at the first prefix hit.

AgentX TP2 scheduling

For MI355X TP2 with AgentX-like traffic, we recommend the scheduler settings from InferenceX #4409, using FP8 (0909) (deepseek-ai/DeepSeek-V4.1-Flash):

Select MI355X, FP8 (0909), Tensor Parallel, Rust, Speculative decoding and Text only above. Keep TP2, Engram CPU offload and five-token DSpark with real block rejection (enable_adaptive_verification:false). Append the table's flags plus --max-model-len 1048576 --max-cudagraph-capture-size 1024 --disable-uvicorn-access-log to the generated serve arguments. Replace the Docker image with the following, adding -e OMP_NUM_THREADS=1 -e PYTHONUNBUFFERED=1 before it:

vllm/vllm-openai-rocm:nightly-e68efc685ab1e6f1eda004ced7157e3ca411da2f@sha256:520e098f54ec96837228aaddea0a833cbf6374dbdffd480c1d65f26ffab7701b

Audited config · Sweep. The sweep used synthetic acceptance length 3.51 for throughput; production uses real block rejection. Excess clients queue behind the sequence cap. These workload-specific limits are not generic defaults; this sweep did not qualify TP4 c16/c32.

MI300X Tensor Parallel

On MI300X, Tensor Parallel takes the recipe default of TP4 with the Engram tables offloaded to pinned host memory (--engram-config '{"cpu_offload":true}'). At TP4 the resident weights are roughly 81 GiB per rank, which leaves a healthy KV pool on a 192 GB card.

TP2 does not fit on this SKU. At TP2 the offloaded weights rise to roughly 145 GiB per rank, and a measured run reported Available KV cache memory: -13.51 GiB, so the engine refused to start with No available memory for the cache blocks. That was already with the sparse-attention indexer buffer halved to 4096 batched tokens. Raising --gpu-memory-utilization and halving the buffer again recover roughly 13 GiB between them, which still leaves no usable KV for a 1M-context model, so TP2 is not offered here. MI325X runs the same TP2 configuration successfully on its larger 256 GB card.

MI300X is gfx942 and needs the same two workarounds as MI325X: no FP4 MFMA, so the MXFP4 experts run through a Triton kernel and device memory is better spent on KV than on resident Engram tables; and piecewise graph capture segfaults every worker, so this recipe sets VLLM_USE_BREAKABLE_CUDAGRAPH=0 and captures only full decode graphs (--compilation-config '{"cudagraph_mode":"FULL_DECODE_ONLY"}'), with prefill eager.

This recipe does not pass --monokernel. With the kernels' weight copies included, #60731 measured 170.11 GiB of weights a rank. At --gpu-memory-utilization 0.9 a 192 GiB MI300X leaves about 2.7 GiB beside those weights.

The offload resolves on gfx942 only on builds carrying vllm-project/vllm#57491, the same gate MI325X and MI355X need. Those builds also default cpu_offload on through VLLM_PLE_CPU_OFFLOAD, so set it explicitly rather than inheriting the default.

MI325X Tensor Parallel

On MI325X, Tensor Parallel takes the recipe default of TP4, with the Engram tables offloaded to pinned host memory (--engram-config '{"cpu_offload":true}'). A 256 GB card leaves roughly 81 GiB per rank of resident weights at TP4 and roughly 145 GiB at TP2; TP2 fits but leaves the smallest KV pool of any arm in this recipe, so TP4 is the default here rather than the TP2 that MI355X uses on its larger 288 GiB card. For TP2, pass --tensor-parallel-size 2.

MI325X is gfx942, which differs from gfx950 in two ways that matter. It has no FP4 MFMA, so the MXFP4 experts run through a Triton kernel and device memory is better spent on KV than on resident Engram tables. It also segfaults in every worker during piecewise graph capture, so this recipe sets VLLM_USE_BREAKABLE_CUDAGRAPH=0 and captures only full decode graphs (--compilation-config '{"cudagraph_mode":"FULL_DECODE_ONLY"}'); prefill runs eagerly.

--monokernel selects the gfx942 mono decode kernels from vllm-project/vllm#60731. They run at TP4 only, on decode steps of up to 42 rows (7 requests with five DSpark drafts). Larger steps, prefill, and mixed batches take the regular path, and TP2 stays on vLLM's path. With their weight copies included, #60731 measured 170.11 GiB of weights a rank, up from 98.88 GiB. At --gpu-memory-utilization 0.9 the KV pool still held 11.18 million tokens. Use a ROCm nightly from 2026-10-20 or later; nightly-rocm100-df3bd094aeea829f27baa0a5282830464f601d9f is the first. Earlier builds have the flag for gfx950 only and refuse to start on MI325X.

The offload resolves on gfx942 only on builds carrying vllm-project/vllm#57491, the same gate MI355X needs. Those builds also default cpu_offload on through VLLM_PLE_CPU_OFFLOAD, so set it explicitly rather than inheriting the default.

H200 Tensor Parallel

On H200, Tensor Parallel uses the TP4 default with Engram CPU offload (--engram-config '{"cpu_offload":true}'). The Engram tables move to pinned host DRAM and are read through UVA, so the output is unchanged. On 141 GB cards that offload is what makes TP4 practical: it moves 23.6 GiB per rank off the GPU, leaving 81.2 GiB of resident weights and 38.5 GiB of KV per GPU (about 20M tokens). No scheduler caps are needed at this replica size.

Blackwell TP and DEP

On B200, B300, GB200 and GB300, Tensor Parallel uses TP2 with FLASHINFER_MLA_SPARSE_DSV41 and Engram CPU offload (--engram-config '{"cpu_offload":true}'), which moves the Engram tables to pinned host DRAM and reads them through UVA; the output is unchanged. Data + Expert Parallel uses the hardware's GPU count: DEP4 on GB200/GB300 trays, DEP8 on B200/B300 nodes. It enables --enable-expert-parallel, FLASHMLA_MEGA_ATTN_DSV41, --kernel-config '{"moe_backend":"deep_gemm_mega_moe"}', and --engram-config '{"embedding_across_dp":true}'.

Both configurations set indexer_kv_dtype to mxfp4, indexer_sparse_logits to true, --kv-cache-dtype fp8, and --max-num-seqs 128. Reasoning uses --reasoning-parser deepseek_v41.

Each Blackwell SKU carries an exact-GPU override that takes precedence over the shared Blackwell defaults. B300 keeps its existing CUDA Graph capture configuration (--max-cudagraph-capture-size 8190), --max-num-batched-tokens 8192 and --max-num-seqs 256; B200, GB200 and GB300 add Engram CPU offload and otherwise inherit the shared Blackwell settings. B200 is the exception for both strategies: it renders the qualified B200 AgentX settings described in AgentX tuning on NVIDIA GPUs, so B200 DEP uses FLASHINFER_MLA_SPARSE_DSV41, {"enable_flashinfer_autotune":true,"moe_backend":"deep_gemm_mega_moe"}, {"cpu_offload":true,"embedding_across_dp":true} and --max-num-seqs 32. Scheduler limits are workload-dependent at TP2, so size --max-num-batched-tokens and --max-cudagraph-capture-size for your own serving profile. For high interactivity on any of them, use --tensor-parallel-size 4.

AgentX tuning on NVIDIA GPUs

The B200 AgentX lane (SemiAnalysisAI/InferenceX#3837) serves the 0909 launch checkpoint (deepseek-ai/DeepSeek-V4.1-Flash; pick FP8 (0909) to match it) with FlashInfer autotuning (--kernel-config '{"enable_flashinfer_autotune":true}'), --gpu-memory-utilization 0.97 and VLLM_USE_V2_MODEL_RUNNER=1 on every point.

TP4 serves concurrency 1, 4, 8, 16, 32 and 128 with --tensor-parallel-size 4, THP-backed Engram tables (--engram-config '{"cpu_offload":true,"use_thp":true}') and these per-point limits:

Every TP4 point captures piecewise graphs at the DSpark verification multiples [6,12,18,24,30,36,48,60,72,96,120,144,192,240,288,384,480,576,768,1020,1536,2046,3072,4092], extended with 6144,8190 from c8. From c16 it also captures decode-replay graphs separately, with the same decoder_replay_cudagraph_capture_sizes as DEP4 below.

TP2 at concurrency 1 runs the TP4 c1 settings on two GPUs (--max-num-seqs 8, --max-num-batched-tokens 4096, capture to 4092).

DEP2 (TP1 x DP2 with expert parallelism) serves concurrency 8 to 32 with --max-num-batched-tokens 4096. --max-num-seqs is per DP rank: 16 at c8 and c16 and 32 at c24 and c32. At c8 and c16 graph capture stops at 576 tokens (--max-cudagraph-capture-size 576) and --prefill-schedule-interval stays at the vLLM default of 1, while c24 and c32 extend piecewise graphs to 4096 tokens and set --prefill-schedule-interval 64.

DEP4 (TP1 x DP4) serves concurrency 96 and 128 with --max-num-batched-tokens 8192 and --max-cudagraph-capture-size 8190. --max-num-seqs is 96 at c96 and 128 at c128, and --prefill-schedule-interval is 48 at both. Decode-replay graphs are captured separately:

--compilation-config '{"mode":"VLLM_COMPILE","cudagraph_mode":"FULL_AND_PIECEWISE","cudagraph_capture_sizes":[6,12,18,24,30,36,48,60,72,96,120,144,192,240,288,384,480,576,768,1020,1536,2046,3072,4092,6144,8190],"decoder_replay_cudagraph_capture_sizes":[6,12,18,24,30,36,48,60,72,96,120,144,192,240,288,384,480,576,768,1020,1536,2048,2304,2560,2816,3072,3328,3584,3840,4096]}'

DEP2 with host KV at c64 and c128 adds vLLM's native CPU KV offloading with --kv-offloading-backend native --kv-offloading-size 32 (GiB per DP rank, 64 GiB per server), --gpu-memory-utilization 0.955 and --max-num-seqs 48. --max-num-seqs is per DP rank, so the server runs up to 96 sequences: nothing queues at c64, and about 32 clients wait at c128. It keeps --prefill-schedule-interval at 1 and captures graphs only up to 576 tokens (--max-cudagraph-capture-size 576), as DEP2 does at c8 and c16.

Router. DEP2 and DEP4 sit behind vLLM Router 0.1.14, which fronts the DP ranks of the single server:

vllm-router --policy consistent_hash --request-timeout-secs 14400 --disable-retries \
  --intra-node-data-parallel-size <DP> --worker-urls http://<server>:<port>

consistent_hash replaces the router's cache_aware default, --request-timeout-secs raises its 1800 s default so long agentic requests finish, and --disable-retries stops the router from replaying a failed request. Set --intra-node-data-parallel-size to the server's --data-parallel-size (2 or 4). The benchmark also binds the router's Prometheus endpoint to localhost; that is telemetry, not a serving setting.

In the command builder, pick B200 and the FP8 (0909) checkpoint, and tick Speculative decoding and Text only (both are opt-in). Tensor Parallel then renders the TP2 c1 settings, and Data + Expert Parallel renders the DEP2 c24/c32 engine settings. The builder's DEP spans all eight GPUs, so set --data-parallel-size 2 (or 4 for DEP4), select the matching GPUs, change the vLLM Router's --intra-node-data-parallel-size to the same value, and take the per-concurrency values above.

Aggregate serving on B300

The B300 aggregate lanes in InferenceX #4332 (full config) serve the original DeepSeek-V4.1-Flash checkpoint on vllm/vllm-openai:cu134-nightly-x86_64-dev-20261019-mk with --kernel-config '{"enable_flashinfer_autotune":true}' and --max-model-len 1048576.

TP4 uses --engram-config '{"cpu_offload":true,"use_thp":true}' and --additional-config '{"decode_backend":"latency_specialized"}'. The specialized decode comes from #61868 and #61869, which are not yet on main, and serves pure decode steps only, hence the long prefill interval at c8 and c16. TP4 adds graph capture size 5, so the DSpark draft gets a one-request graph.

DEP2 (TP1 x DP2 with expert parallelism) runs behind a consistent-hash vLLM Router with --kernel-config '{"enable_flashinfer_autotune":true,"moe_backend":"deep_gemm_mega_moe"}' and --engram-config '{"cpu_offload":true,"embedding_across_dp":true}'. At c128 and c192, KV blocks are offloaded lazily to DRAM, 50 GiB per DP rank, and are written only when evicted from HBM:

--kv-transfer-config '{"kv_connector":"SimpleCPUOffloadConnector","kv_role":"kv_both","kv_connector_extra_config":{"cpu_bytes_to_use_per_rank":53687091200,"lazy_offload":true}}'

Dynamo disaggregated serving on B300 and GB300

The AgentX lanes in InferenceX #4231 (B300) and InferenceX #4266 (GB300) run N DEP2 prefill engines and one DEP decode engine, each with its own sidecar, behind one KV-routing frontend. Prefill pushes KV to decode with NIXL and keeps a host KV tier. The push_as_nixl connector and VLLM_MAX_ASYNC_KV_ADMITS_PER_STEP come from the dev images vllm/vllm-openai:nightly-x86_64-dev-20261012-59e75f4 (B300) and vllm/vllm-openai:cu134-nightly-aarch64-dev-20261019 (GB300).

Both roles use TP1 with expert parallelism and these settings:

export VLLM_USE_V2_MODEL_RUNNER=1 VLLM_ALLREDUCE_USE_SYMM_MEM=0 VLLM_RANDOMIZE_DP_DUMMY_INPUTS=1 NCCL_CUMEM_ENABLE=1
export UCX_MEMTYPE_CACHE=n UCX_MEMTYPE_REG_WHOLE=n UCX_RNDV_PIPELINE_ERROR_HANDLING=y
export VLLM_ENGINE_READY_TIMEOUT_S=7200 VLLM_RPC_TIMEOUT=600000
--attention-config '{"backend":"FLASHMLA_MEGA_ATTN_DSV41","indexer_kv_dtype":"mxfp4","indexer_sparse_logits":true}'
--kernel-config '{"enable_flashinfer_autotune":true,"moe_backend":"deep_gemm_mega_moe","sparse_indexer_topk_backend":"deep_select"}'
--speculative-config '{"method":"dspark","num_speculative_tokens":5,"draft_sample_method":"probabilistic","enable_adaptive_verification":false}'
--kv-cache-dtype fp8 --gpu-memory-utilization 0.95 --prefill-schedule-interval 2 --enable-ep-weight-filter --numa-bind

On B300, set --cpu-distributed-timeout-seconds 7200 for the Engram load. NIXL's LIBFABRIC plugin also needs libfabric and rdma-core from the AWS EFA installer and the CUDA 12 runtime (nvidia-cuda-runtime-cu12), which the image lacks.

Prefill runs --data-parallel-size 2 per GPU pair with --max-num-seqs 64, --long-prefill-token-threshold 8192 and --prefix-cache-retention-interval 32768. Its KV config, shown for B300 (GB300 uses "backends":["UCX"] and 85899345920 bytes):

{"kv_connector":"MultiConnector","kv_role":"kv_both","kv_connector_extra_config":{"connectors":[
  {"kv_connector":"NixlConnector","kv_connector_module_path":"vllm.distributed.kv_transfer.kv_connector.v1.nixl.push_as_nixl","kv_role":"kv_both","kv_load_failure_policy":"fail","kv_buffer_device":"cuda","kv_connector_extra_config":{"backends":["LIBFABRIC"],"enforce_handshake_compat":false,"kv_lease_duration":120}},
  {"kv_connector":"SimpleCPUOffloadConnector","kv_role":"kv_both","kv_connector_extra_config":{"cpu_bytes_to_use":75161927680,"lazy_offload":true}}]}}

Prefill graphs: {"mode":"VLLM_COMPILE","cudagraph_mode":"FULL_AND_PIECEWISE","cudagraph_capture_sizes":[6,12,18,24,30,36,48,60,72,96,120,144,192,240,288,384,480,576,768,1020,1536,2046],"decoder_replay_cudagraph_capture_sizes":[6,12,18,24,30,36,48,60,72,96,120,144,192,240,288,384,480,576,768,1020,1536,2048],"max_cudagraph_capture_size":2046}.

Decode uses the NixlConnector entry alone with "kv_role":"kv_consumer" and --no-enable-prefix-caching. It sets --max-num-batched-tokens to ten times --max-num-seqs and captures FULL decode-only graphs in steps of 6 up to six times --max-num-seqs. It balances experts with --enable-eplb and --eplb-config '{"communicator":"symm_mem","num_redundant_experts":16,"window_size":1000,"step_interval":3000,"use_async":true}'.

Launch the sidecars as in Dynamo sidecar serving, with --grpc-startup-deadline-secs 3600. Start the frontend so that every request prefills on a prefill engine and each session stays on one prefill engine's prefix and host KV cache. On GB300, also set DYN_ROUTER_DECODE_ACTIVE_REQUEST_WEIGHT=10000000.

export DYN_ROUTER_TEMPERATURE=0 DYN_ROUTER_QUEUE_THRESHOLD=None DYN_ROUTER_ACTIVE_REQUEST_EXPIRY_SECS=3600
export DYN_TOKENIZER=fastokens DYN_TOKENIZER_CACHE=1 DYN_TOKENIZER_CACHE_BYTES=8000000000
export DYN_TCP_CHANNEL_BUFFER=128 DYN_TCP_REQUEST_TIMEOUT=60
uv run --no-project python -m dynamo.frontend --http-port 8000 --router-mode kv \
  --router-session-affinity-ttl-secs 14400 --active-decode-blocks-threshold None \
  --active-prefill-tokens-threshold None --active-prefill-tokens-threshold-frac None

Dynamo sidecar serving

Dynamo can route requests to vLLM's native gRPC server through a colocated sidecar. Choose the checkpoint, hardware and parallelism in the command builder above. For compatible runtime installation and complete launch examples, see the Dynamo vLLM sidecar guide.

The B300 deployment in InferenceX #4231 validated this integration with the original DeepSeek-V4.1-Flash checkpoint. The builder defaults to DeepSeek-V4.1-Flash-1017; that checkpoint was not covered by this deployment's validation.

Connect the engine and sidecar

Use the generated model and hardware arguments with vllm-rs serve. Add --grpc-port 19000 and set --reasoning-parser None; omit the engine's --tool-call-parser and --enable-auto-tool-choice flags because the sidecar handles reasoning and tool parsing. For KV-aware routing, enable KV events on the engine with --kv-events-config:

{"publisher":"zmq","topic":"kv-events","enable_kv_cache_events":true,"endpoint":"tcp://*:5200"}

With etcd running, set its reachable address in each frontend and sidecar shell, along with the shared request and event planes:

export ETCD_ENDPOINTS=http://ETCD_HOST:2379
export DYN_REQUEST_PLANE=tcp DYN_EVENT_PLANE=zmq

In a separate shell on the engine host, connect one sidecar to the engine:

export DYN_VLLM_KV_EVENT_PORT=5200
uv run --no-project python -m dynamo.vllm.sidecar \
  --grpc-endpoint 127.0.0.1:19000 \
  --dyn-tool-call-parser deepseek_v41 --dyn-reasoning-parser deepseek_v41

Start a shared frontend:

uv run --no-project python -m dynamo.frontend --http-port 8000 --router-mode kv

For disaggregated serving, configure the engines as in Dynamo disaggregated serving on B300 and GB300. Add --disaggregation-mode prefill --component prefill to prefill sidecars and --disaggregation-mode decode to decode sidecars. Allocate distinct gRPC and KV-event ports when running several engines on the same host.

Check the frontend after the workers register:

curl --fail http://FRONTEND_HOST:8000/health
curl --fail http://FRONTEND_HOST:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"deepseek-ai/DeepSeek-V4.1-Flash","messages":[{"role":"user","content":"Hello"}],"max_tokens":32}'

Set the request's model to the served model name selected for the engine.

DGX Station (GB300) Tensor Parallel

One GB300 exposes 252 GB of HBM next to 496 GB of Grace LPDDR5X, and the 476 GiB checkpoint does not fit the HBM. The Station row serves TP1 with both Engram tables (189 GiB) and 60 GiB of routed experts in pinned host memory, read through UVA. At --gpu-memory-utilization 0.97 the KV pool holds about 3.09M tokens at full 1M context. 60 GiB is the lowest verified offload at 1M context with speculative decoding on (55 fails the minimum-KV check). For more concurrent 1M contexts, raise --cpu-offload-gb to 70 and accept ~10% lower single-stream decode.

Measured on vllm/vllm-openai:nightly-dee37d89:

• Autotune. The first boot on a new flag set or image runs a 75–90 minute FlashInfer autotune. Mount /root/.cache/vllm on a persistent volume; later boots start in about 7 minutes.
• Host memory. With the server loaded, ~430 of the 494 GiB Grace pool is in use (pinned weights plus loader shared memory). VLLM_WEIGHT_OFFLOADING_DISABLE_PIN_MEMORY=1 frees ~66 GB at about 7% lower single-stream decode.

Launch scripts and full measurements: J-M-Recipes/recipes.

KV cache offloading

Offloading requires vLLM 0.31.0+, the first release containing vllm-project/vllm#57145. Validation used deepseek-ai/DeepSeek-V4.1-Flash on 4×H200 (TP4), vLLM commit d1b4028d7e, the Python frontend, --enforce-eager, --max-model-len 16384, --max-num-seqs 16 and --max-num-batched-tokens 8192. DeepSeek validation used --gpu-memory-utilization 0.95. Validation used TieringOffloadingSpec: 8 GiB total CPU tier, blocks_per_chunk: 1, a persistent filesystem root_dir, eight read/write threads and PYTHONHASHSEED=0. Three CPU-tier reloads and three filesystem reloads after restarting the server passed. Reloaded responses matched the cold output, including the reasoning field. The five-shot GSM8K diagnostic scored 31/32; this is a small sample, not the full benchmark. These results cover this text-only configuration, not the builder's larger tier/chunk defaults or other variants.

References

• Model card
• DeepSeek-V4.1-Flash-1017 model card
• Technical report
• DeepSeek-V4-Flash recipe — the V4 sibling this diverges from
• DeepSeek-V3.2-Exp recipe — where the indexer first appeared