# vLLM Recipes — full text Concatenated full text of every recipe on https://recipes.vllm.ai, formatted for AI-assistant retrieval. Each section is prefixed with its canonical URL, model metadata, recommended flags, and (where present) the recipe's full guide body. Recipe count: 220 Generated at build time from the canonical recipe YAMLs, assembled from their source repositories (see `sources.yaml` in vllm-project/recipes). --- # mistralai/Mistral-Large-4-1T-A52B on vLLM Source: https://recipes.vllm.ai/mistralai/Mistral-Large-4-1T-A52B Provider: Mistral AI Released: 2026-09-30T18:35:13.000Z Updated: 2026-10-29 Architecture: moe Parameters: 1T Context length: 1,048,576 Min vLLM: 0.34.0 Tasks: multimodal Verified hardware: GB300 NVL4, B300, B200, H200, MI350X, MI355X Summary: Mistral Large 4: a 1T-total / 49B-active multimodal MoE with sparse MLA attention, a built-in DSpark drafter, tool calling and reasoning, in FP8, NVFP4 and AMD MXFP4 checkpoints. ## Variants - default: precision=nvfp4, vram_minimum_gb=733, model_id=mistralai/Mistral-Large-4-1T-A52B-NVFP4 ModelOpt NVFP4 experts with FP8 attention and shared experts (~76 GiB per GPU on 8 GPUs), with the DSpark drafter. ~1.5x the KV capacity of FP8. Validated on GB300, B300 and B200. - fp8: precision=fp8, vram_minimum_gb=1273 FP8 block-scaled weights with the DSpark drafter (~128 GiB of weights per GPU on 8 GPUs). On AMD the routed experts are re-quantized to MXFP4 at load, which also fits TP4. Validated on GB300, B300, H200 and MI350X. - mxfp4: precision=mxfp4, vram_minimum_gb=694, model_id=amd/Mistral-Large-4-1T-A52B-Quark-MXFP4 AMD Quark OCP MXFP4 routed experts (static weight, dynamic activation scales); attention, shared experts, dense layers, router, head, vision encoder and the DSpark drafter keep the original precision. ~578 GB, fits four MI350 Series GPUs. ## Recommended base flags ```bash vllm serve 'mistralai/Mistral-Large-4-1T-A52B' \ --tokenizer-mode \ mistral \ --config-format \ mistral \ --load-format \ mistral \ --max-model-len \ 262144 \ --max-num-batched-tokens \ 8192 \ --max-num-seqs \ 128 \ --gpu-memory-utilization \ 0.85 ``` Environment: ```bash VLLM_ENGINE_READY_TIMEOUT_S='6000' ``` ## Guide ## Overview Mistral Large 4 is Mistral AI's frontier mixture-of-experts multimodal model: 1T parameters in total, 49B active per token (52B including embeddings and output layers). It has 72 layers, 128 routed experts (top-4) plus one shared expert, MLA attention with a DSA sparse indexer (top-2048 keys), cross-layer KV sharing, a Pixtral vision encoder and a built-in DSpark drafter. It supports tool calling and reasoning (`reasoning_effort`). | Variant | Checkpoint | Weights | Validated on | |---|---|---|---| | NVFP4 (default) | [`mistralai/Mistral-Large-4-1T-A52B-NVFP4`](https://huggingface.co/mistralai/Mistral-Large-4-1T-A52B-NVFP4) | ModelOpt NVFP4 experts, FP8 attention / shared experts, ~610 GB | GB300, B200 | | FP8 (default on AMD) | [`mistralai/Mistral-Large-4-1T-A52B`](https://huggingface.co/mistralai/Mistral-Large-4-1T-A52B) | FP8 block-scaled, ~1.06 TB (on AMD: MXFP4 experts at load) | GB300, H200, MI350X | | MXFP4 (Quark) | [`amd/Mistral-Large-4-1T-A52B-Quark-MXFP4`](https://huggingface.co/amd/Mistral-Large-4-1T-A52B-Quark-MXFP4) | AMD Quark OCP MXFP4 routed experts, rest unchanged, ~578 GB | MI350 Series | Both checkpoints ship the DSpark drafter. The speculative depth is fixed at 3 by the checkpoint. vLLM serves the model on model runner V2 (the default) with breakable CUDA graphs, FlashMLA sparse attention and an FP8 latent KV cache (`fp8_ds_mla`) by default, so no attention or KV-cache flags are needed. The checkpoint advertises 1,048,576 positions; we validated up to 256k, which is the `--max-model-len` set here. ## Prerequisites - **vLLM**: the `vllm/vllm-openai:mistral-large-4` image. No vLLM release or nightly wheel includes Mistral Large 4 support yet. - **AMD**: the ROCm image `vllm/vllm-openai-rocm:mistral-large-4`. - **mistral_common**: `mistral_common[image] >= 1.13.0` (included in both images). - **MegaMoE** (P/D presets only): the image does not include `nvshmem4py-cu13`; install it in the container, or drop the `--kernel-config` flag. - **Hardware**: validated on NVIDIA GB300 (one 4-GPU tray with NVFP4, or 8 GPUs as two trays with NVFP4 and FP8), 8x B300 (NVFP4 and FP8), 8x B200 (NVFP4) and 2x 8 H200 (FP8, TP16 or PP2 x TP8). FP8 needs ~128 GiB of weights per GPU on 8 GPUs, NVFP4 ~76 GiB. AMD Instinct MI350X (8 GPUs, TP8 or two TP4 replicas) with the ROCm image. ## Choosing a configuration All numbers are NVFP4 on 8x GB300 with DSpark unless stated; the presets carry the full tables. On H200 use TP16 (or TP16 DCP8 for long context), with PP2 x TP8 as the alternative; we took no performance numbers there. | Goal | Configuration | Why | |---|---|---| | Default, best interactivity | TP8 + DSpark | c1 TPOT 3.13 ms (NVFP4) / 3.56 ms (FP8), AL ~2.9; 1.9–2.2x output throughput vs no spec | | Throughput, short/medium inputs | DP8 + EP + DSpark | 209.8 output tok/s/GPU at 8k (c32, 55 tok/s/user), 50.6 at 32k | | Long context, KV-bound | TP8 DCP8 (+ DSpark) | 28.76M KV tokens vs 3.60M on TP8 (~27 concurrent 1M-token requests by KV size) | | Lowest TTFT on long prompts | TP1×PCP8×DCP8 + EP (+ DSpark) | c1 TTFT 0.16 / 0.54 / 1.08 s at 8k / 32k / 60k, ~3x faster than TP8; text only | | Disaggregated throughput | P DP4/EP → D TP4 | 241 output tok/s/GPU at 32k (≥40 tok/s/user), 129 at 60k | | Disaggregated, lowest TTFT | P PCP4×DCP4+EP → D TP4 | c1 TTFT 0.35 / 0.64 / 1.21 s at 8k / 32k / 60k | | Disaggregated, short inputs | P DP4/EP → D DP4/EP | 766 output tok/s/GPU at 8k | | Agentic, KV-bound, no spec | P PCP4×DCP4+EP → D TP4 DCP4 | at 32 sessions TTFT p50 0.72 s vs 14.95 s with D TP4 | Colocated TP8 + DSpark at ≥100 tok/s/user: 110.2 output tok/s/GPU at 8k (c8), 30.5 at 32k (c2), 21.4 at 60k (c2). DCP8 doesn't pay off colocated until KV capacity binds. Disaggregated 4+4 beats colocated 8-GPU serving by 1.8–2.2x at 8k and 4–5x at 32k/60k, because colocated decode shares steps with prefill chunks. Tuning knobs used by the presets: - `--kernel-config '{"moe_backend":"flashinfer_moe_ep_cutedsl"}'` (MegaMoE) with expert parallelism, +14–17% prefill req/s at 32k. Needs `nvshmem4py-cu13`. On EP decode nodes only with `--max-num-batched-tokens` below 512 (256 here). - `--cp-kv-cache-interleave-size 64` on the side that runs DCP in P/D. - Prefix caching (on by default) for agentic workloads, with session-sticky routing to the DP ranks of DP sides. NVFP4 vs FP8 at TP8: 1.09–1.18x faster per token without spec and 1.14–1.22x with DSpark, and ~1.5x the KV capacity. ## AMD (ROCm) Mistral Large 4 runs on AMD Instinct MI350 Series GPUs (gfx950) with the ROCm build of vLLM (image `vllm/vllm-openai-rocm:mistral-large-4`). The routed experts run as OCP MXFP4 on AITER's fused MoE, either from the AMD Quark MXFP4 checkpoint or from the FP8 checkpoint re-quantized at load (`VLLM_FP8_MOE_REQUANT=mxfp4`, layer by layer, so peak load memory stays low). With MXFP4 experts the model fits four GPUs, so one 8-GPU node serves two TP4 replicas. The sparse MLA runs on AITER's MLA decode kernels (`ROCM_AITER_MLA_SPARSE`), the DSA indexer on AITER's gfx950 indexer kernels, and the latent KV cache is plain FP8 (E4M3) by default, 1.86x the capacity of bf16. The builder adds the AITER environment on AMD. | Goal | Configuration | Why | |---|---|---| | Default | TP8 | one replica per node; DSpark at TP8 not yet validated on ROCm | | Interactive, more replicas | TP4 + DSpark (preset) | AL 3.1; TPOT 2.7x / 1.9x / 1.8x lower than no spec at c1 / c8 / c32 | | Lowest latency, 1–4 users | TP4 specialized decode (preset) | one fused kernel per MoE layer: TPOT 1.5x lower at c1; 2.9x with DSpark | MXFP4 vs FP8 experts at TP8: +22–71% output tok/s at 1–64 concurrent requests. ## Accuracy Validated against Mistral's reference implementation on 2x GB300 TP8: per-layer parity and 256k long-context perplexity equal to the reference. GSM8K (1319 questions, 5-shot) scores 0.909 on the FP8 checkpoint and 0.918 with DSpark, which sits at the base server's own noise floor. All P/D and colocated configurations scored 0.965–0.980 on GSM8K-200 (zero-shot chat). ## Client usage ```bash curl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{ "model": "mistralai/Mistral-Large-4-1T-A52B", "messages": [{"role": "user", "content": "Explain sparse attention in two sentences."}], "reasoning_effort": "high" }' ``` Image input (allow more images per prompt with `--limit-mm-per-prompt '{"image": N}'`): ```python from openai import OpenAI client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1") response = client.chat.completions.create( model="mistralai/Mistral-Large-4-1T-A52B", messages=[{ "role": "user", "content": [ {"type": "text", "text": "What does this chart show?"}, {"type": "image_url", "image_url": {"url": "https://example.com/chart.png"}}, ], }], ) print(response.choices[0].message.reasoning, response.choices[0].message.content) ``` ## Limitations - Prefill context parallelism is text-only (`--language-model-only`). - DSpark doesn't run on a DCP decode node in P/D yet. - DSpark wasn't validated with pipeline parallelism. - AMD: NVFP4 checkpoints, DCP / PCP, NIXL P/D and HiSparse aren't supported on ROCm yet; TP8 with DSpark and MI300 Series (gfx942) are not validated. ## References - [Mistral-Large-4-1T-A52B (FP8)](https://huggingface.co/mistralai/Mistral-Large-4-1T-A52B) - [Mistral-Large-4-1T-A52B-NVFP4](https://huggingface.co/mistralai/Mistral-Large-4-1T-A52B-NVFP4) - [Mistral-Large-4-1T-A52B-Quark-MXFP4](https://huggingface.co/amd/Mistral-Large-4-1T-A52B-Quark-MXFP4) - [Cross-layer MLA KV sharing (vllm#64095)](https://github.com/vllm-project/vllm/pull/64095) - [DSpark sliding-window MLA drafts (vllm#63795)](https://github.com/vllm-project/vllm/pull/63795) and [DSpark under DCP (vllm#63796)](https://github.com/vllm-project/vllm/pull/63796) - [NIXL connector](https://docs.vllm.ai/en/latest/features/nixl_connector_usage.html) --- # moonshotai/Kimi-K3 on vLLM Source: https://recipes.vllm.ai/moonshotai/Kimi-K3 Provider: Moonshot AI Released: 2026-06-13T06:42:57.000Z Updated: 2026-10-29 Architecture: moe Parameters: 2.8T Context length: 1,048,576 Min vLLM: 0.29.0 Tasks: multimodal, text Verified hardware: H200, B200, B300, GB200 NVL4, GB300 NVL4, MI355X, Ascend 910C Summary: Pre-release 2.8T-parameter native multimodal MoE with Kimi Delta Attention, Gated MLA, Attention Residuals, and a 1M-token context window ## Variants - default: precision=mxfp4, vram_minimum_gb=1680 Quantization-aware-trained MXFP4 weights with MXFP8 activations; footprint is estimated until the checkpoint is published. - nvfp4: precision=nvfp4, vram_minimum_gb=1650, model_id=RedHatAI/Kimi-K3-NVFP4 RedHatAI compressed-tensors NVFP4 quantization; MoE layers quantized to NVFP4, attention and embeddings at full precision - ascend_w4a8: precision=int4, vram_minimum_gb=1711, model_id=Eco-Tech/Kimi-K3-w4a8 ModelSlim INT4 W4A8 for Ascend NPU. Requires --quantization ascend. Validated on 4× 910C mixed P/D. - fp8: precision=fp8, vram_minimum_gb=3360, model_id=RedHatAI/Kimi-K3-FP8-BLOCK RedHatAI FP8 block quantization ## Recommended base flags ```bash vllm serve 'moonshotai/Kimi-K3' \ --trust-remote-code \ --gpu-memory-utilization \ 0.95 ``` ## Guide ## Overview Kimi K3 is a 2.8-trillion-parameter Mixture-of-Experts model (16 of 896 experts active per token) built on the Kimi Delta Attention (KDA) and Attention Residuals (AttnRes), with a 1M-token context window and native vision. ## Prerequisites - **vLLM:** Use the vllm/vllm-openai:kimi-k3 docker - **CUDA:** The kimi-k3 image ships as a CUDA 13 (cu130) build only — there is no -cu129 tag, and the K3-enabled wheels are not on the cu129 nightly index. The host needs an r580+ NVIDIA driver; on a CUDA 12.9 (r575) host, upgrade the driver or build vLLM from the K3 branch against cu129 PyTorch yourself. - **Hardware:** At least 8x GB300. Multi-node for real production traffic. - **ROCm:** At least 8x MI355X/MI350X. On MI355X the command builder uses the ROCm 10.0 nightly image `vllm/vllm-openai-rocm:nightly-rocm100`; the day-0 `vllm/vllm-openai-rocm:kimi-k3` image predates the DSpark and MoE fixes that path needs (see the [MI355X AgentX preset](#preset-mi355x_agentx)). ## Client Usage Once the vLLM server is running, consume it via the OpenAI-compatible API: ```python import time from openai import OpenAI client = OpenAI( api_key="EMPTY", base_url="http://localhost:8000/v1", timeout=3600 ) messages = [ { "role": "user", "content": [ { "type": "image_url", "image_url": { "url": "https://ofasys-multimodal-wlcb-3-toshanghai.oss-accelerate.aliyuncs.com/wpf272043/keepme/image/receipt.png" } }, { "type": "text", "text": "Read all the text in the image." } ] } ] start = time.time() response = client.chat.completions.create( model="moonshotai/Kimi-K3", messages=messages, max_tokens=2048 ) print(f"Response costs: {time.time() - start:.2f}s") print(f"Generated text: {response.choices[0].message.content}") ``` ## Notes - **Cross node communitcation**: Use `--all2all-backend deepep_v2` for RDMA and `--all2all-backend flashinfer_nvlink_one_sided` for NVLink. - **MoE backend**: Recommend to use `deep_gemm_mega_moe` for DEP deployments with cross-node NVLink e.g. GB200 and GB300. Note that DeepGEMM MegaMoE is not compatible with cross-node RDMA. - **Rust Frontend**: Selected by default in the command builder. Switch to Python if you encounter unsupported features or compatibility issues. - **Model Runner v2**: `VLLM_USE_V2_MODEL_RUNNER=1`: Model Runner v2 fully supports this model and can be enabled if needed. - **Tool calling**: K3 occasionally emit a tool-call format its own parser doesn't expect. Suggest to run do schema validation and retry. - **AMD (MI355X / MI350X, CDNA4 gfx950)**: `VLLM_ROCM_USE_AITER_MOE_SITUV2=1` (set above) enables AITER's SiTUv2 a4w4 FlyDSL MoE path on builds that carry vllm-project/vllm#53940 (ROCm nightlies since 2026-09-16). v0.30.0 predates it: there the variable is ignored and the MoE runs the default a16w4 path. Unset it or set `0` for the default a16w4 path. On builds with #53940, `VLLM_ROCM_USE_AITER_MOE_SITUV2_A8W4` is a deprecated alias for the same a4w4 path and vLLM removes `AITER_SITUV2_A8W4` at startup while the toggle is on; on v0.30.0 and older, `VLLM_ROCM_USE_AITER_MOE_SITUV2_A8W4=1` selects the a8w4 path instead. Do not set `AITER_SITUV2_A8W4=1` by itself: vLLM does not derive its MoE weight layout from that AITER flag (vllm-project/vllm#53954 is open), so the two can disagree and corrupt output without an error. - **max-model-len**: Adjust max-model-len for different benchmark scenarios for best performance. - **RDMA**: If RDMA is enabled, set `UCX_TLS="rc,cuda_copy"` to make sure KV Cache transfer goes through RDMA. - **MNNVL environments** (GB200/GB300 NVL): recommend adding `NCCL_MNNVL_ENABLE=1`, `NCCL_CUMEM_ENABLE=1`, and `NCCL_NVLS_ENABLE=1`. - **mlx5 dmabuf registration failures**: if engine init fails with "NCCL error: unhandled system error" and the log shows `mlx5dv_reg_dmabuf_mr` errno 524, the kernel/driver lacks mlx5 dmabuf support (NCCL 2.28 registers dmabuf by default). Set `NCCL_DMABUF_ENABLE=0` to fall back to `nvidia_peermem` (must be loaded on the nodes) — still GPUDirect RDMA. - **FP8 KV**: If FP8 KV cache is needed, please also add `--attention-config '{"use_prefill_query_quantization":true,"mla_prefill_backend":"TOKENSPEED_MLA"}'` when serving vLLM. `TOKENSPEED_MLA` is what the Blackwell profile above emits (paired with `--attention-backend TOKENSPEED_MLA` for decode), but it is not the only supported choice: vLLM 0.27.1 registers three MLA prefill backends — `FLASHINFER`, `TRTLLM_RAGGED` and `TOKENSPEED_MLA` (see `vllm/v1/attention/backends/mla/prefill/registry.py`) — and any of them may be substituted for `mla_prefill_backend` alongside `use_prefill_query_quantization`. Which one wins depends on the input-length distribution, so benchmark before switching. Those backends do not run on ROCm: on MI355X the builder pairs FP8 KV with `--attention-config '{"mla_prefill_backend":"ROCM_AITER_FA"}'` instead. - **Prefix-match unit**: K3 is a hybrid model: its MLA attention layers and KDA (Mamba-like) layers form two KV-cache groups under the hybrid KV-cache manager. With prefix caching on, vLLM pads the attention block size up to match the Mamba state page, so both groups resolve to one (large) block size and the default prefix-cache hit boundary lands on that block — very coarse. The Blackwell profile sets `--prefix-match-unit 128`: the inflation formula always yields a multiple of 128, so 128 divides whatever block size results, and it aligns with the MLA kernel's native block boundary, giving finer prefix-hit granularity. The value is only effective when prefix caching (or a KV connector) is active; it is a no-op otherwise, and it is safe under all tensor-parallel layouts. - **Decode context parallelism (DCP)**: the qualified DCP layouts are part of the presets that ran them: DCP8 on the GB200 aggregated TP8 x PP2 and GB300 disaggregated TP8 pools, DCP16 on the GB200 TP16 disaggregated pools, DCP16/DCP8 at the GB300 low-latency points, and DCP8 in the MI355X AgentX cells. Each pairs `--decode-context-parallel-size` (a divisor of the tensor-parallel size) with `--dcp-comm-backend a2a` and an MLA attention backend that supports it (`TOKENSPEED_MLA` with `mla_prefill_backend: TRTLLM_RAGGED` on Blackwell, `ROCM_AITER_MLA` with `mla_prefill_backend: ROCM_AITER_FA` on MI355X). On builds that carry them, the `VLLM_USE_DIRECT_DCP_A2A`, `VLLM_USE_DIRECT_DCP_Q_GATHER` and `VLLM_USE_DIRECT_DCP_KV_GATHER` knobs default to auto; set them to `1` to pin the direct DCP ops or `0` to disable them, as the MI355X cells do. - Ascend W4A8 checkpoint (ModelScope): https://www.modelscope.cn/models/Eco-Tech/Kimi-K3-w4a8 - vLLM Ascend tutorial: https://docs.vllm.ai/projects/ascend/en/v0.23.0/tutorials/models/Kimi-K3.html ## AMD MI355X: disaggregated prefill and decode This example uses one TP8/DCP8 prefill worker and one TP8/DCP8 decode worker on separate 8-GPU MI355X nodes. MoRIIO transfers the KV cache in READ mode; `MultiConnector` combines it with `SimpleCPUOffloadConnector` on prefill only. Both roles use the same target and DSpark draft checkpoints and FP8 KV. These standalone commands are based on the 1P1D c48 configuration in [InferenceX#4391](https://github.com/SemiAnalysisAI/InferenceX/pull/4391), with real block verification for serving and accuracy evaluation. They are separate from the command builder's NIXL PD configuration. ### Prerequisites - Make [moonshotai/Kimi-K3](https://huggingface.co/moonshotai/Kimi-K3) and [Inferact/Kimi-K3-DSpark](https://huggingface.co/Inferact/Kimi-K3-DSpark) available inside every worker container. The commands below use `/models/Kimi-K3` and `/models/Inferact-Kimi-K3-DSpark` as example container paths; adjust them to match your deployment. Use identical checkpoint revisions on all workers. - Use the worker and router images below, with the same worker image build on every node. The worker image contains the required vLLM support; no source patches are needed. - Expose all eight GPUs, `/dev/infiniband`, host networking, and sufficient shared memory to each worker container. Allow locked memory for RDMA. The container's verbs provider must be compatible with the host NIC and driver. - The prefill offload budget is 2,399 GB of host DRAM **across all eight ranks**, in addition to weight-loading and runtime memory. Reduce `CPU_OFFLOAD_BYTES` if the host cannot accommodate that allocation; the documented configurations use `2399000000000`. - Set `SOCKET_IFNAME` and `RDMA_DEVICES` to your network interface and HCA list. `ROUTER_IP` and each `WORKER_IP` must be reachable addresses, not `0.0.0.0`. Allow router TCP ports 8000 and 36367, worker HTTP port 6100, and MoRIIO handshake/notification ports 26000–26007 and 27000–27007, plus the site's RDMA traffic. ```bash WORKER_IMAGE=vllm/vllm-openai-rocm:nightly ROUTER_IMAGE=vllm/vllm-router:nightly ``` ### Start the router Start one router before the workers. Discovery registers workers automatically; do not add static `--prefill` or `--decode` endpoints. ```bash docker run --rm --network host --entrypoint vllm-router "$ROUTER_IMAGE" \ --vllm-pd-disaggregation \ --kv-connector moriio \ --vllm-discovery-address 0.0.0.0:36367 \ --host 0.0.0.0 --port 8000 \ --worker-startup-timeout-secs 7200 \ --policy consistent_hash \ --prefill-policy consistent_hash \ --decode-policy consistent_hash \ --log-level info ``` ### Start the workers Save the following as `kimi-k3-pd.sh` on each worker. Run it with Bash **inside the worker image**, with the devices and model paths described above. ```bash #!/usr/bin/env bash set -euo pipefail ROLE=${1:?Usage: bash kimi-k3-pd.sh prefill|decode} : "${ROUTER_IP:?Set the router reachable IP}" : "${WORKER_IP:?Set this worker reachable IP}" : "${SOCKET_IFNAME:?Set the host network interface}" : "${RDMA_DEVICES:?Set the comma-separated HCA list}" CPU_OFFLOAD_BYTES=${CPU_OFFLOAD_BYTES:-2399000000000} export NCCL_SOCKET_IFNAME="$SOCKET_IFNAME" GLOO_SOCKET_IFNAME="$SOCKET_IFNAME" export NCCL_IB_HCA="$RDMA_DEVICES" export VLLM_ROCM_USE_AITER=1 AITER_BF16_FP8_MOE_BOUND=0 export AITER_FLYDSL_STAGE2_FP8=1 export VLLM_ROCM_AITER_MLA_ASM_PADDING=asm VLLM_ROCM_AITER_MLA_DCP_VERIFY=asm export VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4 export VLLM_ROCM_QUICK_REDUCE_MAX_SIZE_BYTES_MB=256 export SAFETENSORS_FAST_GPU=1 export VLLM_SSM_CONV_STATE_LAYOUT=DS VLLM_KV_CACHE_LAYOUT=HND export NCCL_DMABUF_ENABLE=0 HSA_ENABLE_IPC_MODE_LEGACY=1 HIP_FORCE_DEV_KERNARG=1 export PYTHONHASHSEED=42 PYTHONNOUSERSITE=1 export VLLM_USE_DIRECT_DCP_A2A=0 VLLM_USE_DIRECT_DCP_Q_GATHER=0 VLLM_USE_DIRECT_DCP_KV_GATHER=0 export VLLM_ENGINE_READY_TIMEOUT_S=7200 VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=1200 export MORI_IO_SQ_BACKOFF_TIMEOUT_US=50000 export MORI_IO_QP_MAX_SEND_WR=16384 MORI_IO_QP_MAX_CQE=32768 MORI_IO_QP_MAX_SGE=2 export MORI_IO_TC_DISABLE=0 CUSTOM_OPS='["+fused_rms_norm_gated","+quant_fp8","+grouped_topk","+sparse_attn_indexer","none"]' case "$ROLE" in prefill) MAX_SEQS=96 BATCHED=8192 KV_ROLE=kv_producer export HSA_NO_SCRATCH_RECLAIM=0 COMPILATION="{\"mode\":3,\"cudagraph_mode\":\"NONE\",\"max_cudagraph_capture_size\":0,\"custom_ops\":$CUSTOM_OPS}" ;; decode) MAX_SEQS=96 BATCHED=512 KV_ROLE=kv_consumer CAPTURES="$(seq -s, 1 128),$(seq -s, 130 10 240)" COMPILATION="{\"mode\":3,\"cudagraph_mode\":\"FULL_DECODE_ONLY\",\"max_cudagraph_capture_size\":240,\"cudagraph_capture_sizes\":[$CAPTURES],\"custom_ops\":$CUSTOM_OPS}" ;; *) echo "Role must be prefill or decode" >&2; exit 1 ;; esac MORI="{\"kv_connector\":\"MoRIIOConnector\",\"kv_role\":\"$KV_ROLE\",\"kv_load_failure_policy\":\"fail\",\"kv_connector_extra_config\":{\"backend\":\"rdma\",\"read_mode\":true,\"proxy_ip\":\"$ROUTER_IP\",\"proxy_ping_port\":\"36367\",\"host_ip\":\"$WORKER_IP\",\"http_port\":\"6100\",\"handshake_port\":\"26000\",\"notify_port\":\"27000\"}}" KV_CONFIG=$MORI if [ "$ROLE" = prefill ]; then CPU="{\"kv_connector\":\"SimpleCPUOffloadConnector\",\"kv_role\":\"kv_both\",\"kv_connector_extra_config\":{\"cpu_bytes_to_use\":$CPU_OFFLOAD_BYTES,\"lazy_offload\":false}}" KV_CONFIG="{\"kv_connector\":\"MultiConnector\",\"kv_role\":\"kv_both\",\"kv_load_failure_policy\":\"fail\",\"kv_connector_extra_config\":{\"connectors\":[$MORI,$CPU]}}" fi exec vllm serve /models/Kimi-K3 \ --host 0.0.0.0 --port 6100 \ --served-model-name moonshotai/Kimi-K3 \ --trust-remote-code \ --tensor-parallel-size 8 \ --decode-context-parallel-size 8 \ --cp-kv-cache-interleave-size 1 \ --dcp-comm-backend a2a \ --moe-backend auto \ --load-format fastsafetensors \ --gpu-memory-utilization 0.90 \ --language-model-only \ --enable-auto-tool-choice \ --tool-call-parser kimi_k3 --reasoning-parser kimi_k3 \ --max-model-len 1048576 \ --stream-interval 10 \ --enable-prefix-caching --prefix-match-unit 128 \ --kv-cache-dtype fp8 --mamba-ssm-cache-dtype bfloat16 \ --block-size 128 \ --attention-backend ROCM_AITER_MLA \ --attention-config '{"mla_prefill_backend":"ROCM_AITER_FA","use_prefill_query_quantization":true}' \ --speculative-config '{"model":"/models/Inferact-Kimi-K3-DSpark","method":"dspark","num_speculative_tokens":4,"attention_backend":"ROCM_AITER_MLA","kv_cache_dtype":"fp8","draft_sample_method":"probabilistic","rejection_sample_method":"block"}' \ --max-num-seqs "$MAX_SEQS" \ --max-num-batched-tokens "$BATCHED" \ --compilation-config "$COMPILATION" \ --kv-transfer-config "$KV_CONFIG" ``` On the prefill worker, replace the example addresses, interface, and HCAs with your site's values: ```bash export ROUTER_IP=192.0.2.10 WORKER_IP=192.0.2.11 export SOCKET_IFNAME=eth0 RDMA_DEVICES=rdma0,rdma1,rdma2,rdma3,rdma4,rdma5,rdma6,rdma7 bash kimi-k3-pd.sh prefill ``` On the decode worker: ```bash export ROUTER_IP=192.0.2.10 WORKER_IP=192.0.2.12 export SOCKET_IFNAME=eth0 RDMA_DEVICES=rdma0,rdma1,rdma2,rdma3,rdma4,rdma5,rdma6,rdma7 bash kimi-k3-pd.sh decode ``` Prefill uses an 8,192-token batch budget without CUDA graphs; decode uses a 512-token budget with `FULL_DECODE_ONLY` graphs. Both roles use a 96-sequence limit and four DSpark draft tokens, with BF16 SSM. INT4 QuickReduce is a quantized collective optimization; validate accuracy for your workload and keep the same setting on all workers. The RDMA queue backoff is 50,000 microseconds (50 ms). Prefill scratch reclamation is enabled with `HSA_NO_SCRATCH_RECLAIM=0`, matching the c48 configurations. ### Scaling and concurrency tuning For 1P2D or 1P3D, run the decode command on additional nodes with their own `WORKER_IP`. Discovery registers each worker automatically; the port ranges can be reused on separate hosts. These are router-balanced replicas, not a cross-node TP group. For the offload configurations, let `C` be global client concurrency, `N` the number of decode workers, and `K` the draft length: ```text P max-num-seqs = 2 * C D max-num-seqs = ceil(2 * C / N) D graph max G = (K + 1) * C ``` These are tuning starting points; sequence limits provide scheduling headroom. Keep P/D SSM dtypes, DCP settings, and cache layouts matched. | Topology | Global concurrency | P / per-D max sequences | Draft tokens | SSM state | P CPU offload | D graph dense / max | | --- | ---: | --- | ---: | --- | --- | --- | | 1P1D | 8 | 1 / 1 | 7 | FP32 | None | Explicit `[1, 8]` | | 1P1D | 48 | 96 / 96 | 4 | BF16 | 2399 GB | 128 / 240 | | 1P2D | 24 | 48 / 24 | 4 | BF16 | 2399 GB | 64 / 120 | | 1P2D | 48 | 96 / 48 | 4 | BF16 | 2399 GB | 128 / 240 | | 1P3D | 24 | 48 / 16 | 4 | BF16 | 2399 GB | 64 / 120 | For c8, set both sequence limits to 1, use K7 and FP32 SSM, and remove prefill CPU offload (use MoRIIO alone). On decode only, set `VLLM_USE_V2_MODEL_RUNNER=1`, add `--additional-config '{"decode_backend":"latency_specialized"}'`, and use graph sizes `[1, 8]` with maximum 8. Benchmark concurrency sweeps and harness settings remain in the linked InferenceX configuration. Benchmark runs use its measured golden acceptance curve; the serving commands retain real block verification. ### Verify routed inference Wait for every worker to load and register with the router. Check each worker's `/health` endpoint and the router's `/v1/models`, then send a request through the router: ```bash curl --fail-with-body "http://${ROUTER_IP}:8000/v1/chat/completions" \ -H 'Content-Type: application/json' \ -d '{"model":"moonshotai/Kimi-K3","messages":[{"role":"user","content":"What is 17 + 25?"}],"reasoning_effort":"low","temperature":0.6,"max_tokens":4096}' ``` Confirm that a final answer is returned, and inspect the prefill/decode logs for successful KV transfers and transfer errors. Health checks alone do not exercise RDMA. Preserve `reasoning_content` and tool calls when building multi-turn conversations, as described in the [model card](https://huggingface.co/moonshotai/Kimi-K3#6-model-usage). --- # autotrust/JEV-27B-VL on vLLM Source: https://recipes.vllm.ai/autotrust/JEV-27B-VL Provider: AutoTrust Released: 2026-09-30T05:02:23.000Z Updated: 2026-10-28 Architecture: dense Parameters: 27B Context length: 262,144 Min vLLM: 0.32.0 Tasks: multimodal, text Summary: Dense 27B vision-language decision model on a Qwen3.8-27B backbone, with a rank-32 LoRA (jev-decision) for calibrated choices over text and images. ## Variants - default: precision=bf16, vram_minimum_gb=67 BF16 weights (about 52–56 GB). The model card's quick start targets one GPU with 80 GB or more; this recipe does not mark any GPU verified. ## Recommended base flags ```bash vllm serve 'autotrust/JEV-27B-VL' \ --served-model-name \ autotrust/JEV-27B-VL \ --enable-lora \ --max-lora-rank \ 32 \ --lora-modules \ jev-decision=JEV-27B-VL/adapter_vllm \ --logprobs-mode \ processed_logprobs \ --max-model-len \ 32768 \ --enable-prefix-caching \ --mamba-cache-mode \ align \ --limit-mm-per-prompt \ {"image": 8} \ --max-num-seqs \ 8 \ --trust-request-chat-template ``` ## Guide ## Overview [JEV-27B-VL](https://huggingface.co/autotrust/JEV-27B-VL) is AutoTrust's vision-language decision model. The checkpoint is a dense BF16 `Qwen3_5ForConditionalGeneration` model (about 27.8B parameters, vision tower included) continued from [Qwen/Qwen3.8-27B](https://huggingface.co/Qwen/Qwen3.8-27B). System 2 is the served checkpoint, named `autotrust/JEV-27B-VL`. System 1 is the rank-32 LoRA `jev-decision` (`adapter_vllm/`, `r` 32, `lora_alpha` 64), used for calibrated choice probabilities over text or images. This recipe has no `verified` hardware entry. The model card's quick start asks for one GPU with 80 GB or more, and the card's published latency figures are from a B200. Those runs are the author's, not a badge on this page. ## Prerequisites - **vLLM:** >= 0.32.0. Earlier releases are not supported for this checkpoint. - **Weights:** `hf download autotrust/JEV-27B-VL --local-dir JEV-27B-VL` so the LoRA path below exists. The same `adapter_vllm` directory is in the [ModelScope](https://modelscope.cn/models/autotrust/JEV-27B-VL) checkout; if that folder is not named `JEV-27B-VL`, point the module at `/adapter_vllm` (the HF snapshot path ends in `snapshots//adapter_vllm`). - **Docker:** the image only bind-mounts the Hugging Face cache. Mount the checkout into the container and keep `jev-decision` pointed at that `adapter_vllm` directory. - **Context:** `text_config.max_position_embeddings` is 262144. The validated command uses `--max-model-len 32768` (the `serve.sh` default). The card says a full 256K prompt needs about 17 GB of KV cache on top of the weights, and suggests `--max-model-len 131072` on an 80 GB GPU. ## Decision LoRA and serve flags These flags are the model card's validated `vllm serve` line. They are base arguments, not optional toggles. - **`jev-decision`** — `--enable-lora`, `--max-lora-rank 32`, and `--lora-modules jev-decision=JEV-27B-VL/adapter_vllm`. The module name stays `jev-decision`. The path is the local checkout `JEV-27B-VL/adapter_vllm`, or the `adapter_vllm` directory inside an HF or ModelScope checkout. - **`--mamba-cache-mode align`** — the Qwen3.8 hybrid backbone mixes full attention with Mamba-style layers; the card serves it in `align` mode. - **`--limit-mm-per-prompt '{"image": 8}'`** — at most 8 images per request. - **`--max-num-seqs 8`** — required. The card says that with more than 8 sequences in one batch, vLLM's LoRA path for this multimodal class returns wrong System 1 probabilities. Extra requests queue. - **`--logprobs-mode processed_logprobs`** — System 1 reads post-processor logprobs. On `/v1/chat/completions` or `/v1/completions` for `jev-decision`, the card also says to send `top_k: 0` and `top_p: 1.0`: `generation_config.json` sets `top_k` 20 and `top_p` 0.95, and those defaults would truncate the returned distribution. - **`--trust-request-chat-template`** — image decisions send a raw decision chat template on the request. System 2 keeps the checkpoint's own template. - **`--served-model-name autotrust/JEV-27B-VL`** — public name for System 2. `--enable-prefix-caching` matches the card's serve line. `POST /v1/decide` is not part of stock `vllm serve`. The checkpoint ships `serve_decide.py`, which starts the OpenAI server with these same flags and adds that route. Stock `vllm serve` still exposes System 2 as `autotrust/JEV-27B-VL` and System 1 as the LoRA model `jev-decision`. ## Client usage System 2 (the base checkpoint), with an image: ```python client.chat.completions.create( model="autotrust/JEV-27B-VL", messages=[{ "role": "user", "content": [ {"type": "image_url", "image_url": {"url": image_url}}, {"type": "text", "text": "What does this image show?"}, ], }], ) ``` System 1 through the checkpoint's `/v1/decide` route (`serve_decide.py`): ```bash curl localhost:8000/v1/decide -H 'Content-Type: application/json' -d '{ "kind": "choice", "state": "Customer: my card was charged twice for one coffee.", "question": "Which team should handle this?", "options": ["billing", "shipping", "tech support"]}' ``` Text-only System 1 calls the LoRA directly (`model` `jev-decision`). The card says those text decisions match [autotrust/JEV-27B](https://huggingface.co/autotrust/JEV-27B). ## References - [Hugging Face model card](https://huggingface.co/autotrust/JEV-27B-VL) - [ModelScope (autotrust/JEV-27B-VL)](https://modelscope.cn/models/autotrust/JEV-27B-VL) - [ModelScope mirror listed on the card (AutoTrust-Labs/JEV-27B-VL)](https://www.modelscope.ai/models/AutoTrust-Labs/JEV-27B-VL) - [Base model: Qwen3.8-27B](https://huggingface.co/Qwen/Qwen3.8-27B) --- # deepseek-ai/DeepSeek-V4-Pro on vLLM Source: https://recipes.vllm.ai/deepseek-ai/DeepSeek-V4-Pro Provider: DeepSeek Released: 2026-04-22T06:04:45.000Z Updated: 2026-10-28 Architecture: moe Parameters: 1600B Context length: 1,048,576 Min vLLM: 0.20.0 Tasks: text Verified hardware: H200, B200, GB200 NVL4, B300, GB300 NVL4, MI355X Summary: DeepSeek V4 flagship MoE (1.6T total / 49B active) with hybrid CSA+HCA attention, manifold-constrained hyper-connections, Muon-trained on 32T+ tokens, and three-tier reasoning. ## Variants - default: precision=fp8, vram_minimum_gb=960, model_id=deepseek-ai/DeepSeek-V4-Pro-0813 Official DeepSeek-V4-Pro release (2026-08-13) superseding the preview — substantially stronger agentic performance, with the DSpark draft module attached. - fp8: precision=fp8, vram_minimum_gb=960 Native FP4+FP8 mixed checkpoint (MoE experts FP4, remaining params FP8) - nvfp4: precision=nvfp4, vram_minimum_gb=960, model_id=nvidia/DeepSeek-V4-Pro-NVFP4 NVIDIA modelopt NVFP4 checkpoint (MoE experts NVFP4; attention, shared experts, router head, and MTP stay FP8) for Blackwell GPUs. - dspark: precision=fp8, vram_minimum_gb=960, model_id=deepseek-ai/DeepSeek-V4-Pro-DSpark Preview FP4+FP8 weights with the DSpark draft module baked in — same base checkpoint as the fp8 variant plus a fused speculative decoder. - redhat_nvfp4: precision=nvfp4, vram_minimum_gb=1000, model_id=RedHatAI/DeepSeek-V4-Pro-NVFP4-FP8 RedHatAI NVFP4+FP8 mixed-precision checkpoint ## Recommended base flags ```bash vllm serve 'deepseek-ai/DeepSeek-V4-Pro' \ --trust-remote-code \ --kv-cache-dtype \ fp8 \ --block-size \ 256 ``` ## Guide ## Overview DeepSeek-V4-Pro is the flagship of the V4 preview family: a 1.6T-total / 49B-active Mixture-of-Experts model. It pairs a **hybrid attention stack** — Compressed Sparse Attention (CSA) + Heavily Compressed Attention (HCA) — with **Manifold-Constrained Hyper-Connections (mHC)** to reach 27% of V3.2's per-token inference FLOPs and 10% of V3.2's KV cache at 1M context. Pre-trained on 32T+ tokens with the **Muon optimizer** for faster convergence; post-training is a two-stage pipeline (domain-specific expert cultivation + unified consolidation via on-policy distillation). Checkpoint is **FP4+FP8 mixed**: MoE expert weights are stored in FP4 while the remaining (attention / norm / router) params stay in FP8. An **NVFP4 variant** (`nvidia/DeepSeek-V4-Pro-NVFP4`) is also available — NVIDIA modelopt re-quantizes the MoE experts to standard NVFP4 while attention, shared experts, router head, and MTP stay FP8. Pick it from the **Variant** row; it runs on Blackwell GPUs with the FP4 indexer cache. ## Checkpoints Four checkpoints are on the **Variant** row. They differ in *which weights you serve* and *which draft module ships with them* — the speculative method itself is picked on the **Spec Decoding** row. | Variant | Repo | Draft | Notes | | :-- | :-- | :-- | :-- | | **FP8 (0813)** (default) | `deepseek-ai/DeepSeek-V4-Pro-0813` | DSpark | Official release, preview structure + DSpark | | **FP8 (Preview)** | `deepseek-ai/DeepSeek-V4-Pro` | MTP | Preview FP4+FP8 mixed weights | | **NVFP4** | `nvidia/DeepSeek-V4-Pro-NVFP4` | MTP | modelopt re-quant, Blackwell | | **DSpark** | `deepseek-ai/DeepSeek-V4-Pro-DSpark` | DSpark | Preview weights + fused DSpark module | **0813 is the official DeepSeek-V4-Pro release** and the default here, superseding the preview with substantially stronger agentic capability — 87.9 on Terminal Bench 2.1, 62.7 on DeepSWE and 42.7 on HLE (60.0 with tools), versus 72.1, 12.8 and 37.7 for the preview. It is built on the preview model structure with a DSpark speculative decoding module attached. The preview weights remain available as the **FP8 (Preview)** variant for reproducing earlier results. **DSpark** is *not* new weights — it is the preview checkpoint with a speculative decoding module attached (see [DeepSpec](https://github.com/deepseek-ai/DeepSpec)). Both fused checkpoints add a `dspark_*` block to `config.json`; the preview and NVFP4 checkpoints have no such block, which is why the DSpark method is offered only on the two variants that carry the draft. Selecting either one auto-enables **Spec Decoding** with `method: dspark`, emitting: ``` --speculative-config '{"method":"dspark","num_speculative_tokens":7,"draft_sample_method":"probabilistic"}' ``` The fused checkpoints are ~893 GB on disk versus ~865 GB for the preview — the draft module is the difference. Both require **vLLM 0.25.0**, above this recipe's 0.20.0 baseline; the Install block bumps automatically when either is selected. ## Reasoning modes The chat template exposes three reasoning-effort modes: - **Non-think** — fast, intuitive responses. - **Think High** — explicit chain-of-thought for complex problem-solving and planning. - **Think Max** — maximum reasoning effort; requires `--max-model-len >= 393216` (384K tokens) to avoid truncation. Recommended sampling: `temperature = 1.0`, `top_p = 1.0`. On the **0813** variant the effort levels are named `low` / `high` / `max`, and DeepSeek recommends `top_p = 0.95` for agentic scenarios (`1.0` otherwise) with `temperature = 1.0`. Allow up to **384K output tokens** at the `high` and `max` levels. Note that the 0813 release ships no Jinja chat template — the repo provides an `encoding/` folder with `encode_messages` / `parse_message_from_completion_text` helpers instead. Serving through vLLM with `--tokenizer-mode deepseek_v4` (the **Tool Calling** pill) applies the built-in DeepSeek-V4 encoding, so the OpenAI- compatible endpoint works without the helper scripts. ### OpenAI Client Example For DeepSeek-V4, keep reasoning controls in `chat_template_kwargs`, as it exposes a custom **Think Max** mode via `"reasoning_effort": "max"`. ```python from openai import OpenAI client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY") model = "deepseek-ai/DeepSeek-V4-Pro-0813" messages = [{"role": "user", "content": "What is 17*19? Return only the final integer."}] # Non-think resp = client.chat.completions.create( model=model, messages=messages, ) # Think High resp = client.chat.completions.create( model=model, messages=messages, extra_body={ "chat_template_kwargs": { "thinking": True, "reasoning_effort": "high", }, }, ) # Think Max resp = client.chat.completions.create( model=model, messages=messages, extra_body={ "chat_template_kwargs": { "thinking": True, "reasoning_effort": "max", }, }, ) ``` ## Recommended deployments - **B300 (8× GPU)**: single-node DP + EP with `--data-parallel-size 8`. - **H200 (8× GPU)**: DP + EP with `--data-parallel-size 8`. Context is capped at 800K tokens (`--max-model-len 800000`) to leave KV headroom with dense params replicated across ranks — applies to both single-node and multi-node H200. - **MI355X (8× GPU)**: validated with ROCm + AITER (`VLLM_ROCM_USE_AITER=1`), `--gpu-memory-utilization 0.9`, `--max-num-seqs 128`, `--max-num-batched-tokens 8192`, and `--distributed-executor-backend mp`. - **GB200 NVL4 (4× GPU per tray)**: the ~960 GB mixed-precision checkpoint does not fit on one tray; run multi-node DP + EP across **2 trays** (8 GPUs total) with `--data-parallel-size 8`. Pick the "Multi-Node" tab and set nodes to 2. ### MI355X (8×288GB) ```bash export VLLM_ROCM_USE_AITER=1 vllm serve deepseek-ai/DeepSeek-V4-Pro \ --host localhost \ --port 8001 \ --dtype auto \ --kv-cache-dtype fp8 \ --tensor-parallel-size 8 \ --max-num-seqs 512 \ --max-num-batched-tokens 8192 \ --distributed-executor-backend mp \ --trust-remote-code \ --gpu-memory-utilization 0.9 \ --tokenizer-mode deepseek_v4 \ --reasoning-parser deepseek_v4 \ --tool-call-parser deepseek_v4 \ --enable-auto-tool-choice \ --compilation-config '{"mode": 3, "cudagraph_mode": "FULL_DECODE_ONLY"}' ``` #### MI355X is validated on GSM8K dataset: ##### Launch command ```bash MODEL=deepseek-ai/DeepSeek-V4-Pro lm_eval --model local-completions \ --model_args model=$MODEL,base_url=http://0.0.0.0:8001/v1/completions,num_concurrent=128,max_retries=10,max_gen_toks=2048,timeout=60000 \ --batch_size auto \ --tasks gsm8k \ --num_fewshot 8 \ --output_path . 2>&1 | tee -a eval.log ``` ##### Reported result ```text local-completions ({'model': 'deepseek-ai/DeepSeek-V4-Pro', 'base_url': 'http://0.0.0.0:8001/v1/completions', 'num_concurrent': 128, 'max_retries': 10, 'max_gen_toks': 2048, 'timeout': 60000}), gen_kwargs: ({}), limit: None, num_fewshot: 8, batch_size: auto |Tasks|Version| Filter |n-shot| Metric | |Value | |Stderr| |-----|------:|----------------|-----:|-----------|---|-----:|---|-----:| |gsm8k| 3|flexible-extract| 8|exact_match|↑ |0.9538|± |0.0058| | | |strict-match | 8|exact_match|↑ |0.9545|± |0.0057| ``` ## KV Cache Offloading Agentic and multi-turn workloads reuse long prefixes whose KV state can exceed on-GPU capacity. The **KV Offload** row attaches a host-DRAM KV tier to any serving strategy — pick one of three connectors: - **Simple** — `SimpleCPUOffloadConnector`: spills KV blocks to a per-rank region of CPU DRAM on each node. The simplest way to extend effective KV capacity for a single instance. - **Mooncake** — `MooncakeStoreConnector`: pools CPU DRAM into a distributed shared KV store — either embedded (each rank's vLLM worker runs a Mooncake client that donates a DRAM segment) or standalone (a per-node `mooncake_store_service` owns the node's DRAM and contributes it to the pool, decoupling cache lifetime from the engine). A cluster-wide `mooncake_master` coordinates the pool; across 2+ instances, GPU workers share it for cross-instance prefix reuse. Install from the extra-install block above; see the [Mooncake docs](https://kvcache-ai.github.io/Mooncake/). - **LMCache** — `LMCacheMPConnector`: a node-local KV pool served by a companion `lmcache server` process, launched before `vllm serve`. Single-node strategies only. Install from the extra-install block above; see the [LMCache docs](https://docs.lmcache.ai/). ## Agentic serving on GB200/GB300 (DSpark) For agentic serving, use the following common settings with [DeepSeek-V4-Pro-0813](https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro-0813/tree/72e1d3230f6c080a530b0a1d46f8eb4602340597) and its native DSpark draft. These are separate from the command-builder defaults. ### Common settings Set `VLLM_USE_V2_MODEL_RUNNER=1` and use these engine arguments: ```text --tokenizer-mode deepseek_v4 --trust-remote-code --kv-cache-dtype fp8 --block-size 256 --no-disable-hybrid-kv-cache-manager --disable-flashinfer-autotune --attention-config '{"backend":"FLASHINFER_MLA_SPARSE_DSV4","use_prefill_query_quantization":true,"indexer_kv_dtype":"mxfp4"}' --speculative-config '{"method":"dspark","num_speculative_tokens":6,"draft_sample_method":"probabilistic","rejection_sample_method":"block"}' ``` The example uses `K=6`, where `K` is `num_speculative_tokens`. Tune K and the prefill/decode (PD) ratio for each target concurrency to balance throughput, per-user latency and time to first token. Use the same K on both PD roles. For aggregate TP serving, use `--disable-custom-all-reduce`, `VLLM_ALLREDUCE_USE_FLASHINFER=1`, `VLLM_FLASHINFER_ALLREDUCE_BACKEND=auto`; do not select an EP MoE backend. For DEP (TP1 with data and expert parallelism), both pools instead use `--enable-expert-parallel --enable-ep-weight-filter --enable-cumem-allocator`, `--moe-backend deep_gemm_amxf4_mega_moe` and `VLLM_DSV4_MEGA_FP8_COMBINE=1`. This MoE configuration supports the native mixed FP4/FP8 checkpoint, unlike the FP8-only `deep_gemm_mega_moe` setting above. For aggregate and decode, set `cudagraph_mode=FULL_DECODE_ONLY` and `mode=0` in `--compilation-config`, with explicit capture sizes `n*(K+1)` for `n=1..S`. `S` is `max-num-seqs` per DP rank, or per engine for pure TP. Set the maximum capture size to `S*(K+1)`; aggregate serving can also include `n*K` sizes. Recompute the capture sizes when K or S changes. Size `max-num-batched-tokens` to cover verification tokens plus any prefill work scheduled on that engine. ### Scheduling, NUMA binding, and streaming **DEP decode scheduling:** the decode example below uses: ```text --prefill-schedule-interval 16 --long-prefill-token-threshold 512 ``` The interval controls how often DP ranks admit new prefill requests. Increasing it can balance work across ranks but delays admission. The threshold limits long-prefill chunk size when requests share the token budget. Compare these overrides with leaving both flags unset, measuring throughput, time to first token, and inter-token latency at your target load. Retune when changing the workload, pool sizes, hardware, or engine version. The prefill example uses `--long-prefill-token-threshold 1024` and leaves `--prefill-schedule-interval` unset. **DEP NUMA binding:** on every prefill and decode node, enable binding and provide one NUMA node ID per visible GPU. For the four-GPU layout where GPUs 0/1 are local to NUMA node 0 and GPUs 2/3 to NUMA node 1, use: ```text --numa-bind --numa-bind-nodes 0 0 1 1 ``` Match the list to the local GPU order and topology on each node, including GB300 deployments. See the [vLLM NUMA binding guide](https://docs.vllm.ai/en/latest/configuration/optimization/#numa-binding-for-multi-socket-gpu-nodes) for automatic detection and explicit mappings. **Aggregate streaming:** add `--stream-interval 10` to the aggregate TP8 worker at concurrency 1, 4, or 8. This batches streamed output into larger updates to reduce host overhead; use a smaller interval when smoother token delivery matters more. The DEP configurations leave this override unset. ### Running with Dynamo The example below runs a DEP8 prefill pool and a DEP8 decode pool, each across two four-GPU GB200 nodes, plus a host for shared services and the frontend. Run the Python commands in `vllm/vllm-openai:nightly-dev-arm64-cu13.0.1-20ebb28` with the model, recipe files, GPUs, and RDMA devices accessible; use host networking so the advertised addresses are reachable across nodes. Install the matching Dynamo packages in each frontend/worker container: ```bash python3 -m pip install --extra-index-url https://pypi.nvidia.com \ ai-dynamo==1.3.0.dev20260720 ai-dynamo-runtime==1.3.0.dev20260720 ``` Save the following as `dynamo-env.sh` and source it in each shell. Replace the example addresses with routable addresses on your cluster. `MODEL_PATH` must contain the pinned 0813 checkpoint on every worker. ```bash export INFRA_IP=10.0.0.10 export ETCD_ENDPOINTS="http://${INFRA_IP}:2379" export NATS_SERVER="nats://${INFRA_IP}:4222" export DYN_NAMESPACE=dspark export MODEL_PATH=/models/DeepSeek-V4-Pro-0813 export SERVED_MODEL=deepseek-ai/DeepSeek-V4-Pro-0813 export PREFILL_HEAD_IP=10.0.0.11 export DECODE_HEAD_IP=10.0.0.21 ``` **Shared services.** Provide `etcd`, `nats-server`, and `mooncake_master` on the service host's `PATH`. See the official [Dynamo service setup](https://github.com/ai-dynamo/dynamo/blob/59dd504105db498b11e5715854a2512d57b7a1e0/dev/docker-compose.yml) and [Mooncake installation](https://kvcache-ai.github.io/Mooncake/) for installation options. Start these services once on the private cluster network, before workers: ```bash source ./dynamo-env.sh etcd --data-dir ./etcd-data --listen-client-urls http://0.0.0.0:2379 \ --advertise-client-urls "$ETCD_ENDPOINTS" >etcd.log 2>&1 & cat >nats.conf <<'EOF' port: 4222 max_payload: 33554432 jetstream { store_dir: "./nats-data" } EOF nats-server -c ./nats.conf >nats.log 2>&1 & mooncake_master --port=8700 --enable_http_metadata_server=true \ --http_metadata_server_port=8701 --eviction_high_watermark_ratio=0.9 \ --default_kv_lease_ttl=10000 --rpc_thread_num=16 \ --enable_metric_reporting=true --metrics_port=8702 >mooncake.log 2>&1 & ``` Wait for the services to listen before continuing. All hosts must reach the shared endpoints; workers must also reach each other's DP rendezvous and NIXL addresses. Existing managed services can supply the same endpoints. **Frontend.** In a separate shell, start the HTTP endpoint for the disaggregated deployment: ```bash source ./dynamo-env.sh export DYN_ROUTER_ACTIVE_REQUEST_EXPIRY_SECS=3600 export DYN_TCP_CHANNEL_BUFFER=128 export DYN_TCP_REQUEST_TIMEOUT=60 python3 -m dynamo.frontend --http-port 8000 --router-mode random \ --router-session-affinity-ttl-secs 900 ``` - Aggregate: `--router-mode kv`, `DYN_ROUTER_TEMPERATURE=0` and `DYN_TOKENIZER=fastokens`. - Disaggregated: `--router-mode random` with session affinity, or `kv` with the same tokenizer settings. Send a stable conversation ID as `x-dynamo-session-id` and set `--router-session-affinity-ttl-secs` to cover the conversation lifetime. - Both DEP roles use `MultiConnector` with `NixlConnector` for PD transfer and `MooncakeStoreConnector` for shared prefix storage. Enable store lookup only on prefill, and set `VLLM_CONNECTOR_PREFETCH_DEPTH=0` there. Role-specific `--kv-transfer-config` values: Prefill — save as `prefill-kv.json` in the worker's working directory: ```json {"kv_connector":"MultiConnector","kv_role":"kv_both","kv_connector_extra_config":{"connectors":[{"kv_connector":"NixlConnector","kv_role":"kv_both","kv_load_failure_policy":"fail","kv_buffer_device":"cuda","kv_connector_extra_config":{"enforce_handshake_compat":false,"enable_cross_layers_blocks":false}},{"kv_connector":"MooncakeStoreConnector","kv_role":"kv_both","kv_connector_extra_config":{"load_async":true,"lookup_async":true,"enable_cross_layers_blocks":false,"enable_offload":false}}]}} ``` Decode — save as `decode-kv.json` in the worker's working directory: ```json {"kv_connector":"MultiConnector","kv_role":"kv_consumer","kv_connector_extra_config":{"connectors":[{"kv_connector":"NixlConnector","kv_role":"kv_consumer","kv_load_failure_policy":"fail","kv_buffer_device":"cuda","kv_connector_extra_config":{"enforce_handshake_compat":false,"enable_cross_layers_blocks":false}},{"kv_connector":"MooncakeStoreConnector","kv_role":"kv_consumer","kv_connector_extra_config":{"load_async":true,"lookup_async":false,"enable_lookup":false,"enable_cross_layers_blocks":false,"enable_offload":false}}]}} ``` **Worker environment.** Run the following in Bash on each worker node. Set `NODE_IP` to that node's routable address and `NODE_RANK` to 0 or 1 within its own pool. For example, prefill nodes `10.0.0.11` / `10.0.0.12` use ranks 0 / 1; decode nodes `10.0.0.21` / `10.0.0.22` also use ranks 0 / 1. All four GPUs are visible to one launch process per node. Adjust the NUMA and NIC mappings to match the hardware. The embedded Mooncake store uses 140 GB per local rank in this example; ensure enough host memory or resize it for your deployment. ```bash source ./dynamo-env.sh export NODE_IP=10.0.0.11 # Change on each worker node. NODE_RANK=0 # Use 1 on the second node of either pool. export CUDA_VISIBLE_DEVICES=0,1,2,3 export VLLM_HOST_IP="$NODE_IP" export VLLM_NIXL_SIDE_CHANNEL_HOST="$NODE_IP" export VLLM_NIXL_SIDE_CHANNEL_PORT=20096 export DYN_SYSTEM_PORT=8081 export VLLM_USE_V2_MODEL_RUNNER=1 VLLM_DSV4_MEGA_FP8_COMBINE=1 export PYTHONHASHSEED=0 export MOONCAKE_MASTER="${INFRA_IP}:8700" export MOONCAKE_TE_META_DATA_SERVER="http://${INFRA_IP}:8701/metadata" export MOONCAKE_LOCAL_HOSTNAME="$NODE_IP" export MOONCAKE_CONFIG_PATH="$PWD/mooncake.json" cat >"$MOONCAKE_CONFIG_PATH" < \ vllm serve MiniMaxAI/MiniMax-M3.1-Flash \ -tp 4 \ --reasoning-parser minimax_m3 \ --block-size 128 \ --kv-cache-dtype nvfp4 \ --attention-config '{"indexer_kv_dtype": "nvfp4"}' \ --quantization-config.moe mxfp8 \ --speculative-config '{"method":"dspark","num_speculative_tokens":3,"attention_backend":"TRITON_ATTN","kv_cache_dtype":"fp8","draft_sample_method":"probabilistic"}' \ --disable-hybrid-kv-cache-manager \ --max-num-batched-tokens 32768 \ --port 8100 \ --kv-transfer-config '{"kv_connector":"MultiConnector","kv_role":"kv_both","kv_connector_extra_config":{"connectors":[ {"kv_connector":"NixlConnector","kv_role":"kv_producer","kv_connector_extra_config":{"num_threads":4}}, {"kv_connector":"SimpleCPUOffloadConnector","kv_role":"kv_both","kv_connector_extra_config":{"cpu_bytes_to_use_per_rank":601295421440,"cpu_hugepage_block_size":"1GB","lazy_offload":false}}]}}' ``` Decode node: ```bash VLLM_ROCM_USE_AITER=1 VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS=1 \ VLLM_KV_CACHE_LAYOUT=BLHNC VLLM_NIXL_SIDE_CHANNEL_HOST= \ vllm serve MiniMaxAI/MiniMax-M3.1-Flash \ -tp 4 \ --reasoning-parser minimax_m3 \ --block-size 128 \ --kv-cache-dtype nvfp4 \ --attention-config '{"indexer_kv_dtype": "nvfp4", "indexer_decode_cp": true}' \ --quantization-config.moe mxfp8 \ --speculative-config '{"method":"dspark","num_speculative_tokens":3,"attention_backend":"TRITON_ATTN","kv_cache_dtype":"fp8","draft_sample_method":"probabilistic"}' \ --disable-hybrid-kv-cache-manager \ --port 8200 \ --kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_consumer","kv_connector_extra_config":{"num_threads":4}}' ``` - `` and `` are each node's own address, reachable from the other node (the NIXL side channel). - `VLLM_KV_CACHE_LAYOUT=BLHNC` goes on both nodes. It keeps all layers of a block contiguous, so NIXL reads one region per block instead of one per layer. - The prefill node never decodes, so it leaves `indexer_decode_cp` off. - The prefill node also offloads KV to host memory (`SimpleCPUOffloadConnector`): 560 GiB per rank in 1 GiB HugeTLB pages, so 2240 pages for the four ranks. Reserve them on the host before the start (and allow them in the container's hugetlb limit), or the server fails at startup with `Cannot map 560.00 GiB of 1024 MiB HugeTLB pages`: ```bash echo 2240 | sudo tee /sys/kernel/mm/hugepages/hugepages-1048576kB/nr_hugepages ``` ### Prefill/decode disaggregation (1P1D, DEP2 + DEP4, NVIDIA GB200 / GB300) A DEP2 prefill on one 4-GPU tray feeds a DEP4 decode on a second tray over NIXL. Both roles run attention data parallel with the NVFP4 experts on the DeepGEMM MegaMoE kernel over expert parallelism, the V2 model runner, NUMA binding, and a seven-token DSpark draft on the FlashInfer PrimiTS (`FLASHINFER_TS`) backend. The settings were tuned on GB300; GB200 uses the same commands. Prefill node: ```bash VLLM_FLOAT32_MATMUL_PRECISION=high VLLM_USE_V2_MODEL_RUNNER=1 \ VLLM_ENGINE_READY_TIMEOUT_S=3600 VLLM_NIXL_SIDE_CHANNEL_HOST= \ vllm serve MiniMaxAI/MiniMax-M3.1-Flash \ --data-parallel-size 2 \ --enable-expert-parallel \ --moe-backend deep_gemm_mega_moe \ --numa-bind \ --reasoning-parser minimax_m3 \ --block-size 128 \ --kv-cache-dtype nvfp4 \ --attention-config '{"indexer_kv_dtype": "nvfp4", "use_prefill_query_quantization": true}' \ --compilation-config '{"cudagraph_mode": "FULL_AND_PIECEWISE"}' \ --speculative-config '{"method":"dspark","num_speculative_tokens":7,"attention_backend":"FLASHINFER_TS","kv_cache_dtype":"fp8","draft_sample_method":"probabilistic","rejection_sample_method":"block"}' \ --max-num-seqs 64 \ --max-num-batched-tokens 32768 \ --long-prefill-token-threshold 8192 \ --gpu-memory-utilization 0.9 \ --enable-prefix-caching \ --disable-hybrid-kv-cache-manager \ --port 8100 \ --kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_producer","kv_load_failure_policy":"fail","kv_connector_extra_config":{"engine_ttl":0}}' ``` Decode node: ```bash VLLM_FLOAT32_MATMUL_PRECISION=high VLLM_USE_V2_MODEL_RUNNER=1 \ VLLM_ENGINE_READY_TIMEOUT_S=3600 VLLM_NIXL_SIDE_CHANNEL_HOST= \ vllm serve MiniMaxAI/MiniMax-M3.1-Flash \ --data-parallel-size 4 \ --enable-expert-parallel \ --moe-backend deep_gemm_mega_moe \ --numa-bind \ --reasoning-parser minimax_m3 \ --block-size 128 \ --kv-cache-dtype nvfp4 \ --attention-config '{"indexer_kv_dtype": "nvfp4", "use_prefill_query_quantization": true}' \ --compilation-config '{"cudagraph_mode": "FULL_DECODE_ONLY", "max_cudagraph_capture_size": 224}' \ --speculative-config '{"method":"dspark","num_speculative_tokens":7,"attention_backend":"FLASHINFER_TS","kv_cache_dtype":"fp8","draft_sample_method":"probabilistic","rejection_sample_method":"block"}' \ --max-num-seqs 28 \ --max-num-batched-tokens 8192 \ --gpu-memory-utilization 0.95 \ --watermark 0.05 \ --no-enable-prefix-caching \ --disable-hybrid-kv-cache-manager \ --port 8200 \ --kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_consumer","kv_load_failure_policy":"fail","kv_connector_extra_config":{"engine_ttl":0}}' ``` - `VLLM_ENGINE_READY_TIMEOUT_S=3600`: with data parallel, each API server waits this long for its engines to start (default 600 s). A cold prefill start (JIT builds, CUDA graph capture, FlashInfer autotune) takes longer than 10 minutes. - `--max-num-seqs` is per DP rank. The decode CUDA graphs cover 28 requests of 1 + 7 tokens (224). - `--watermark 0.05` keeps 5% of the decode KV cache free when admitting requests, so extra requests wait in the queue instead of preempting running ones. On GB300 (about 4.5M KV tokens per decode rank) at a 200K-token mean context, expect about 19-21 running requests per rank. - Keep the decode `--max-num-batched-tokens` at 8192: a preempted request recomputes its context in chunks of that size, and every DP rank runs eager steps until it finishes. - `engine_ttl: 0` turns off NIXL's cleanup of idle remote engines; with the default one-hour TTL, a decode crashed an hour after its prefill went away. - Prefix caching stays on the prefill node only. For multi-turn traffic, the prefill node can also offload KV to host memory, 180 GiB per rank: `--kv-transfer-config '{"kv_connector":"MultiConnector","kv_role":"kv_both","kv_load_failure_policy":"fail","kv_connector_extra_config":{"connectors":[{"kv_connector":"NixlConnector","kv_role":"kv_producer","kv_connector_extra_config":{"engine_ttl":0}},{"kv_connector":"SimpleCPUOffloadConnector","kv_role":"kv_both","kv_connector_extra_config":{"cpu_bytes_to_use_per_rank":193273528320,"lazy_offload":true}}]}}'`. For a quick test, put the toy NIXL proxy from the vLLM repository in front of the two nodes and send requests to port 8000: ```bash python tests/v1/kv_connector/nixl_integration/toy_proxy_server.py --port 8000 \ --prefiller-host --prefiller-port 8100 \ --decoder-host --decoder-port 8200 ``` ## Client Usage Recommended sampling parameters (from the model card): `temperature = 1.0`, `top_p = 0.95`. Set the reasoning effort per request with `chat_template_kwargs: {"reasoning_effort": ""}`: `low`, `medium`, `high`, `xhigh` or `max`. ```bash curl http://localhost:8000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "MiniMaxAI/MiniMax-M3.1-Flash", "temperature": 1.0, "top_p": 0.95, "messages": [ {"role": "user", "content": "Explain MSA sparse attention in 3 bullets."} ] }' ``` ## Benchmarking ```bash vllm bench serve \ --backend vllm \ --model MiniMaxAI/MiniMax-M3.1-Flash \ --endpoint /v1/completions \ --dataset-name random \ --random-input 8192 \ --random-output 1024 \ --max-concurrency 32 \ --num-prompts 320 ``` ## Troubleshooting - **`--block-size` mismatch.** MSA's sparse block size is 128; the vLLM KV cache block size must match (otherwise `No common block size for 16`). - **Parsers.** `--tool-call-parser` and `--reasoning-parser` both use `minimax_m3`. - **HugeTLB pages (PD prefill node).** See the AMD PD section above. ## References - [Model card](https://huggingface.co/MiniMaxAI/MiniMax-M3.1-Flash) - [MiniMax-M3 recipe](https://recipes.vllm.ai/MiniMaxAI/MiniMax-M3) - [MiniMax](https://www.minimax.io/) --- # Qwen/Qwen3.8-Flash-Next on vLLM Source: https://recipes.vllm.ai/Qwen/Qwen3.8-Flash-Next Provider: Qwen Released: 2026-08-24T08:24:59.000Z Updated: 2026-10-27 Architecture: moe Parameters: 176B Context length: 262,144 Min vLLM: 0.30.0 Tasks: multimodal, text Verified hardware: H200, H100, GB200 NVL4, GB300 NVL4, MI355X, RTX Pro 6000 4x, DGX Station (GB300) Summary: Qwen4 architecture preview with a 125B-parameter main model, supplemented by an additional 51B N-gram embeddings, with 6B parameters activated per token. ## Variants - default: precision=nvfp4, vram_minimum_gb=130, model_id=Inferact/Qwen3.8-Flash-Next-NVFP4 Inferact quantized NVFP4 weights with per-layer embeddings in nvfp4 too - fp8: precision=fp8, vram_minimum_gb=250, model_id=Qwen/Qwen3.8-Flash-Next-FP8 Official dynamic FP8 checkpoint with blockwise weight quantization - bf16: precision=bf16, vram_minimum_gb=423, model_id=Qwen/Qwen3.8-Flash-Next Official BF16 checkpoint - mxfp4: precision=mxfp4, vram_minimum_gb=219, model_id=amd/Qwen3.8-Flash-Next-Quark-MXFP4 AMD Quark MXFP4 checkpoint with MXFP4 routed experts, for MI355X ## Recommended base flags ```bash vllm serve 'Qwen/Qwen3.8-Flash-Next' \ --max-num-seqs \ 256 \ --gpu-memory-utilization \ 0.90 \ --enable-prefix-caching ``` ## Guide ## Overview [Qwen3.8-Flash-Next](https://huggingface.co/Qwen/Qwen3.8-Flash-Next) is a multimodal, ultra-sparse Mixture-of-Experts model. It has 125B parameters, including an additional 51B N-gram embedding table, while activating 6B parameters per token. The architecture combines four main ideas: - **GDN + QSA:** three of every four layers use Gated DeltaNet to compress history; the fourth uses Qwen Sparse Attention for precise long-range retrieval. - **Gated Residual:** four residual branches dynamically control cross-layer reads and writes. - **N-gram Embedding:** a 51B lookup memory adds capacity with little per-token compute and can be asynchronously offloaded to host memory. (note that offload currently only runs on Nvidia devices) - **MTP:** the built-in Multi-Token Prediction module supports speculative decoding. The checkpoint natively supports 262,144 tokens. Qwen reports that QSA reaches up to 10.2x prefill and 6.6x decode attention-kernel speedups at one million tokens. On GB300, TEP4 is the recommended full-tray configuration. On an 8x H200 node, use TEP8 for the official FP8 checkpoint; plain TP8 is incompatible with its 128-wide quantization blocks. ## Prerequisites - **Runtime:** use the nightly image, `vllm/vllm-openai:nightly` on NVIDIA or `vllm/vllm-openai-rocm:nightly` on AMD, or the nightly PyPI wheels. Stable ≥0.30.0 runs the model, but this recipe is tuned and verified on nightly. - **Hardware:** NVIDIA CUDA and AMD ROCm instructions are provided below. - **FP8 memory:** the checkpoint is 172.78 GiB; it fits on a single GB300 GPU (288 GB HBM3e). - **BF16 memory:** the checkpoint is 335.28 GiB. The validated TP2 configuration used about 190 GiB per GB300 GPU. - **MXFP4 memory:** the AMD Quark MXFP4 checkpoint is 169.8 GiB; it fits on a single MI355X GPU (288 GB HBM3E). - **Host memory for N-gram Embedding offload:** at least 51 GB plus runtime headroom. The Install block above generates the nightly-wheel and Docker launch flows. ## Serving ### Recommended FP8 configuration on 4x GB300 Use the official [`Qwen/Qwen3.8-Flash-Next-FP8`](https://huggingface.co/Qwen/Qwen3.8-Flash-Next-FP8) checkpoint tested by this recipe. ```bash vllm serve Qwen/Qwen3.8-Flash-Next-FP8 \ --tensor-parallel-size 4 \ --enable-expert-parallel \ --gpu-memory-utilization 0.90 \ --max-num-seqs 256 \ --enable-prefix-caching \ --kv-cache-dtype fp8 \ --attention-config.indexer_kv_dtype fp8 \ --speculative-config '{"method":"mtp","num_speculative_tokens":3}' \ --enable-auto-tool-choice \ --tool-call-parser qwen3_coder \ --reasoning-parser qwen3 ``` TEP4 is the recommended full-tray configuration — we recommend it over plain TP4 because expert parallelism is consistently faster for this model. TP4 and TEP4 are both validated full-tray configurations, including with MTP3. To serve BF16, use the `Qwen/Qwen3.8-Flash-Next` checkpoint with the same arguments. ### Single-GPU deployment on GB200/GB300 The model fits on a single GPU: the FP8 checkpoint on one GB300 (288 GB HBM3e), and the NVFP4 variant on GB200 as well. Single-GPU TP1 suits evaluation and development; multi-GPU TEP remains the recommended deployment for serving throughput. ### Recommended FP8 configuration on 8x H200 Use TEP8 with the Triton MoE backend on Hopper: ```bash vllm serve Qwen/Qwen3.8-Flash-Next-FP8 \ --tensor-parallel-size 8 \ --enable-expert-parallel \ --moe-backend triton \ --gpu-memory-utilization 0.85 \ --max-num-seqs 256 \ --enable-prefix-caching \ --kv-cache-dtype fp8 \ --attention-config.indexer_kv_dtype fp8 \ --speculative-config '{"method":"mtp","num_speculative_tokens":3}' \ --enable-auto-tool-choice \ --tool-call-parser qwen3_coder \ --reasoning-parser qwen3 ``` Plain TP8 is incompatible with the FP8 checkpoint; use TEP8. ### Recommended FP8 configuration on 8x H100 H100's 80GB/GPU is not enough headroom for the 51B N-gram/PLE embedding table — it relies on PLE CPU offload, which vLLM enables by default. Use TEP8 with the Triton MoE backend; as on other hardware, expert parallelism is faster than plain TP, and plain TP8 is incompatible with the FP8 checkpoint's 128-wide quantization blocks. ```bash vllm serve Qwen/Qwen3.8-Flash-Next-FP8 \ --tensor-parallel-size 8 \ --enable-expert-parallel \ --moe-backend triton \ --gpu-memory-utilization 0.85 \ --max-num-seqs 256 \ --enable-prefix-caching \ --kv-cache-dtype fp8 \ --attention-config.indexer_kv_dtype fp8 \ --speculative-config '{"method":"mtp","num_speculative_tokens":3}' \ --enable-auto-tool-choice \ --tool-call-parser qwen3_coder \ --reasoning-parser qwen3 ``` Benchmark MTP with SPEED-Bench (`--dataset-name speed_bench`), not `random`. ### MXFP4 on MI355X On MI355X, the MXFP4 variant serves AMD's [`amd/Qwen3.8-Flash-Next-Quark-MXFP4`](https://huggingface.co/amd/Qwen3.8-Flash-Next-Quark-MXFP4) checkpoint, which quantizes the routed experts to MXFP4 with AMD Quark. The variant sets: - `VLLM_ROCM_USE_AITER=1`, which turns on the AITER kernels. - `VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS=1`, which fuses the shared expert into the routed-expert MoE kernel. - `AITER_USE_FLYDSL_MOE_SORTING=1`, which sorts MoE tokens with a FlyDSL kernel. - `--gdn-prefill-backend aiter_flydsl`, which runs Gated DeltaNet prefill with a FlyDSL kernel. The default deployment is TEP8, and the TP strategy runs TP2. The checkpoint also fits on a single MI355X (288 GB HBM3E), so TP1 with the same flags serves one replica per GPU; the builder offers one TP size per variant, so set `--tensor-parallel-size 1` yourself. TP2 leaves more memory for KV cache and gives slightly lower per-token latency; TP1 gives more throughput per GPU. Validated on `vllm/vllm-openai-rocm:nightly-83c078d955cd6e0c63d3b12abe669c8de936a370` with the builder's TEP8 and TP2 commands, and TP1 as the TP2 command with `--tensor-parallel-size 1`. GSM8K is 5-shot over 1,319 questions through chat completions with thinking off; throughput uses random 8k-token prompts and 1k-token outputs. | MXFP4 on MI355X | GSM8K | Output tok/s, concurrency 8 / 32 | | --- | --- | --- | | TEP8 | 0.965 | 665 / 1831 | | TP2 | 0.967 | 801 / 1384 | | TP1 | 0.967 | 751 / 1282 | ### FP8 configuration on a 4x MI355X GPU ```bash export VLLM_ROCM_USE_AITER=1 export VLLM_ROCM_USE_AITER_MOE=0 vllm serve Qwen/Qwen3.8-Flash-Next-FP8 \ --tensor-parallel-size 4 \ --max-model-len auto \ --gpu-memory-utilization 0.9 ``` MTP is opt-in on MI355X; to turn it on, add the option from the MTP speculative decoding section below. ### Full native context The checkpoint advertises a native 262,144-token context, which vLLM uses when `--max-model-len` is omitted. Startup and bounded evaluation were validated with this limit configured, but a single 262K-token request was not tested. ### MTP speculative decoding Add the following option: ```bash --speculative-config '{"method":"mtp","num_speculative_tokens":3}' ``` ### Offload the 51B N-gram embedding table Builds containing N-gram Embedding offload can keep the N-gram lookup memory in host RAM and asynchronously prefetch the required rows. The initial implementation supports both ModelRunner V1 and V2. PLE CPU offload is required for DEP, and also required for TP/TEP on 80GB-class GPUs (H100) where the embedding table alone exceeds available headroom — optional only on GPUs with more VRAM per rank (H200 and above). vLLM enables it by default; set it explicitly with `--engram-config '{"cpu_offload":true}'`. ```bash vllm serve Qwen/Qwen3.8-Flash-Next-FP8 \ --tensor-parallel-size 4 \ --enable-expert-parallel \ --max-model-len 262144 \ --enable-auto-tool-choice \ --tool-call-parser qwen3_coder \ --reasoning-parser qwen3 ``` ### Extend to one million tokens with YaRN The checkpoint is native at 262K. For a one-million-token workload, turn on the **Long Context** feature above: it applies the model card's YaRN RoPE configuration under `text_config` (where this checkpoint carries `rope_parameters`) and raises `--max-model-len` to 1,000,000. Evaluate shorter-context quality before making it the default — static YaRN holds the scaling factor constant regardless of input length. ## Verifying ```python from openai import OpenAI client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1") response = client.chat.completions.create( model="Qwen/Qwen3.8-Flash-Next-FP8", messages=[{ "role": "user", "content": "Explain how Gated DeltaNet and Qwen Sparse Attention complement each other.", }], max_tokens=8196, ) print(response.choices[0].message.content) ``` A correct response should explain that GDN maintains a compact recurrent summary, while QSA selectively retrieves important regions from the full token history. ## Troubleshooting - **FP8 on eight GPUs:** use TEP8; plain TP8 is incompatible with this checkpoint. TP4 is also supported if GPU memory capacity allows. - **Mamba-cache capacity error at startup:** keep `--max-num-seqs 256`. - **Runtime OOM on large multimodal batches:** keep `--gpu-memory-utilization 0.90`. - **Out of memory while loading:** enable N-gram Embedding CPU offload, increase TP size, or reduce `--max-model-len` to reserve less KV cache. On 4x H100 80GB, plain TP4 OOMs at engine startup without PLE CPU offload — keep the default `--engram-config '{"cpu_offload":true}'`. - **XPU/TPU startup error:** these platforms are not supported by the initial implementation. - **Pipeline-parallel startup error:** N-gram Embedding does not initially support pipeline parallelism; use single-node TP or TEP instead. - **DEP startup failure:** DEP requires PLE CPU offload (on by default); DEP without it is not supported. - **MTP memory pressure:** reduce `num_speculative_tokens` below 3. - **PLE:** a network layer that injects N-gram Embeddings into the main model. ## KV cache offloading Offloading is available upstream since vLLM v0.30.0 ([vllm-project/vllm#57145](https://github.com/vllm-project/vllm/pull/57145)). Validation used [Qwen/Qwen3.8-Flash-Next-FP8](https://huggingface.co/Qwen/Qwen3.8-Flash-Next-FP8) on 4×H200 (TP4), vLLM commit `d1b4028d7e`, the Python frontend, `--enforce-eager`, `--max-model-len 16384`, `--max-num-seqs 16` and `--max-num-batched-tokens 8192`. Qwen validation used the FP8 variant with `--limit-mm-per-prompt '{"image":0,"video":0}'`. Validation used `TieringOffloadingSpec`: 8 GiB total CPU tier, `blocks_per_chunk: 1`, a persistent filesystem `root_dir`, eight read/write threads and `PYTHONHASHSEED=0`. Three CPU-tier reloads and three filesystem reloads after restarting the server passed. Reloaded responses matched the cold output, including the reasoning field. The five-shot GSM8K diagnostic scored 31/32; this is a small sample, not the full benchmark. These results cover this text-only configuration, not the builder's larger tier/chunk defaults or other variants. ## References - [Qwen3.8-Flash-Next model card](https://huggingface.co/Qwen/Qwen3.8-Flash-Next) - [Qwen3.8-Flash-Next FP8 checkpoint](https://huggingface.co/Qwen/Qwen3.8-Flash-Next-FP8) - [Qwen3.8-Flash-Next: A New Architecture, Towards Ultimate Cost-Efficiency](https://qwen.ai/blog?id=qwen3.8-flash-next) - [Qwen3-Next recipe](../Qwen3-Next-80B-A3B-Instruct) --- # inclusionAI/Ming-3.0-asr-tiny on vLLM Source: https://recipes.vllm.ai/inclusionAI/Ming-3.0-asr-tiny Provider: inclusionAI Released: 2026-10-14T09:27:09.000Z Updated: 2026-10-27 Architecture: moe Parameters: 8.55B Context length: 8,192 Min vLLM: nightly Tasks: multimodal Summary: Chinese, English, and dialect speech recognition with topic, hotword, and dialogue context. ## Variants - default: precision=bf16, vram_minimum_gb=22 BF16 checkpoint; the capacity estimate includes the audio tower and is not a measured minimum. ## Recommended base flags ```bash vllm serve 'inclusionAI/Ming-3.0-asr-tiny' \ --trust-remote-code ``` ## Guide ## Overview [Ming-3.0-asr-tiny](https://huggingface.co/inclusionAI/Ming-3.0-asr-tiny) supports Chinese, English, and dialect transcription with optional topic, hotword, and dialogue context. This guide covers single-GPU serving, transcription, streaming, Context ASR, and offline inference with vLLM. ## Public example audio Download the three clips from the official inference repository: ```bash MING_AUDIO_BASE=https://raw.githubusercontent.com/inclusionAI/Ming-3.0-asr/6141e8c552815db724d8df0f72c9d865c547a1ea/examples/cases/audio curl -fL "$MING_AUDIO_BASE/english_common_voice.wav" -o english.wav curl -fL "$MING_AUDIO_BASE/chinese_common_voice.wav" -o chinese.wav curl -fL "$MING_AUDIO_BASE/context_same_sentence_pcm16.wav" -o context.wav ``` | Clip | Observed transcription | | --- | --- | | English | `same way you did` | | Mandarin | `漓江和湘江有秦代开凿的灵渠沟通` | | Context clip with the hints below | Contains the film title `同囚` | The Context clip illustrates correction of `铜球` to `同囚` with a relevant topic and hotwords. These are sample-specific observations, not a general Context ASR accuracy claim. Audio provenance and licenses are documented in the [official examples](https://github.com/inclusionAI/Ming-3.0-asr/blob/6141e8c552815db724d8df0f72c9d865c547a1ea/examples/cases/README.md). For accuracy comparisons, keep batch settings and attention backends consistent; greedy transcription outputs can differ across these configurations. ## Transcription API Use the audio requests below to verify speech recognition. The catalog's generic text-only cURL and benchmark shortcuts do not exercise ASR. ```bash curl http://localhost:8000/v1/audio/transcriptions \ -F model=inclusionAI/Ming-3.0-asr-tiny \ -F file=@chinese.wav \ -F temperature=0 \ -F max_completion_tokens=256 \ -F 'vllm_xargs={"no_repeat_ngram_size":10}' ``` Omit `language` for automatic recognition, or use `language=zh` / `language=en` to seed the known language. These are the supported language-hint codes; dialect transcription does not introduce additional hint codes. The response contains the transcript without the generated language label. `response_format=json` and `response_format=text` are supported. ### OpenAI Python client ```python import json from openai import OpenAI client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY") with open("english.wav", "rb") as audio: result = client.audio.transcriptions.create( model="inclusionAI/Ming-3.0-asr-tiny", file=audio, temperature=0, extra_body={ "max_completion_tokens": 256, "vllm_xargs": json.dumps({"no_repeat_ngram_size": 10}), }, ) print(result.text) ``` `vllm_xargs` is JSON-encoded for the multipart request, as in the cURL examples. ### Context ASR ```bash curl http://localhost:8000/v1/audio/transcriptions \ -F model=inclusionAI/Ming-3.0-asr-tiny \ -F file=@context.wav \ -F language=zh \ -F temperature=0 \ -F max_completion_tokens=256 \ -F 'prompt=This is an audio about 同囚.' \ -F 'hotwords=同囚,黄国权,游学修,关楚耀,赵永洪,赵咏瑶,狱警,长镜头,吞玻璃戏,糖化玻璃,真实案例数据.' \ -F 'vllm_xargs={"no_repeat_ngram_size":10,"context_chunks":true}' ``` `prompt` accepts topic or preceding dialogue appended to the official ASR instruction, or the full official instruction followed by context. Context and hotwords must be single-line text without audio placeholders. To reproduce the vendor's full prompt exactly, put its hotword text in `prompt` and omit the separate `hotwords` field. ### Streaming ```bash curl -N http://localhost:8000/v1/audio/transcriptions \ -F model=inclusionAI/Ming-3.0-asr-tiny \ -F file=@english.wav \ -F language=en \ -F temperature=0 \ -F stream=true \ -F max_completion_tokens=256 \ -F 'vllm_xargs={"no_repeat_ngram_size":10}' ``` Streaming sends transcript deltas after the complete audio upload. It is not live microphone input. The English clip produces `same way you did` without leaking the language prefix into the transcript. ## Chunking and repetition control Audio is downmixed and resampled to 16 kHz using torchaudio. The processor extracts the entire clip's log-mel features before splitting encoder chunks. Chunk settings are request-local and are included in multimodal cache keys. | Option in `vllm_xargs` | Behavior | | --- | --- | | No chunk option | Use 3,000-mel-frame encoder chunks, approximately 30 seconds each. | | `"context_chunks": true` | Use 3,000 frames up to 30 seconds and 6,000 frames for 30–60 seconds; reject longer clips. | | `"audio_chunk_frames": 6000` | Explicit chunk size; any integer from 1 through 6,000 is accepted. | | `"no_repeat_ngram_size": 10` | Block repeated 10-grams over prompt plus generated tokens. | `context_chunks` and an explicit `audio_chunk_frames` are mutually exclusive. Ordinary chunking can handle clips beyond 60 seconds within the shared 8,192-token input/output budget and audio-upload size limit. Internal encoder chunking does not create separate transcription requests. The no-repeat feature loads a plugin; each request must still set a nonzero `no_repeat_ngram_size` to enable blocking. Omitted values and `0` leave ordinary sampling in effect. Blocking uses the full prompt/output history, and requires Model Runner V2. Speculative decoding and MTP are unsupported with this plugin. ## Offline inference From the same vLLM source checkout, use the shared [audio language model example](https://github.com/vllm-project/vllm/blob/main/examples/generate/multimodal/audio_language_offline.py) with its built-in audio assets: ```bash python examples/generate/multimodal/audio_language_offline.py --model-type ming_v3_asr ``` This uses the shared example's sampling defaults (`temperature=0.2`, `max_tokens=64`) and prints raw model output, including the language label. For the downloaded Context clip, use the native prompt builder with greedy decoding and 10-gram blocking: ```python from vllm import LLM, SamplingParams from vllm.config.speech_to_text import SpeechToTextParams from vllm.model_executor.models.ming_v3_asr import ( MingV3ASRForConditionalGeneration as MingASR, ) from vllm.multimodal.audio import AudioResampler from vllm.multimodal.media.audio import load_audio def main(): llm = LLM( model="inclusionAI/Ming-3.0-asr-tiny", trust_remote_code=True, dtype="bfloat16", max_model_len=8192, max_num_seqs=4, enforce_eager=True, logits_processors=[ "vllm.model_executor.models.ming_v3_asr_logits_processor:" "NoRepeatNGramLogitsProcessor", ], ) audio, rate = load_audio("context.wav", sr=None, mono=True) audio = AudioResampler(16000).resample(audio, orig_sr=rate) params = SpeechToTextParams( audio=audio, stt_config=MingASR.get_speech_to_text_config(llm.model_config, "transcribe"), model_config=llm.model_config, language="zh", request_prompt="This is an audio about 同囚.", hotwords=( "同囚,黄国权,游学修,关楚耀,赵永洪,赵咏瑶,狱警,长镜头," "吞玻璃戏,糖化玻璃,真实案例数据." ), extra_args={"context_chunks": True}, ) outputs = llm.generate( MingASR.get_generation_prompt(params), SamplingParams( temperature=0, max_tokens=256, extra_args={"no_repeat_ngram_size": 10}, ), ) print(MingASR.post_process_output(outputs[0].outputs[0].text)) if __name__ == "__main__": main() ``` ## Limitations The [official model card](https://huggingface.co/inclusionAI/Ming-3.0-asr-tiny#usage-notes) describes transcription text output without word-level timestamps or speaker diarization. The current vLLM integration does not support translation, segment timestamps, or live audio input. Translation and timestamp requests are rejected. The `context_chunks=true` policy accepts clips up to 60 seconds; ordinary chunking, including requests with context prompts, can accept longer clips subject to the model context and upload limits. Longer recordings can produce incomplete transcripts; accepting an upload does not guarantee complete transcription. ## References - [model card](https://huggingface.co/inclusionAI/Ming-3.0-asr-tiny) - [Official inference repository](https://github.com/inclusionAI/Ming-3.0-asr) - [vLLM speech-to-text API](https://docs.vllm.ai/en/latest/serving/online_serving/speech_to_text.html) --- # MiniMaxAI/MiniMax-M3 on vLLM Source: https://recipes.vllm.ai/MiniMaxAI/MiniMax-M3 Provider: MiniMax Released: 2026-06-02T07:49:31.000Z Updated: 2026-10-26 Architecture: moe Parameters: 427B Context length: 1,048,576 Min vLLM: 0.24.0 Tasks: text, multimodal Verified hardware: H200, B200, B300, GB200 NVL4, MI300X, MI355X Summary: MiniMax M3 vision-language MoE (427B total / 26B active) for frontier coding, agent toolchains, and 1M-token reasoning via MSA sparse attention — native multimodal (image + video + computer use); BF16 plus MXFP8, NVIDIA Blackwell NVFP4, and AMD MI355X MXFP4 variants. Runs on NVIDIA (Hopper/Blackwell) and AMD CDNA4/CDNA3. ## Variants - default: precision=bf16, vram_minimum_gb=1025 Native BF16 checkpoint — multi-node TP recommended for long-context headroom - mxfp8: precision=mxfp8, vram_minimum_gb=513, model_id=MiniMaxAI/MiniMax-M3-MXFP8 NVIDIA-quantized MXFP8 weights — Blackwell (B200/B300) for native MX tensor cores, and AMD CDNA4 (MI350X/MI355X, gfx950) for native MXFP8 Matrix Cores. - nvfp4: precision=nvfp4, vram_minimum_gb=257, model_id=nvidia/MiniMax-M3-NVFP4 NVIDIA-quantized NVFP4 weights (ModelOpt) for Blackwell (B200/B300/GB200) — native FP4 tensor cores, ~half the VRAM of MXFP8. - mxfp4: precision=mxfp4, vram_minimum_gb=257, model_id=amd/MiniMax-M3-MXFP4 AMD Quark OCP MXFP4 weights and dynamic MXFP4 activations for MI355X (gfx950). ## Recommended base flags ```bash vllm serve 'MiniMaxAI/MiniMax-M3' \ --block-size \ 128 ``` ## Guide ## Overview [MiniMax-M3](https://huggingface.co/MiniMaxAI/MiniMax-M3) is a frontier vision-language MoE model from [MiniMax](https://www.minimax.io/). - **MSA (MiniMax Sparse Attention)** — scalable sparse-attention architecture that lifts the context window to **1M tokens**. MiniMax reports per-token compute at 1M context reduced to ~1/20 of the previous generation, with >9× prefill and >15× decode speedup vs dense baselines. - **Frontier coding and agent capabilities** — SWE-Bench Pro 59.0%, Terminal-Bench 2.1 66.0%, SWE-fficiency 34.8%, KernelBench Hard 28.8%, MCP Atlas 74.2%. - **Native multimodal** — image + video inputs, plus computer-use; trained multimodally from step 0. - **Two reasoning modes** — `thinking` (complex reasoning / agents) and `non-thinking` (latency-sensitive), switchable per request. ## Prerequisites - **OS:** Linux - **Python:** 3.10 - 3.13 - **NVIDIA:** compute capability >= 9.0 (Hopper) recommended; 8x H200 / H20 for a tight single-node BF16 fit, or multi-node TP for long-context headroom - **AMD:** MI350X/MI355X (gfx950), MI300X/MI325X (gfx942), ROCm 7.2+. BF16 needs TP=8; the MXFP8 variant runs from TP=4. - **`--block-size 128` is mandatory on every platform** (MSA sparse/index cache). ### Docker (NVIDIA) MiniMax-M3 support has not yet shipped in a stable vLLM release — use the dedicated Docker image: ```bash docker pull vllm/vllm-openai:minimax-m3 ``` ### Docker (AMD ROCm) Use the nightly image, which is the image the AMD variants of this recipe pin: ```bash docker pull vllm/vllm-openai-rocm:nightly ``` ```bash docker run --rm -it --device /dev/kfd --device /dev/dri --group-add video \ --cap-add SYS_PTRACE --security-opt seccomp=unconfined --ipc=host \ --shm-size=16g -p 8000:8000 \ --entrypoint /bin/bash \ vllm/vllm-openai-rocm:nightly ``` ## Launching the Server ### NVIDIA — TP8 (8x H200 / H20) ```bash vllm serve MiniMaxAI/MiniMax-M3 \ --tensor-parallel-size 8 \ --block-size 128 \ --tool-call-parser minimax_m3 \ --reasoning-parser minimax_m3 \ --enable-auto-tool-choice ``` #### TP8 + Expert Parallel ```bash vllm serve MiniMaxAI/MiniMax-M3 \ --tensor-parallel-size 8 \ --enable-expert-parallel \ --block-size 128 \ --tool-call-parser minimax_m3 \ --reasoning-parser minimax_m3 \ --enable-auto-tool-choice ``` #### DP8 + Expert Parallel ```bash vllm serve MiniMaxAI/MiniMax-M3 \ --data-parallel-size 8 \ --enable-expert-parallel \ --block-size 128 \ --tool-call-parser minimax_m3 \ --reasoning-parser minimax_m3 \ --enable-auto-tool-choice ``` ### AMD ROCm (MI350X/MI355X (gfx950), MI300X/MI325X (gfx942)) On AMD MI300X / MI325X / MI355X, run with CUDA graphs and set the following before any of the serve commands below. It avoids the MiniMax-M3 decode breakable-cudagraph path that would otherwise force eager execution (per @hongxiayang): ```bash export VLLM_USE_BREAKABLE_CUDAGRAPH=0 ``` For gfx950: Prefer using the MXFP8 variant `MiniMaxAI/MiniMax-M3-MXFP8` for TP=4 and a smaller footprint. Use TP=8 for lower latency or long context length, or the default bf16 model. #### TP8 (Text or Vision) ```bash vllm serve MiniMaxAI/MiniMax-M3 \ --tensor-parallel-size 8 \ --block-size 128 \ --attention-backend TRITON_ATTN \ --mm-encoder-tp-mode data \ --mm-encoder-attn-backend ROCM_AITER_FA \ --tool-call-parser minimax_m3 \ --reasoning-parser minimax_m3 \ --enable-auto-tool-choice ``` Add the vision-encoder flags (`--mm-encoder-tp-mode data --mm-encoder-attn-backend ROCM_AITER_FA`) for multimodal serving. #### TP8 + `--quantization fp8_per_channel` Online **per-channel FP8 (PTPC)** quantization of the BF16 checkpoint halves the weight footprint (≈100 → **≈50 GiB/GPU**, ~1.75× more KV cache) and lifts batched throughput with **gsm8k unchanged from BF16** (lossless); These env vars and the flag are for the **BF16 checkpoint only**. ```bash export VLLM_USE_BREAKABLE_CUDAGRAPH=0 export VLLM_ROCM_USE_AITER=1 export VLLM_ROCM_USE_AITER_MOE=0 vllm serve MiniMaxAI/MiniMax-M3 \ --tensor-parallel-size 8 \ --block-size 128 \ --attention-backend TRITON_ATTN \ --quantization fp8_per_channel \ --tool-call-parser minimax_m3 \ --reasoning-parser minimax_m3 \ --enable-auto-tool-choice ``` #### TP8 + Expert Parallel ```bash vllm serve MiniMaxAI/MiniMax-M3 \ --tensor-parallel-size 8 \ --enable-expert-parallel \ --block-size 128 \ --attention-backend TRITON_ATTN \ --mm-encoder-tp-mode data \ --mm-encoder-attn-backend ROCM_AITER_FA \ --tool-call-parser minimax_m3 \ --reasoning-parser minimax_m3 \ --enable-auto-tool-choice ``` #### DP8 + Expert Parallel ```bash vllm serve MiniMaxAI/MiniMax-M3 \ --data-parallel-size 8 \ --enable-expert-parallel \ --block-size 128 \ --attention-backend TRITON_ATTN \ --mm-encoder-tp-mode data \ --mm-encoder-attn-backend ROCM_AITER_FA \ --tool-call-parser minimax_m3 \ --reasoning-parser minimax_m3 \ --enable-auto-tool-choice ``` ## FP8 KV Cache Add `--kv-cache-dtype fp8` to any command for ~1.5× the KV pool — lossless in our testing across the full native context. Especially worth it for high concurrency or long context, where KV is the binding constraint. ## Context Length & GPU Memory The full **1M-token** window (`context_length: 1048576`) needs a large KV cache. To save GPU memory, you can optionally cap the context with `--max-model-len`: ```bash vllm serve MiniMaxAI/MiniMax-M3 \ --tensor-parallel-size 8 \ --block-size 128 \ --max-model-len 131072 # 128K instead of the full 1M ``` AMD ROCm notes: Native context is 512K. To go past it, supply a YaRN rope_scaling on the text config (a top-level override silently misses the decoder's config) and allow the long max length. TP=8 + fp8 KV is the practical combo at 1M: ```bash VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 \ vllm serve MiniMaxAI/MiniMax-M3 \ --block-size 128 \ --kv-cache-dtype fp8 \ --tensor-parallel-size 8 \ --max-model-len 1048576 \ --attention-backend TRITON_ATTN \ --mm-encoder-tp-mode data \ --mm-encoder-attn-backend ROCM_AITER_FA \ --tool-call-parser minimax_m3 \ --enable-auto-tool-choice \ --reasoning-parser minimax_m3 \ --hf-overrides '{"text_config":{"rope_scaling":{"rope_type":"yarn","factor":2.0,"original_max_position_embeddings":524288}}}' ``` - Set `--max-model-len` to the longest prompt + output you actually serve (e.g. `32768`, `131072`, `262144`). A smaller window frees KV-pool headroom for higher concurrency and lets the model fit on fewer GPUs; if you need the full 1M window, consider scaling out with multi-node TP instead. ## Client Usage Recommended sampling parameters (from the model card): - `temperature = 1.0` - `top_p = 0.95` - `top_k = 40` Default system prompt: ``` You are a helpful assistant. Your name is MiniMax-M3 and is built by MiniMax. ``` Example chat request: ```bash curl http://localhost:8000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "MiniMaxAI/MiniMax-M3", "temperature": 1.0, "top_p": 0.95, "messages": [ {"role": "system", "content": "You are a helpful assistant. Your name is MiniMax-M3 and is built by MiniMax."}, {"role": "user", "content": "Explain MSA sparse attention in 3 bullets."} ] }' ``` ## Thinking Modes M3 reasoning is controlled by the `thinking_mode`, there are three values: - **`enabled`** — the model thinks before every response, including after tool results. Use for complex reasoning and agents. - **`disabled`** — no thinking; the model answers directly. Use for latency-sensitive turns. - **`adaptive`** *(default when unset)* — the model decides whether to think based on the task. Pass it per request through `chat_template_kwargs`. The same value also tunes the `minimax_m3` reasoning parser, so `reasoning` and `content` are split correctly in every mode. ```python # Start the MiniMax-M3 model by referring to the command above first. from openai import OpenAI client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1") response = client.chat.completions.create( model="MiniMaxAI/MiniMax-M3", messages=[{"role": "user", "content": "Prove there are infinitely many primes."}], extra_body={"chat_template_kwargs": {"thinking_mode": "enabled"}}, ) msg = response.choices[0].message print(msg.reasoning) # the block print(msg.content) # the final answer ``` ## Benchmarking ```bash vllm bench serve \ --backend vllm \ --model MiniMaxAI/MiniMax-M3 \ --endpoint /v1/completions \ --dataset-name random \ --random-input 2048 \ --random-output 1024 \ --max-concurrency 10 \ --num-prompt 100 ``` ## Quantized Variant (MXFP8) [`MiniMaxAI/MiniMax-M3-MXFP8`](https://huggingface.co/MiniMaxAI/MiniMax-M3-MXFP8) is an MXFP8 checkpoint quantized by NVIDIA from the original FP16 weights — roughly half the VRAM of the BF16 release. Select the **mxfp8** variant above, or pass the repo id directly to `vllm serve`: ```bash vllm serve MiniMaxAI/MiniMax-M3-MXFP8 \ --tensor-parallel-size 8 \ --block-size 128 \ --tool-call-parser minimax_m3 \ --reasoning-parser minimax_m3 \ --enable-auto-tool-choice ``` For best MXFP8 throughput, prefer Blackwell (B200/B300) for native MX tensor cores, or AMD CDNA4 (MI350X/MI355X, gfx950) for native MXFP8 matrix cores. On AMD MI300X, the **mxfp8** variant stacks two accuracy-safe scheduling levers from the InferenceX MI300X tuning recipe and avoids the high-concurrency expert-parallel regression by using plain TP8: ```bash export VLLM_ENGINE_READY_TIMEOUT_S=3600 export VLLM_USE_BREAKABLE_CUDAGRAPH=0 export VLLM_ROCM_USE_AITER=1 export VLLM_ROCM_USE_AITER_MHA=0 export TORCH_BLAS_PREFER_HIPBLASLT=1 export NCCL_MIN_NCHANNELS="${NCCL_MIN_NCHANNELS:-112}" export GPU_MAX_HW_QUEUES="${GPU_MAX_HW_QUEUES:-2}" vllm serve MiniMaxAI/MiniMax-M3-MXFP8 \ --tensor-parallel-size 8 \ --block-size 128 \ --language-model-only \ --max-model-len "$MAX_MODEL_LEN" \ --attention-backend TRITON_ATTN \ --async-scheduling \ --max-num-batched-tokens 16384 \ --tool-call-parser minimax_m3 \ --reasoning-parser minimax_m3 \ --enable-auto-tool-choice ``` On AMD MI355X the **mxfp8** variant additionally enables the AITER kernels, the fused shared-experts MoE, and INT4 QuickReduce, and pulls the nightly ROCm image (`vllm/vllm-openai-rocm:nightly`) — selected automatically in the command builder. If you launch by hand: ```bash export VLLM_USE_BREAKABLE_CUDAGRAPH=0 export VLLM_ROCM_USE_AITER=1 export VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS=1 export VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4 vllm serve MiniMaxAI/MiniMax-M3-MXFP8 \ --tensor-parallel-size 8 \ --block-size 128 \ --attention-backend TRITON_ATTN \ --tool-call-parser minimax_m3 \ --reasoning-parser minimax_m3 \ --enable-auto-tool-choice ``` ### MXFP8 high-concurrency options on MI355X The command above is the default serving configuration and is tuned for the general case. **AITER page-16 sparse paged attention** helps specifically at **long context and high concurrency**. It maps MiniMax-M3's top-k 128-token MSA blocks onto AITER page-16 block tables and runs AITER Gluon paged attention over only the selected KV pages. The high-concurrency configuration is: ```bash # Long context + high concurrency only (isl >= 8k, conc >= 64). Regresses # short-context / low-concurrency serving. export VLLM_USE_BREAKABLE_CUDAGRAPH=0 export VLLM_ROCM_USE_AITER=1 export VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS=1 export VLLM_ROCM_SHUFFLE_KV_CACHE_LAYOUT=1 export VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4 export VLLM_ROCM_QUICK_REDUCE_CAST_BF16_TO_FP16=0 export VLLM_ROCM_QUICK_REDUCE_QUANTIZATION_MIN_SIZE_KB=256 vllm serve MiniMaxAI/MiniMax-M3-MXFP8 \ --tensor-parallel-size 4 \ --block-size 128 \ --language-model-only \ --moe-backend aiter \ --kv-cache-dtype fp8 \ --max-num-batched-tokens 32768 \ --attention-backend TRITON_ATTN \ --linear-backend emulation \ --tool-call-parser minimax_m3 \ --reasoning-parser minimax_m3 \ --enable-auto-tool-choice ``` Note: Outside the long-context/high-batch regime the shuffled layout costs more than it saves; leave it at `0`: ```bash export VLLM_ROCM_SHUFFLE_KV_CACHE_LAYOUT=0 ``` ## Quantized Variant (NVFP4 on Blackwell) [`nvidia/MiniMax-M3-NVFP4`](https://huggingface.co/nvidia/MiniMax-M3-NVFP4) is an NVFP4 checkpoint quantized by NVIDIA with ModelOpt. It requires a vLLM build with MiniMax-M3 NVFP4 support (vllm-project/vllm PR #46380). Select the **nvfp4** variant and **B200** hardware in the command builder, or launch the tested B200 baseline directly: ```bash VLLM_FLOAT32_MATMUL_PRECISION=high \ vllm serve nvidia/MiniMax-M3-NVFP4 \ --tensor-parallel-size 8 \ --trust-remote-code \ --block-size 128 \ --tool-call-parser minimax_m3 \ --reasoning-parser minimax_m3 \ --enable-auto-tool-choice ``` Pair it with EAGLE3 speculative decoding (the *Spec decoding* feature) for faster decode — the validated B200 benchmark runs the NVFP4 target against the `Inferact/MiniMax-M3-EAGLE3` draft head (3 speculative tokens): ```bash VLLM_FLOAT32_MATMUL_PRECISION=high \ vllm serve nvidia/MiniMax-M3-NVFP4 \ --tensor-parallel-size 8 \ --trust-remote-code \ --block-size 128 \ --language-model-only \ --speculative-config '{"method": "eagle3", "model": "Inferact/MiniMax-M3-EAGLE3", "num_speculative_tokens": 3, "attention_backend": "FLASH_ATTN"}' ``` For long-context or high-batch workloads where draft KV memory is the binding constraint, swap in the **GQA** variant: ```bash VLLM_FLOAT32_MATMUL_PRECISION=high \ vllm serve nvidia/MiniMax-M3-NVFP4 \ --tensor-parallel-size 8 \ --trust-remote-code \ --block-size 128 \ --language-model-only \ --speculative-config '{"method": "eagle3", "model": "Inferact/MiniMax-M3-EAGLE3-GQA", "num_speculative_tokens": 3, "attention_backend": "FLASH_ATTN"}' ``` You may also consider FlashInfer for draft attention if it performs better for your workload, and local argmax reduction to reduce communication traffic: ```bash --speculative-config '{"method":"eagle3","model":"Inferact/MiniMax-M3-EAGLE3-GQA","num_speculative_tokens":3,"attention_backend":"FLASHINFER","use_local_argmax_reduction":true}' ``` ## GB200: InferenceX AgentX aggregated (Dynamo + vLLM) The GB200 AgentX lane in [SemiAnalysis InferenceX](https://inferencex.semianalysis.com/) (`minimaxm3-fp4-gb200-dynamo-vllm-agentic-agg-mtp`, [SemiAnalysisAI/InferenceX#4281](https://github.com/SemiAnalysisAI/InferenceX/pull/4281)) serves the NVFP4 checkpoint with the GQA EAGLE3 draft as aggregated vLLM workers behind the Dynamo frontend's KV-aware router. Its measured points are two opt-in presets: [AgentX TP4](#preset-gb200_agentx), one worker per 4-GPU tray, whose larger points add host KV offload; and [AgentX TP8](#preset-gb200_agentx_tp8), one worker across two trays. Each point renders the worker's engine flags, environment and pinned image. The TP4 preset's runbook has the etcd, NATS, Dynamo worker and frontend steps that the command builder doesn't render. ## Quantized Variant (MXFP4 on MI355X) [`amd/MiniMax-M3-MXFP4`](https://huggingface.co/amd/MiniMax-M3-MXFP4) uses static OCP MXFP4 weights and dynamic OCP MXFP4 activations produced with AMD Quark. It targets AMD Instinct MI355X (gfx950) and requires a ROCm nightly containing the Quark MXFP4 fix from vLLM PR #45794. Select the **mxfp4** variant and **MI355X** hardware in the command builder, or launch the tested TP8 baseline directly: ```bash vllm serve amd/MiniMax-M3-MXFP4 \ --tensor-parallel-size 8 \ --trust-remote-code \ --block-size 128 \ --attention-backend TRITON_ATTN \ --mm-encoder-tp-mode data \ --mm-encoder-attn-backend ROCM_AITER_FA \ --tool-call-parser minimax_m3 \ --enable-auto-tool-choice \ --reasoning-parser minimax_m3 ``` ### MXFP4 AgentX on MI355X The opt-in [AgentX preset](#preset-mi355x_agentx) follows [InferenceX #4418](https://github.com/SemiAnalysisAI/InferenceX/pull/4418) on its pinned ROCm image: TP4 at concurrency 1, 4, 5, 8, 10, 12, 15, 20, 24 and 32 enables monokernel; TP2 at 1 and 2 uses ordinary decode. Both use Model Runner V2, the three-token GQA EAGLE3 draft and real target verification, with no offload. These text-only MXFP4 settings are workload-specific; the general serving configuration remains unchanged. Synthetic acceptance is for throughput reproduction only, as described in the preset's runbook. ## Troubleshooting - **`--block-size` mismatch.** MSA's sparse block size is 128; the vLLM KV cache block size must match. Using the default (16) misaligns the sparse attention indexing (on AMD it crashes with `No common block size for 16`). - **Parsers.** `--tool-call-parser` and `--reasoning-parser` both use `minimax_m3` — distinct from `minimax_m2` used by earlier releases. - **Long context KV cache.** See *Context Length & GPU Memory* above — cap `--max-model-len` or scale to multi-node TP if you OOM. - **Vision encoder.** The encoder is small, so at high TP the *Encoder Parallel* option runs it data-parallel (`--mm-encoder-tp-mode data`) to avoid TP comm overhead; it also turns on the vision-encoder attention backend (FlashInfer on NVIDIA, `--mm-encoder-attn-backend FLASHINFER`; AITER FlashAttention on AMD, `ROCM_AITER_FA`) and the host-shared-memory multimodal processor cache (`--mm-processor-cache-type shm`). For **text-only** workloads enable *Text only* (`--language-model-only`) to skip loading the encoder and free VRAM — it is mutually exclusive with Encoder Parallel. ## EAGLE3 Speculative Decoding Two Inferact draft heads are available as selectable modes in the *Spec decoding* feature — both use `method: eagle3` with 3 speculative tokens and share MiniMax-M3's embedding table and LM head: | Mode | Repo | KV heads | Draft KV cache | |------|------|----------|----------------| | **MHA** (default) | `Inferact/MiniMax-M3-EAGLE3` | 64 | baseline | | **GQA** | `Inferact/MiniMax-M3-EAGLE3-GQA` | 4 | **16× smaller** | The GQA head uses Grouped Query Attention (4 KV heads grouped from 64 query heads) — the only architectural difference from the MHA head. The 16× reduction in draft KV state makes it preferable when draft memory is the binding constraint (long-context serving or large batch sizes). ## References - [Model card](https://huggingface.co/MiniMaxAI/MiniMax-M3) - [MXFP8 variant](https://huggingface.co/MiniMaxAI/MiniMax-M3-MXFP8) - [NVFP4 variant](https://huggingface.co/nvidia/MiniMax-M3-NVFP4) - [AMD MXFP4 variant](https://huggingface.co/amd/MiniMax-M3-MXFP4) - [EAGLE3 draft head (MHA)](https://huggingface.co/Inferact/MiniMax-M3-EAGLE3) - [EAGLE3 draft head (GQA)](https://huggingface.co/Inferact/MiniMax-M3-EAGLE3-GQA) - [MiniMax](https://www.minimax.io/) - [MiniMax Agent](https://agent.minimax.io/) - [MiniMax Platform](https://platform.minimax.io/) - [Previous release: MiniMax-M2.7](https://huggingface.co/MiniMaxAI/MiniMax-M2.7) --- # deepseek-ai/DeepSeek-V4.1-Flash on vLLM Source: https://recipes.vllm.ai/deepseek-ai/DeepSeek-V4.1-Flash Provider: DeepSeek Released: 2026-09-10T02:17:58.000Z Updated: 2026-10-25 Architecture: moe Parameters: 552B Context length: 1,048,576 Min vLLM: 0.31.0 Tasks: text, multimodal Verified hardware: H100, H200, B200, GB200 NVL4, GB300 NVL4, B300, MI300X, MI325X, MI350X, MI355X, DGX Station (GB300) Summary: DeepSeek V4.1 Flash vision-language MoE (552B backbone; 8B active per prompt token, 16B per output token) combining sliding-window plus compressed sparse attention with a two-level indexer, engram n-gram memory, hyper-connections, and a DSpark multi-token draft head. ## Variants - default: precision=fp8, vram_minimum_gb=614, model_id=deepseek-ai/DeepSeek-V4.1-Flash-1017 DeepSeek-V4.1-Flash-1017 (2026-10-17): native mixed checkpoint (MXFP4 MoE experts, MXFP8 block-quantized dense weights, UE8M0 scales) with updated weights — stronger reasoning, coding, agentic and visual results, and shorter outputs. - fp8: precision=fp8, vram_minimum_gb=614 Launch checkpoint (2026-09-09), superseded by 1017 — kept for reproducing earlier results. Same mixed MXFP4/MXFP8 format. ## Recommended base flags ```bash vllm serve 'deepseek-ai/DeepSeek-V4.1-Flash' \ --tokenizer-mode \ deepseek_v41 ``` Environment: ```bash VLLM_ENGINE_READY_TIMEOUT_S='3600' ``` ## Guide ## Overview DeepSeek-V4.1 is a vision-language Mixture-of-Experts model: **552B backbone parameters plus 196B of Engram memory, 8B active per prompt token and 16B per output token**, 40 transformer layers at hidden size 5120, with a 32-layer ViT and aligner in front of the text stack. Six things distinguish it from V4: - **Causal encoder-decoder.** The 40 layers split into a 20-layer encoder and a 20-layer decoder, and the decoder's global KV is projected from the encoder's final hidden state instead of being computed layer by layer. Prefill therefore only runs the encoder half, which is where the 8B-prefill / 16B-decode split comes from. - **Two-tier sparse attention.** Every layer attends over a 128-token sliding window. Layers that carry a compression ratio add compressed KV latents reaching further back, pooled by a learned softmax gate. A small side attention — the *indexer*, inherited from V3.2-Exp — scores those latents and keeps the best 512 per query, pre-filtered by a candidate stage that selects 2048 blocks of 8. Only four layers (2, 8, 14, 20) actually compress their own KV; the rest read that cache. V4.1 uses only compression ratios 1 and 2, where V4 used 4 and 128. - **Engram n-gram memory.** Layers 1 and 14 each own a hash table of ~384M rows x 256 dims, looked up by 2-, 3- and 4-gram hashes of the input and written into the residual stream through a learned gate. These two tables alone are **196.6B parameters (~183 GiB)** — plan capacity for them, they dominate everything except the experts. - **Hyper-Connections.** The residual stream is carried as 4 parallel copies; each sublayer derives its own pre/post/combine coefficients from the stream, with the combine matrix made doubly stochastic by 20 Sinkhorn iterations. - **DSpark draft head.** Three stages (128 routed experts each, 3 activated) draft a block of 5 tokens, reading the attention input of layers 37-39, with a Markov bias head and a confidence head on the last stage. - **Mixed MXFP4/MXFP8 checkpoint.** Routed expert weights are MXFP4; everything else is MXFP8 block-quantized, UE8M0 scales throughout. Embedding and LM head are BF16. Routing is 6 of 384 experts per token plus one shared expert, scored with `sqrtsoftplus` and a `noaux_tc` bias — with a **separate routing bias for tokens inside an image span**, so vision and text tokens do not compete for the same experts. ## Checkpoints | Variant | Checkpoint | Notes | |---|---|---| | **FP8 (1017)** (default) | [deepseek-ai/DeepSeek-V4.1-Flash-1017](https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017) | 2026-10-17 weight update — recommended | | **FP8 (0909)** | [deepseek-ai/DeepSeek-V4.1-Flash](https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash) | Launch release (2026-09-09) | **1017 is the default here.** It is a minor update with the same architecture, `config.json`, tokenizer and prompt encoding as the launch checkpoint — only the weights change. DeepSeek reports gains across reasoning, coding, agentic and visual benchmarks (e.g. Terminal-Bench 4.0 31.2 → 42.3, HLE w/o tools 36.8 → 42.6) along with shorter outputs. Both serve with the same command; the Variant row only swaps the model id. Like the launch repo it ships no Jinja chat template, which the `--tokenizer-mode deepseek_v41` flag already covers. Pick **FP8 (0909)** to reproduce earlier results. The verified hardware badges and the measured numbers in the sections below (benchmarks, KV offloading validation) were taken on the 0909 launch checkpoint; the 1017 weights share its config and footprint but have not been separately re-run. ## Context length 1,048,576 tokens, reached by YaRN with factor 16 over a 65,536-token training window. Compressed KV rotates at its own RoPE theta (160,000) because one latent stands for several tokens, so its positions are further apart than the raw stream's. ## Images Images enter the prompt as `<|deepseek_image|>` spans; every position in the span carries the image token id, and the per-position role (start / newline / end / image) comes from the processor. The vision tower is a 32-layer ViT at hidden size 1024, patch 14, with a 3x downsampling aligner and a cap of 1024 tokens per image (minimum 295,936 pixels). There is **no limit on images per prompt**. Merged embeddings enter the text model as `inputs_embeds`, before the hyper-connection stream expansion, while raw token ids still flow through so the router can apply the image routing bias. Tick **Encoder parallel** to run the ViT data-parallel (`--mm-encoder-tp-mode data`) instead of tensor-parallel: at 32 layers / hidden 1024 the encoder is small enough that TP communication costs more than it saves, which can significantly reduce TTFT for multi-image requests. It is mutually exclusive with **Text only**. ## Reasoning and tool calling Two thinking modes and a **numeric reasoning budget** rather than discrete tiers. Send them as `chat_template_kwargs`: | key | values | |---|---| | `thinking` / `enable_thinking` | boolean; if you send both they must agree | | `reasoning_effort` | `low` (25), `high` (50), `xhigh` (75), `max` (100), **or an integer 1-100** | The top-level OpenAI `reasoning_effort` field also works, where `"none"` turns thinking off. `"minimal"` and `"medium"` are rejected — they are not part of this model's set. Note that the DeepSeek API maps its `low` / `high` / `max` tiers to 50 / 75 / 100, while the open-source prompt encoder (and therefore vLLM) maps `low` / `high` / `xhigh` / `max` to 25 / 50 / 75 / 100. A request tuned at `high` on the API runs at effort 50 here; pass an integer if you need parity. > **With both keys unset, thinking is ON at effort 50.** The do-nothing config is > the most verbose one, so a request with a small `max_tokens` spends its budget on > the trace and returns empty `content` with `finish_reason=length`. That reads as a > broken model and is not. Either pin `thinking: false` or give the budget room. In thinking mode the budget is rendered into the prompt as a `Reasoning Effort: N` prefix on the first turn only. ```python resp = client.chat.completions.create( model="deepseek-ai/DeepSeek-V4.1-Flash-1017", messages=[{"role": "user", "content": "What is 17*19?"}], extra_body={"chat_template_kwargs": {"thinking": True, "reasoning_effort": 25}}, ) ``` DeepSeek's published numbers use `temperature=1.0`, `top_p=0.95` and effort 100 with `max_tokens` of at least 256K. The tech report notes that efforts of 60-80 recover most of that accuracy at well under half the tokens, so reserve `max` for hard tasks. Tool calls are wrapped in DSML tag blocks rather than JSON fences, and tool output comes back in `` tags. ## Speculative decoding DSpark is the only speculative method for this checkpoint. V4.1 dropped the MTP module that V3 and V4 trained alongside the backbone. **Speculative decoding** enables a 5-token block. On NVIDIA it also turns on adaptive verification (see the feature description). On AMD, vLLM currently refuses adaptive verification. The generated command sets `enable_adaptive_verification:false` and verifies the full block. Two checks in `maybe_create_adaptive_verification_manager` fail on ROCm: the indexer helper `supports_device_cpu_query_lens_mismatch()` is False, and `DeepseekV41ROCMAiterSparseSWABackend` reports `AttentionCGSupport.UNIFORM_BATCH` rather than `ALWAYS`. Remove the AMD override only after a serve with the true flag starts. Acceptance depends on the workload, so measure it on your own traffic before sizing a deployment around it. The drafter's experts add roughly 14B parameters to the load. ## Serving text-only Tick **Text only** to add `--language-model-only`, which skips the vision encoder entirely. Worth it whenever the workload is text: it drops the ViT and aligner from the load and frees that VRAM for KV cache. It is mutually exclusive with `encoder_parallel`. The verified GB200 runs (TP4 and 1P1D) were text-only. ## Prefill/Decode disaggregation The **Prefill/Decode Disaggregation** strategy is the verified 1P1D layout on GB200 NVL4: one tray (4 GPUs) per role, TP4 in each pool, KV handed over through NIXL, fronted by `vllm-router --vllm-pd-disaggregation`. Both pools disable FlashInfer autotune plus JIT and CuTeDSL warmup via `--kernel-config`, skip the DeepGEMM warmup (`VLLM_DEEP_GEMM_WARMUP=skip`), and cap `--max-num-seqs` at 32. On 8-GPU nodes the same layout becomes TP8 per role. With **Speculative decoding** on, DSpark runs in both pools so the transferred KV stays compatible. ### GB300 GB300's default **AgentX PD** preset runs both pools as DEP, per [SemiAnalysisAI/InferenceX#3941](https://github.com/SemiAnalysisAI/InferenceX/pull/3941): one tray of DEP4 prefill, four trays of DEP16 decode. Keep them on DEP; MegaMoE needs expert parallelism. To scale prefill, InferenceX ran two DEP2 engines per tray. Drop the `--data-parallel-{start-rank,address,rpc-port}` flags from the prefill command and run it twice: ```bash CUDA_VISIBLE_DEVICES=0,1 VLLM_NIXL_SIDE_CHANNEL_PORT=5557 vllm serve ... --port 8001 \ --data-parallel-size 2 --data-parallel-size-local 2 CUDA_VISIBLE_DEVICES=2,3 VLLM_NIXL_SIDE_CHANNEL_PORT=5559 vllm serve ... --port 8011 \ --data-parallel-size 2 --data-parallel-size-local 2 # vllm-router: list both ports, with --intra-node-data-parallel-size 1 ``` ## Memory The checkpoint is roughly **511 GB on disk** (476 GiB), which breaks down as: | Component | Entries | Stored | |---|---:|---:| | Routed + DSpark experts (MXFP4) | 557.2B | 259.5 GiB | | Engram tables (FP8) | 196.6B | 183.1 GiB | | Attention, dense projections, routers (FP8) | 7.4B | 6.9 GiB | | Embedding + LM head (BF16), norms (FP32) | 2.0B | 3.9 GiB | | UE8M0 block scales | 23.6B | 21.9 GiB | DeepSeek's 552B backbone figure covers the routed experts, attention and embeddings; the Engram tables, the ~14B DSpark drafter and the block scales sit outside it. `vram_minimum_gb: 614` is that total times the schema's 1.2 headroom factor. It fits one GB200 NVL4 tray (768 GB) at TP4, or one 8-GPU H200 node (1128 GB) with room for KV cache. KV cache is a small share of that budget. The compressed latents are shared across layers and trained to be stored in FP4, and DeepSeek puts the resulting global KV at **890 bytes per token**, about a quarter of V4-Flash; by that accounting a full 1M-token prompt holds under 1 GB of global KV, plus a fixed 128-token sliding window per layer. Weights and batch size, not cache, set the capacity limit — still, measure before assuming a given context/batch fits. ## Prerequisites - vLLM 0.31.0+, from the pip wheel or the release image. The architecture shipped in 0.30.0 ([vllm-project/vllm#56228](https://github.com/vllm-project/vllm/pull/56228)); 0.31.0 adds the KV-offload fix ([vllm-project/vllm#57145](https://github.com/vllm-project/vllm/pull/57145)) and the ROCm Engram host-memory gate ([vllm-project/vllm#57491](https://github.com/vllm-project/vllm/pull/57491)). The launch-day `deepseekv41-flash-0909` image is no longer needed. - On AMD the Docker tab uses `vllm/vllm-openai-rocm:nightly-rocm100`, the ROCm 10.0 nightly the AMD rows were verified on. **MI355X and MI325X need it**: their commands pass `--monokernel`, and MI355X also passes the ROCm MXFP4 indexer. Both are on `main` but not in 0.31.0, so the Install block offers only Docker on those two SKUs. MI300X and MI350X also run from the ROCm pip wheel. - The Rust OpenAI frontend is selected by default in the command builder. Switch to Python if you encounter unsupported features or compatibility issues. - Expect a long first load: `VLLM_ENGINE_READY_TIMEOUT_S=3600` is set for that reason. - On AMD, the generated command sets `VLLM_USE_BREAKABLE_CUDAGRAPH=1`. DeepSeek-V4.1-Flash does not support `torch.compile`, and the ROCm sparse SWA backend only supports uniform-batch CUDA graphs. Without breakable CUDA graphs, default `FULL_AND_PIECEWISE` dies at capture. - On AMD, the generated command passes `--moe-backend aiter` rather than naming a kernel. Naming `aiter_triton_mxfp4_bf16` pins the Triton W4A16 `_moe_gemm_a16w4` kernel; the plain name lets vLLM select the Composable Kernel a8w4 experts, used in the MI355X benchmark below. It also sets `AITER_TRITON_LOG_LEVEL=ERROR` to drop AITER's per-call "Gluon unavailable" warning, which is constant on gfx950 and dominated the server log. - On MI355X, **Tensor Parallel** inherits vLLM's scheduler limits. Tune `--max-num-seqs` for your traffic and memory budget if needed. The benchmark configuration below records an explicit limit for reproducibility. - On AMD, ticking **Speculative decoding** also sets `enable_adaptive_verification:false`. vLLM currently refuses the true flag: `DeepseekV41IndexerBackend.supports_device_cpu_query_lens_mismatch()` is False, and `DeepseekV41ROCMAiterSparseSWABackend` reports `UNIFORM_BATCH` rather than `ALWAYS`. DSpark still drafts 5 tokens per round. ## Verifying Serve, then send one text request and one image request — the vision tower is a separate path and a text-only smoke test will not exercise it: ```bash curl http://localhost:8000/v1/chat/completions \ -H 'Content-Type: application/json' \ -d '{"model":"deepseek-ai/DeepSeek-V4.1-Flash-1017", "messages":[{"role":"user","content":"What is 17*19? Return only the integer."}]}' ``` A correct answer is `323`. ## Performance tuning Start with vLLM's defaults and tune one parameter at a time for your workload: - `--max-num-seqs`: raise the running-sequence limit if it causes queueing and memory permits; lower it if larger batches hurt latency or memory use. - `--max-num-batched-tokens`: balance prefill throughput against decode latency and memory use by adjusting the token budget per scheduling iteration. - `--max-cudagraph-capture-size`: cover expected decode batches, including DSpark draft tokens, while accounting for graph memory and startup cost: 128 sequences with five drafts need `128 * (1 + 5) = 768` tokens, which the MI355X AgentX preset rounds up to 1024. - `--max-model-len`: size the context limit for your prompt plus output lengths. ## MI355X Tensor Parallel On MI355X, **Tensor Parallel** defaults to **TP2** with Engram CPU offload (`--engram-config '{"cpu_offload":true}'`). The Engram tables cost 47.2 GiB of device memory per rank at TP4 and 94.4 GiB at TP2; on a 288 GiB card the TP2 figure does not fit beside half the checkpoint, so the tables move to pinned host memory. TP2 halves the GPUs per server and so doubles the servers per node. For TP4, pass `--tensor-parallel-size 4` and `--engram-config '{"cpu_offload":false}'` to keep the tables resident. The offload resolves on gfx950 only on builds carrying [vllm-project/vllm#57491](https://github.com/vllm-project/vllm/pull/57491), which widened two `is_cuda()` gates to `is_cuda_alike()`. Those builds also default `cpu_offload` on through `VLLM_PLE_CPU_OFFLOAD`, so set it explicitly rather than inheriting the default. `--monokernel` selects the mono decode layer from [vllm-project/vllm#60397](https://github.com/vllm-project/vllm/pull/60397), which [vllm-project/vllm#61082](https://github.com/vllm-project/vllm/pull/61082) extended to all 40 backbone layers and moved from the `VLLM_ROCM_MONO_DECODE` environment variable to this flag. Two persistent launches cover a layer for decode steps of up to 48 rows, which with five DSpark drafts is concurrency 8; larger steps fall back to the regular path, so the gain is largest at low concurrency and fades as concurrency rises. Decoder SWA bounded replay ([vllm-project/vllm#59039](https://github.com/vllm-project/vllm/pull/59039)) is on by default on ROCm, so the recipe no longer passes `--no-swa-bounded-replay`. Use a nightly from 2026-10-13 or later; earlier ROCm builds lack the replay window clamp and fault at the first prefix hit. ### AgentX on MI355X The **AgentX TP2** preset (opt-in; [InferenceX #4480](https://github.com/SemiAnalysisAI/InferenceX/pull/4480)) renders the lane on its pinned ROCm image: TP2 for concurrency 1 to 64 with Engram CPU offload, and TP4 at concurrency 1 and 2 with resident Engram tables. Its points carry the corresponding sequence caps, prefill budgets and graph sizes. The lane served **FP8 (0909)**. These limits are workload-specific, not generic defaults. ## MI300X Tensor Parallel On MI300X, **Tensor Parallel** takes the recipe default of **TP4** with the Engram tables offloaded to pinned host memory (`--engram-config '{"cpu_offload":true}'`). At TP4 the resident weights are roughly 81 GiB per rank, which leaves a healthy KV pool on a 192 GB card. **TP2 does not fit on this SKU.** At TP2 the offloaded weights rise to roughly 145 GiB per rank, and a measured run reported `Available KV cache memory: -13.51 GiB`, so the engine refused to start with `No available memory for the cache blocks`. That was already with the sparse-attention indexer buffer halved to 4096 batched tokens. Raising `--gpu-memory-utilization` and halving the buffer again recover roughly 13 GiB between them, which still leaves no usable KV for a 1M-context model, so TP2 is not offered here. MI325X runs the same TP2 configuration successfully on its larger 256 GB card. MI300X is gfx942 and needs the same two workarounds as MI325X: no FP4 MFMA, so the MXFP4 experts run through a Triton kernel and device memory is better spent on KV than on resident Engram tables; and piecewise graph capture segfaults every worker, so this recipe sets `VLLM_USE_BREAKABLE_CUDAGRAPH=0` and captures only full decode graphs (`--compilation-config '{"cudagraph_mode":"FULL_DECODE_ONLY"}'`), with prefill eager. This recipe does not pass `--monokernel`. With the kernels' weight copies included, #60731 measured 170.11 GiB of weights a rank. At `--gpu-memory-utilization 0.9` a 192 GiB MI300X leaves about 2.7 GiB beside those weights. The offload resolves on gfx942 only on builds carrying [vllm-project/vllm#57491](https://github.com/vllm-project/vllm/pull/57491), the same gate MI325X and MI355X need. Those builds also default `cpu_offload` on through `VLLM_PLE_CPU_OFFLOAD`, so set it explicitly rather than inheriting the default. ## MI325X Tensor Parallel On MI325X, **Tensor Parallel** takes the recipe default of **TP4**, with the Engram tables offloaded to pinned host memory (`--engram-config '{"cpu_offload":true}'`). A 256 GB card leaves roughly 81 GiB per rank of resident weights at TP4 and roughly 145 GiB at TP2; TP2 fits but leaves the smallest KV pool of any arm in this recipe, so TP4 is the default here rather than the TP2 that MI355X uses on its larger 288 GiB card. For TP2, pass `--tensor-parallel-size 2`. MI325X is gfx942, which differs from gfx950 in two ways that matter. It has no FP4 MFMA, so the MXFP4 experts run through a Triton kernel and device memory is better spent on KV than on resident Engram tables. It also segfaults in every worker during piecewise graph capture, so this recipe sets `VLLM_USE_BREAKABLE_CUDAGRAPH=0` and captures only full decode graphs (`--compilation-config '{"cudagraph_mode":"FULL_DECODE_ONLY"}'`); prefill runs eagerly. `--monokernel` selects the gfx942 mono decode kernels from [vllm-project/vllm#60731](https://github.com/vllm-project/vllm/pull/60731). They run at TP4 only, on decode steps of up to 42 rows (7 requests with five DSpark drafts). Larger steps, prefill, and mixed batches take the regular path, and TP2 stays on vLLM's path. With their weight copies included, #60731 measured 170.11 GiB of weights a rank, up from 98.88 GiB. At `--gpu-memory-utilization 0.9` the KV pool still held 11.18 million tokens. Use a ROCm nightly from 2026-10-20 or later; `nightly-rocm100-df3bd094aeea829f27baa0a5282830464f601d9f` is the first. Earlier builds have the flag for gfx950 only and refuse to start on MI325X. The offload resolves on gfx942 only on builds carrying [vllm-project/vllm#57491](https://github.com/vllm-project/vllm/pull/57491), the same gate MI355X needs. Those builds also default `cpu_offload` on through `VLLM_PLE_CPU_OFFLOAD`, so set it explicitly rather than inheriting the default. ## H200 Tensor Parallel On H200, **Tensor Parallel** uses the TP4 default with Engram CPU offload (`--engram-config '{"cpu_offload":true}'`). The Engram tables move to pinned host DRAM and are read through UVA, so the output is unchanged. On 141 GB cards that offload is what makes TP4 practical: it moves 23.6 GiB per rank off the GPU, leaving 81.2 GiB of resident weights and 38.5 GiB of KV per GPU (about 20M tokens). No scheduler caps are needed at this replica size. ## Blackwell TP and DEP On B200, B300, GB200 and GB300, **Tensor Parallel** uses **TP2** with `FLASHINFER_MLA_SPARSE_DSV41` and Engram CPU offload (`--engram-config '{"cpu_offload":true}'`), which moves the Engram tables to pinned host DRAM and reads them through UVA; the output is unchanged. **Data + Expert Parallel** uses the hardware's GPU count: DEP4 on GB200/GB300 trays, DEP8 on B200/B300 nodes. It enables `--enable-expert-parallel`, `FLASHMLA_MEGA_ATTN_DSV41`, `--kernel-config '{"moe_backend":"deep_gemm_mega_moe"}'`, and `--engram-config '{"embedding_across_dp":true}'`. Both configurations set `indexer_kv_dtype` to `mxfp4`, `indexer_sparse_logits` to `true`, `--kv-cache-dtype fp8`, and `--max-num-seqs 128`. Reasoning uses `--reasoning-parser deepseek_v41`. Each Blackwell SKU carries an exact-GPU override that takes precedence over the shared Blackwell defaults. B300 keeps its existing CUDA Graph capture configuration (`--max-cudagraph-capture-size 8190`), `--max-num-batched-tokens 8192` and `--max-num-seqs 256`; B200, GB200 and GB300 add Engram CPU offload and otherwise inherit the shared Blackwell settings. On B200 both strategies open on the InferenceX AgentX presets (see **AgentX tuning on NVIDIA GPUs**); pick **Baseline** in the Preset row for the shared Blackwell settings. Scheduler limits are workload-dependent at TP2, so size `--max-num-batched-tokens` and `--max-cudagraph-capture-size` for your own serving profile. For high interactivity on any of them, use `--tensor-parallel-size 4`. ## AgentX tuning on NVIDIA GPUs The InferenceX AgentX lanes that run as one server are presets, each with the lane's concurrency points (see the **Presets** tab): - **B200**: **AgentX TP** (TP4 for concurrency 1 to 128, TP2 at 1) and **AgentX DEP** (DEP2, DEP2 with a host KV tier, and DEP4), the defaults on B200 ([InferenceX #3837](https://github.com/SemiAnalysisAI/InferenceX/pull/3837)). - **B300**: **AgentX TP4** and **AgentX DEP2** on the specialized decode image ([InferenceX #4332](https://github.com/SemiAnalysisAI/InferenceX/pull/4332)). The lanes served the 0909 launch checkpoint, **FP8 (0909)**. The DEP presets use two or four of the node's GPUs, so run one server per GPU group, each behind its own vLLM Router. ### Dynamo disaggregated serving on B300 and GB300 The AgentX lanes in [InferenceX #4231](https://github.com/SemiAnalysisAI/InferenceX/pull/4231) (B300) and [InferenceX #4266](https://github.com/SemiAnalysisAI/InferenceX/pull/4266) (GB300) run N DEP2 prefill engines and one DEP decode engine, each with its own sidecar, behind one KV-routing frontend. Prefill pushes KV to decode with NIXL and keeps a host KV tier. The `push_as_nixl` connector and `VLLM_MAX_ASYNC_KV_ADMITS_PER_STEP` come from the dev images `vllm/vllm-openai:nightly-x86_64-dev-20261012-59e75f4` (B300) and `vllm/vllm-openai:cu134-nightly-aarch64-dev-20261019` (GB300). | GPU | Clients | Prefill engines | Decode | Decode `--max-num-seqs` | `VLLM_MAX_ASYNC_KV_ADMITS_PER_STEP` | |---|---|---:|---|---|---| | B300 | 256, 768, 1280, 2048, 3584 | 2, 2, 3, 4, 5 | DEP8 | 18, 40, 72, 132, 168 | 4 | | GB300 | 3018, 7388, 10888 | 6, 12, 16 | DEP32 | 18, 50, 76 | 2, 4, 5 | | GB300 | 9888 | 12 | DEP16 | 253 | 6 | Both roles use TP1 with expert parallelism and these settings: ```bash export VLLM_USE_V2_MODEL_RUNNER=1 VLLM_ALLREDUCE_USE_SYMM_MEM=0 VLLM_RANDOMIZE_DP_DUMMY_INPUTS=1 NCCL_CUMEM_ENABLE=1 export UCX_MEMTYPE_CACHE=n UCX_MEMTYPE_REG_WHOLE=n UCX_RNDV_PIPELINE_ERROR_HANDLING=y export VLLM_ENGINE_READY_TIMEOUT_S=7200 VLLM_RPC_TIMEOUT=600000 --attention-config '{"backend":"FLASHMLA_MEGA_ATTN_DSV41","indexer_kv_dtype":"mxfp4","indexer_sparse_logits":true}' --kernel-config '{"enable_flashinfer_autotune":true,"moe_backend":"deep_gemm_mega_moe","sparse_indexer_topk_backend":"deep_select"}' --speculative-config '{"method":"dspark","num_speculative_tokens":5,"draft_sample_method":"probabilistic","enable_adaptive_verification":false}' --kv-cache-dtype fp8 --gpu-memory-utilization 0.95 --prefill-schedule-interval 2 --enable-ep-weight-filter --numa-bind ``` | | B300 | GB300 | |---|---|---| | NIXL backend | `LIBFABRIC` over EFA | `UCX` over MNNVL | | Env | `FI_PROVIDER=efa FI_EFA_USE_DEVICE_RDMA=1 FI_EFA_FORK_SAFE=1 UCX_TLS=cuda_copy,cuda_ipc,srd,tcp NIXL_UCX_SGL_ENABLE=0` | `UCX_TLS=cuda_copy,cuda_ipc,tcp UCX_CUDA_IPC_ENABLE_MNNVL=y NIXL_UCX_SGL_ENABLE=1 NCCL_MNNVL_ENABLE=1 NCCL_NVLS_ENABLE=1 VLLM_RS_TOKEN_IDS_AS_BYTES=1` | | Allocator | `--no-enable-cumem-allocator` (EFA cannot register VMM memory) | `--enable-cumem-allocator` | | DSpark | adds `"rejection_sample_method":"block"` | as above | | Engram | `{"cpu_offload":true,"embedding_across_dp":true,"use_thp":false}` | same; decode sets `"cpu_offload":false` | | Host KV per prefill rank | 70 GiB | 80 GiB | | Prefill `--max-num-batched-tokens` | 16384 | 32768 | | Decode EPLB redundant experts | 16 | 16 at DEP16, 32 at DEP32 | | Decode `--numa-bind-nodes` | `0 0 0 0 1 1 1 1` | `0 0 1 1` | On B300, set `--cpu-distributed-timeout-seconds 7200` for the Engram load. NIXL's LIBFABRIC plugin also needs libfabric and rdma-core from the AWS EFA installer and the CUDA 12 runtime (`nvidia-cuda-runtime-cu12`), which the image lacks. Prefill runs `--data-parallel-size 2` per GPU pair with `--max-num-seqs 64`, `--long-prefill-token-threshold 8192` and `--prefix-cache-retention-interval 32768`. Its KV config, shown for B300 (GB300 uses `"backends":["UCX"]` and `85899345920` bytes): ```json {"kv_connector":"MultiConnector","kv_role":"kv_both","kv_connector_extra_config":{"connectors":[ {"kv_connector":"NixlConnector","kv_connector_module_path":"vllm.distributed.kv_transfer.kv_connector.v1.nixl.push_as_nixl","kv_role":"kv_both","kv_load_failure_policy":"fail","kv_buffer_device":"cuda","kv_connector_extra_config":{"backends":["LIBFABRIC"],"enforce_handshake_compat":false,"kv_lease_duration":120}}, {"kv_connector":"SimpleCPUOffloadConnector","kv_role":"kv_both","kv_connector_extra_config":{"cpu_bytes_to_use":75161927680,"lazy_offload":true}}]}} ``` Prefill graphs: `{"mode":"VLLM_COMPILE","cudagraph_mode":"FULL_AND_PIECEWISE","cudagraph_capture_sizes":[6,12,18,24,30,36,48,60,72,96,120,144,192,240,288,384,480,576,768,1020,1536,2046],"decoder_replay_cudagraph_capture_sizes":[6,12,18,24,30,36,48,60,72,96,120,144,192,240,288,384,480,576,768,1020,1536,2048],"max_cudagraph_capture_size":2046}`. Decode uses the `NixlConnector` entry alone with `"kv_role":"kv_consumer"` and `--no-enable-prefix-caching`. It sets `--max-num-batched-tokens` to ten times `--max-num-seqs` and captures FULL decode-only graphs in steps of 6 up to six times `--max-num-seqs`. It balances experts with `--enable-eplb` and `--eplb-config '{"communicator":"symm_mem","num_redundant_experts":16,"window_size":1000,"step_interval":3000,"use_async":true}'`. Launch the sidecars as in **Dynamo sidecar serving**, with `--grpc-startup-deadline-secs 3600`. Start the frontend so that every request prefills on a prefill engine and each session stays on one prefill engine's prefix and host KV cache. On GB300, also set `DYN_ROUTER_DECODE_ACTIVE_REQUEST_WEIGHT=10000000`. ```bash export DYN_ROUTER_TEMPERATURE=0 DYN_ROUTER_QUEUE_THRESHOLD=None DYN_ROUTER_ACTIVE_REQUEST_EXPIRY_SECS=3600 export DYN_TOKENIZER=fastokens DYN_TOKENIZER_CACHE=1 DYN_TOKENIZER_CACHE_BYTES=8000000000 export DYN_TCP_CHANNEL_BUFFER=128 DYN_TCP_REQUEST_TIMEOUT=60 uv run --no-project python -m dynamo.frontend --http-port 8000 --router-mode kv \ --router-session-affinity-ttl-secs 14400 --active-decode-blocks-threshold None \ --active-prefill-tokens-threshold None --active-prefill-tokens-threshold-frac None ``` ## Dynamo sidecar serving Dynamo can route requests to vLLM's native gRPC server through a colocated sidecar. Choose the checkpoint, hardware and parallelism in the command builder above. For compatible runtime installation and complete launch examples, see the [Dynamo vLLM sidecar guide](https://github.com/ai-dynamo/dynamo/blob/main/docs/fern/pages/developer-guide/knowledge-base/modular-components/backends/vllm/sidecar.md). The B300 deployment in [InferenceX #4231](https://github.com/SemiAnalysisAI/InferenceX/pull/4231) validated this integration with the original [DeepSeek-V4.1-Flash checkpoint](https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash). The builder defaults to DeepSeek-V4.1-Flash-1017; that checkpoint was not covered by this deployment's validation. ### Connect the engine and sidecar Use the generated model and hardware arguments with `vllm-rs serve`. Add `--grpc-port 19000` and set `--reasoning-parser None`; omit the engine's `--tool-call-parser` and `--enable-auto-tool-choice` flags because the sidecar handles reasoning and tool parsing. For KV-aware routing, enable KV events on the engine with `--kv-events-config`: ```json {"publisher":"zmq","topic":"kv-events","enable_kv_cache_events":true,"endpoint":"tcp://*:5200"} ``` With etcd running, set its reachable address in each frontend and sidecar shell, along with the shared request and event planes: ```bash export ETCD_ENDPOINTS=http://ETCD_HOST:2379 export DYN_REQUEST_PLANE=tcp DYN_EVENT_PLANE=zmq ``` In a separate shell on the engine host, connect one sidecar to the engine: ```bash export DYN_VLLM_KV_EVENT_PORT=5200 uv run --no-project python -m dynamo.vllm.sidecar \ --grpc-endpoint 127.0.0.1:19000 \ --dyn-tool-call-parser deepseek_v41 --dyn-reasoning-parser deepseek_v41 ``` Start a shared frontend: ```bash uv run --no-project python -m dynamo.frontend --http-port 8000 --router-mode kv ``` For disaggregated serving, configure the engines as in **Dynamo disaggregated serving on B300 and GB300**. Add `--disaggregation-mode prefill --component prefill` to prefill sidecars and `--disaggregation-mode decode` to decode sidecars. Allocate distinct gRPC and KV-event ports when running several engines on the same host. Check the frontend after the workers register: ```bash curl --fail http://FRONTEND_HOST:8000/health curl --fail http://FRONTEND_HOST:8000/v1/chat/completions \ -H 'Content-Type: application/json' \ -d '{"model":"deepseek-ai/DeepSeek-V4.1-Flash","messages":[{"role":"user","content":"Hello"}],"max_tokens":32}' ``` Set the request's `model` to the served model name selected for the engine. ## DGX Station (GB300) Tensor Parallel One GB300 exposes 252 GB of HBM next to 496 GB of Grace LPDDR5X, and the 476 GiB checkpoint does not fit the HBM. The Station row serves TP1 with both Engram tables (189 GiB) and 60 GiB of routed experts in pinned host memory, read through UVA. At `--gpu-memory-utilization 0.97` the KV pool holds about 3.09M tokens at full 1M context. 60 GiB is the lowest verified offload at 1M context with speculative decoding on (55 fails the minimum-KV check). For more concurrent 1M contexts, raise `--cpu-offload-gb` to 70 and accept ~10% lower single-stream decode. Measured on `vllm/vllm-openai:nightly-dee37d89`: | | 1 stream | 2 | 4 | 8 | 12 | 16 | |---|--:|--:|--:|--:|--:|--:| | aggregate tok/s, prose | 90.5 | 124 | 168 | 302 | 378 | 416–423 | - **Autotune.** The first boot on a new flag set or image runs a 75–90 minute FlashInfer autotune. Mount `/root/.cache/vllm` on a persistent volume; later boots start in about 7 minutes. - **Host memory.** With the server loaded, ~430 of the 494 GiB Grace pool is in use (pinned weights plus loader shared memory). `VLLM_WEIGHT_OFFLOADING_DISABLE_PIN_MEMORY=1` frees ~66 GB at about 7% lower single-stream decode. Launch scripts and full measurements: [J-M-Recipes/recipes](https://github.com/J-M-Recipes/recipes/tree/main/recipes/dgx-station-gb300/deepseek-v4.1-flash-vllm-uva-dspark). ## KV cache offloading Offloading requires vLLM 0.31.0+, the first release containing [vllm-project/vllm#57145](https://github.com/vllm-project/vllm/pull/57145). Validation used [deepseek-ai/DeepSeek-V4.1-Flash](https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash) on 4×H200 (TP4), vLLM commit `d1b4028d7e`, the Python frontend, `--enforce-eager`, `--max-model-len 16384`, `--max-num-seqs 16` and `--max-num-batched-tokens 8192`. DeepSeek validation used `--gpu-memory-utilization 0.95`. Validation used `TieringOffloadingSpec`: 8 GiB total CPU tier, `blocks_per_chunk: 1`, a persistent filesystem `root_dir`, eight read/write threads and `PYTHONHASHSEED=0`. Three CPU-tier reloads and three filesystem reloads after restarting the server passed. Reloaded responses matched the cold output, including the reasoning field. The five-shot GSM8K diagnostic scored 31/32; this is a small sample, not the full benchmark. These results cover this text-only configuration, not the builder's larger tier/chunk defaults or other variants. ## References - [Model card](https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash) - [DeepSeek-V4.1-Flash-1017 model card](https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017) - [Technical report](https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash/blob/main/DeepSeek_V41_Tech_Report.pdf) - [DeepSeek-V4-Flash recipe](/deepseek-ai/DeepSeek-V4-Flash) — the V4 sibling this diverges from - [DeepSeek-V3.2-Exp recipe](/deepseek-ai/DeepSeek-V3.2-Exp) — where the indexer first appeared --- # MiniMaxAI/MiniMax-H3 on vLLM Source: https://recipes.vllm.ai/MiniMaxAI/MiniMax-H3 Provider: MiniMax Released: 2026-07-28T10:45:18.000Z Updated: 2026-10-21 Architecture: dense Parameters: 64B Min vLLM: 0.26.0 Tasks: omni Verified hardware: GB200 NVL4, B300, DGX Spark (GB10), RTX 4090, RTX PRO 5000 Blackwell 4x, RTX 5090, RTX Pro 6000 2x, MI300X, MI325X, MI355X, Ascend 950PR Summary: Open-weight general-purpose multimodal generation model — jointly generates 24 FPS video with native stereo audio from text, image, video, and audio references, served via vLLM-Omni ## Variants - default: precision=bf16, vram_minimum_gb=202 BF16 modular service — two 66.3 GB DiTs share the 51.5 GB Qwen3-VL encoder and ~10.6 GB VAEs. Use --task-type fl2va or ref2va on memory-constrained profiles to load one DiT. ## Recommended base flags ```bash vllm serve 'MiniMaxAI/MiniMax-H3' \ --trust-remote-code \ --host \ 0.0.0.0 \ --port \ 8000 \ --num-gpus \ 4 \ --usp \ 4 \ --ring \ 1 \ --vae-patch-parallel-size \ 4 \ --vae-parallel-mode \ tile \ --vae-use-tiling ``` Environment: ```bash VLLM_WORKER_MULTIPROC_METHOD='spawn' VLLM_OMNI_VIDEO_SYNC_TIMEOUT='1800' ``` ## Guide ## Overview MiniMax H3 is an **open-weight, general-purpose multimodal generation model**. Rather than being confined to one specialized task — generate, edit, or reference — H3 reads a multimodal context that mixes text, images, video, and audio together, interprets the creative intent as a whole, and produces coherent audio-visual output end to end. Architecturally it is a **CFG-distilled joint video/audio diffusion transformer** served through vLLM-Omni's OpenAI-compatible `/v1/videos` API. Every request returns a single MP4 containing H.264 video and **native stereo audio** — the audio is generated jointly with the video by the same DiT, not dubbed on afterwards. Three capability areas the model targets: - **Commercial-grade generation** — film and entertainment, advertising and branding, e-commerce, gaming; dynamic typography, VFX, product showcases, UI/UX motion design. - **Native multimodal understanding + generation** — interprets characters, motion, sound, emotion, cinematography, and visual style across mixed references, then combines them. - **Precise multimodal editing and control** — iterative refinement of characters, objects, scenes, sound, and rhythm with strong instruction following. The checkpoint contains two DiT backbones that share the encoder and VAEs: | DiT | Tasks | Conditioning | |---|---|---| | `FL2VA` | `t2va`, `fl2va` | text only, or text + first/last frame | | `Ref2VA` | `ref2va` | text + omni references (images, video clips, audio clips) | Since vLLM-Omni [#5720](https://github.com/vllm-project/vllm-omni/pull/5720), serving `/path/to/MiniMax-H3` initializes both DiTs by default, while the Qwen3-VL encoder and video/audio VAEs stay shared. One server can therefore handle all three task types. Add `--task-type fl2va` or `--task-type ref2va` only when the hardware profile needs to load one DiT to save GPU or host memory. Components (BF16): two 52-block joint video/audio DiTs (66.3 GB each), a shared Qwen3-VL layer-50 text/vision encoder (51.5 GB), a video VAE (~10 GB), and an audio VAE (~0.6 GB). ## Model specifications | | | |---|---| | Output duration | 4–15 seconds | | Frame rate | 24 FPS (fixed) | | Audio | Native stereo, on every generation | | Resolution — 1440p (2K) | Short edge 1440 px for 16:9 … 9:16; wider formats ≈3.7 MP (e.g. 2976×1248 at 21:9) | | Resolution — 768p | Short edge 768 px for 16:9 … 9:16; wider formats ≈1 MP (e.g. 1536×672 at 21:9). Upscalable to 1440p. *(mode coming soon)* | | Aspect ratio | First/Last Frame follows the uploaded image. Text-to-Video and Omni Reference select from 21:9, 16:9, 4:3, 1:1, 3:4, 9:16 — Omni Reference also offers Auto | | Prompt length | Up to 7,000 characters | ### Input modes and limits **First/Last Frame** — 0, 1, or 2 images. Edge lengths in `[256, 5760]`, aspect ratio 5:2 to 2:5. With no image, H3 falls back to Text-to-Video. **Omni Reference** — up to 9 images; up to 3 video clips (2–15 s each, 15 s total); up to 3 audio clips (2–15 s each, 15 s total). Edge lengths in `[256, 5760]`, video aspect ratio 5:2 to 2:5. **Audio must be paired with at least one image or video** — it cannot be the sole reference. Mixed inputs cap at 12 files total; with none, H3 falls back to Text-to-Video. | | Formats | Per-file size | |---|---|---| | Video | H.264/AVC, H.265/HEVC (embedded audio AAC or MP3) | 50 MB | | Image | JPG, JPEG, PNG, WEBP, HEIC, HEIF | 30 MB | | Audio | WAV, MP3 | 15 MB | Limits are per file, not per combined upload. Keep the whole request body under 64 MB — prefer URL-based media over inlined uploads for anything large. ## Prerequisites H3 can be served directly from the gated HF ID or from a local root directory. The local root contains both DiT subdirectories plus the shared encoder and VAEs. Request access on the gated model card, then download the complete repository: ```bash hf download MiniMaxAI/MiniMax-H3 --local-dir /path/to/MiniMax-H3 ``` That yields `/path/to/MiniMax-H3/FL2VA`, `/path/to/MiniMax-H3/Ref2VA`, and the shared component directories. Serve `/path/to/MiniMax-H3` itself and substitute your own `--local-dir` throughout the commands below. `ffmpeg` and `ffprobe` must be on `PATH` — they are used for reference-video preparation and MP4 muxing. ## Installation ### Docker (recommended) The image bundles H3 handlers and media dependencies. The published tag predates the modular pipeline merged in vLLM-Omni #5720, so clone current `main` before starting: ```bash docker pull vllm/vllm-omni:minimax-h3 git clone --depth 1 https://github.com/vllm-project/vllm-omni.git /path/to/vllm-omni ``` The command builder's Docker mode binds both the root checkpoint and source checkout, then prepends the checkout to `PYTHONPATH`: ```bash -v /path/to/MiniMax-H3:/path/to/MiniMax-H3:ro -v /path/to/vllm-omni:/path/to/vllm-omni:ro -e PYTHONPATH=/path/to/vllm-omni ``` ### pip H3 support ships in vLLM-Omni rather than the `vllm` wheel, so the pip path needs a current source checkout: ```bash uv venv source .venv/bin/activate uv pip install vllm==0.26.0 git clone --depth 1 https://github.com/vllm-project/vllm-omni.git /path/to/vllm-omni cd /path/to/vllm-omni uv pip install -e . ``` The generated datacenter Blackwell route uses the default `TRTLLM_ATTN`; RTX PRO 5000 and the two-GPU RTX profiles use `CUDNN_ATTN`. These routes need no FlashAttention-4 install. Install the optional extra only when explicitly comparing FA4: `uv pip install -e '.[fa4]'`. ## Launch — two 24/32 GB GPUs with distributed layerwise offload vLLM-Omni includes a low-memory TP2 path for H3. Install the current vLLM-Omni source checkout as shown above. The two-GPU profile combines TP2 with distributed layerwise offload (DLO). Each rank keeps its rank-local TP shard in pinned host memory; `--dlo-no-use-allgather` streams those shards directly instead of reconstructing full DiT blocks on every rank. A configurable prefix of DiT blocks stays resident for the complete denoise stage, then releases its HBM before VAE decode. Select the profile first. RTX 5090 uses 20 resident DiT layers at the released 1344×768 shape; RTX 4090 keeps more HBM headroom with 12 resident layers and a 1024×576 starting shape: ```bash # 2× RTX 5090 (32 GB) export DLO_RESIDENT_LAYERS=20 WIDTH=1344 HEIGHT=768 # Or, 2× RTX 4090 (24 GB) # export DLO_RESIDENT_LAYERS=12 WIDTH=1024 HEIGHT=576 ``` Load only the FL2VA DiT on these memory-constrained profiles: ```bash CUDA_VISIBLE_DEVICES=0,1 \ VLLM_WORKER_MULTIPROC_METHOD=spawn \ VLLM_OMNI_VIDEO_SYNC_TIMEOUT=14400 \ vllm serve /path/to/MiniMax-H3 \ --omni \ --task-type fl2va \ --host 0.0.0.0 \ --port 8000 \ --trust-remote-code \ --num-gpus 2 \ --tensor-parallel-size 2 \ --usp 1 \ --ring 1 \ --text-encoder-tp-size 2 \ --vae-patch-parallel-size 2 \ --vae-parallel-mode tile \ --vae-use-tiling \ --enable-distributed-layerwise-offload \ --dlo-no-use-allgather \ --dlo-resident-layers "${DLO_RESIDENT_LAYERS}" \ --enforce-eager \ --diffusion-attention-backend CUDNN_ATTN ``` The resident count changes placement and transfer frequency only; it does not quantize or change the BF16/FP32 denoise math. Use a **384 GiB-class host** for the pinned model shards and offload buffers. Wait for `Application startup complete`, then verify health and run one full 50-step T2VA request: ```bash curl --fail http://127.0.0.1:8000/health curl --fail-with-body -sS \ -X POST http://127.0.0.1:8000/v1/videos/sync \ -F 'prompt=At night, three cats march into a bedroom playing tiny brass instruments, then abruptly file out, with synchronized room ambience.' \ -F "width=${WIDTH}" \ -F "height=${HEIGHT}" \ -F 'fps=24' \ -F 'num_inference_steps=50' \ -F 'flow_shift=12' \ -F 'seed=1101' \ -F 'extra_params={"task":"t2va","duration":5.0,"audio_flow_shift":3.0}' \ -o minimax-h3-t2va.mp4 ffprobe -v error -show_entries \ stream=index,codec_name,width,height,r_frame_rate,sample_rate,channels \ -of json minimax-h3-t2va.mp4 ffmpeg -v error -i minimax-h3-t2va.mp4 \ -map 0:v:0 -map 0:a:0 -f null - ``` **Note:** If using 2× RTX 4090, the curl request must include -F 'aspect_ratio=16:9' to succeed. A successful output has H.264 video at 24 FPS and stereo AAC audio at 32 kHz, and the final `ffmpeg` command exits with status 0. This capacity profile intentionally loads one DiT. Restart the same root-model command with `--task-type ref2va` for Ref2VA; remove `--task-type` only after confirming that the combined modular service fits the target. The same root path is used for both task families; only `--task-type` changes on these capacity profiles. ## Launch — four GPUs (validated best practice) On B300/GB200, the no-offload root service uses Ulysses SP4, tiled VAE patch parallelism 4, regional `torch.compile`, dense BF16 `TRTLLM_ATTN`, and Ring/TP1. Both DiTs load once while the encoder and VAEs remain shared. ```bash CUDA_VISIBLE_DEVICES=0,1,2,3 \ FLASHINFER_DISABLE_VERSION_CHECK=1 \ VLLM_WORKER_MULTIPROC_METHOD=spawn \ VLLM_OMNI_VIDEO_SYNC_TIMEOUT=1800 \ vllm serve /path/to/MiniMax-H3 \ --omni \ --host 0.0.0.0 \ --port 8000 \ --trust-remote-code \ --num-gpus 4 \ --usp 4 \ --ring 1 \ --vae-patch-parallel-size 4 \ --vae-parallel-mode tile \ --vae-use-tiling ``` Omitting the attention flag is intentional: current datacenter Blackwell builds default to dense BF16 `TRTLLM_ATTN`. Install `[fa4]` and add `--diffusion-attention-backend FLASH_ATTN` only for an explicit FA4 comparison. ### Optional lossy attention: SAGE + Skip-Softmax `TRTLLM_ATTN` offers two **lossy** optimizations for the long main DiT attention sequence. Both work under the pure Ulysses parallelism of the four-GPU profile (`--usp 4 --ring 1`) and are datacenter-Blackwell only. Append the following to the four-GPU command above to enable both: ```bash --diffusion-attention-config '{ "default": { "backend": "TRTLLM_ATTN", "quant": {"dtype_qk": "fp8_e4m3", "q_block_size": 1, "k_block_size": 16}, "skip_softmax": {"threshold": 0.05, "disabled_until_timestep": 0.97} }, "per_role": { "minimax_h3.token_refiner": {"backend": "TRTLLM_ATTN"} } }' ``` - **SAGE** quantizes Q/K to `fp8_e4m3` for `QK^T`; P and V are always FP8 in this kernel. B200 additionally supports `int8` Q/K, which preserves accuracy better than FP8. - **Skip-Softmax** skips the Softmax and `PV` work of KV tiles whose scores fall far below the running row maximum. `threshold=0.05` with `disabled_until_timestep=0.97` is a **conservative** setting: a low threshold skips only clearly negligible tiles, and the cutoff keeps the early high-noise steps dense — H3's default video flow shift of 12 keeps the normalized timestep above `0.97` for the first 14 of 49 denoiser forwards at 50 steps. Raise `threshold` or lower `disabled_until_timestep` for more speedup once quality is verified. - The `per_role` entry keeps the short token-refiner attention dense; a per-role spec does not inherit `quant` or `skip_softmax` from `default`. Both optimizations trade fidelity for speed, and their effects compound. The measured B300 speedup and LPIPS for each mode and for the combination are in the vLLM blog's [Quantized and Sparse Attention](https://vllm.ai/blog/2026-09-01-minimax-h3-production-serving#quantized-and-sparse-attention-in-trtllm_attn) section as a reference; tune the settings according to the accuracy and speed required. See the [TRTLLM attention guide](https://docs.vllm.ai/projects/vllm-omni/en/latest/user_guide/diffusion/attention_backends/trtllm/) for the full configuration reference. No restart is required: `extra_params.task` routes T2VA/FL2VA requests to the FL2VA DiT and Ref2VA requests to the Ref2VA DiT. ### FastH3 model family FastH3 is a family of post-trained models based on MiniMax-H3. It generates video with stereo audio in four or eight steps. Choose a starting point for your input and hardware: | Input | Hardware | Model | |---|---|---| | Prompt and reference image | RTX 5090, DGX Spark, RTX PRO 6000 Blackwell | [Omni NVFP4 QAD](https://huggingface.co/FastVideo/FastH3-Omni-NVFP4-QAD), single-card guide below | | Prompt and reference image | RTX 4090 | [Omni BF16](https://huggingface.co/FastVideo/FastVideo-FastH3-Omni-8-Step-V1) with FP8 enabled, single-card guide below | | Prompt and reference image | Eight GPUs | Omni BF16, eight-GPU guide below | | Prompt only | Eight GPUs | [V2](https://huggingface.co/FastVideo/FastVideo-FastH3-8-Step-V2) (experimental) or [four-step adapter](https://huggingface.co/FastVideo/FastVideo-FastH3-4-step-Preview-v1-LoRA) | The command builder above serves native MiniMax-H3. Use the FastH3 setup and commands below for these models. Select one model when starting the server. The single-card profiles are configuration guidance; this documentation change does not independently qualify them with GPU inference. #### Install FastH3 Use Linux, Python 3.12, a CUDA 13-compatible NVIDIA driver, and [uv](https://docs.astral.sh/uv/getting-started/installation/). The eight-step models currently need the source branch from [vLLM-Omni #9167](https://github.com/vllm-project/vllm-omni/pull/9167): ```bash sudo apt-get update sudo apt-get install -y git ffmpeg git clone --branch pr/fasth3-omni-upstream https://github.com/lishunyang12/vllm-omni.git vllm-omni-fasth3 cd vllm-omni-fasth3 uv venv --python 3.12 source .venv/bin/activate uv pip install vllm==0.32.0 --torch-backend=cu130 uv pip install -e . ``` On DGX Spark, run in the native `aarch64` environment. If Hugging Face requests access, accept the model's access terms and run `hf auth login`. Run the following server commands from the `vllm-omni-fasth3` directory. #### FastH3 RefOmni Single Card **RTX 5090 (32 GB), DGX Spark (128 GB), or RTX PRO 6000 Blackwell (96 GB):** ```bash CUDA_VISIBLE_DEVICES=0 VLLM_WORKER_MULTIPROC_METHOD=spawn \ VLLM_OMNI_VIDEO_SYNC_TIMEOUT=7200 \ vllm serve FastVideo/FastH3-Omni-NVFP4-QAD --omni \ --config recipes/FastVideo/fasth3-omni-quantized.yaml ``` This uses the smaller NVFP4 model and fast 8-layer video decoder. Layer offload reduces GPU memory use. **RTX 4090 (24 GB):** ```bash uv pip install -e '.[vsa]' CUDA_VISIBLE_DEVICES=0 VLLM_WORKER_MULTIPROC_METHOD=spawn \ VLLM_OMNI_VIDEO_SYNC_TIMEOUT=7200 \ vllm serve FastVideo/FastVideo-FastH3-Omni-8-Step-V1 --omni \ --config recipes/FastVideo/fasth3-omni-rtx4090.yaml ``` The first start downloads the model; the 4090 also converts eligible weights to FP8 while loading. Later starts reuse the download cache. Keep the terminal open and wait for `Application startup complete`. **Generate.** Save your image as `reference.png`. In a second terminal: ```bash curl --fail-with-body -sS http://localhost:8000/v1/videos/sync \ -F 'prompt=The person in the reference image smiles and waves at the camera. Birds chirp as the person says, "Welcome to the garden!"' \ -F 'input_reference=@reference.png' \ -F 'size=832x480' -F 'seconds=5' -F 'seed=1234' \ -o output.mp4 ``` Open `output.mp4` to play the video with stereo audio. For 768P, change `size=832x480` to `size=1344x768`. If memory is tight, start at 480P and close other GPU workloads. For multiple images, video, or audio references, see the [H3 reference API](https://github.com/lishunyang12/vllm-omni/blob/pr/fasth3-omni-upstream/recipes/MiniMaxAI/MiniMax-H3.md). **Optional settings:** - **FP8 model:** if you have access to [FastH3-Omni-FP8](https://huggingface.co/FastVideo/FastH3-Omni-FP8), replace the model ID in the Blackwell command with `FastVideo/FastH3-Omni-FP8`. A local model directory also works. This release also requires Blackwell because its text encoder uses NVFP4. - **36-layer decoder:** in `recipes/FastVideo/fasth3-omni-quantized.yaml`, change `additional-config.video_vae_decoder.subfolder` from `fast-8L` to `quality-36L`, then restart. Both decoders come from [FastH3-NVFP4-VAE-Decoder](https://huggingface.co/FastVideo/FastH3-NVFP4-VAE-Decoder). - **DGX Spark:** remove `diffusion-offload-config` from the Blackwell YAML to keep the model resident in unified memory and avoid offload transfers. #### FastH3 Omni on eight GPUs Install the attention kernels, then start the server: ```bash uv pip install -e '.[cudnn-bsa]' VLLM_WORKER_MULTIPROC_METHOD=spawn VLLM_OMNI_VIDEO_SYNC_TIMEOUT=7200 \ vllm serve FastVideo/FastVideo-FastH3-Omni-8-Step-V1 --omni \ --config recipes/FastVideo/fasth3-8step-8gpu.yaml ``` Wait for `Application startup complete`, then use the image-and-prompt request above. The configuration sets eight inference steps and handles GPU placement. For an NVFP4 video decoder on Blackwell, add this to the deployment YAML: ```yaml additional-config: video_vae_decoder: model: FastVideo/FastH3-NVFP4-VAE-Decoder subfolder: fast-8L ``` Use `quality-36L` for the deeper decoder, then restart the server. **Performance reference: eight RTX PRO 5000 Blackwell 72 GB GPUs.** BF16, 1344×768, one request at a time; timing includes MP4 encoding and excludes model loading: | Output | Generation time | Reported allocator peak | |---|---:|---:| | 124 frames, about 5 seconds | 41.04 s, median of 3 runs | 35.15 GiB | | 345 frames, about 14 seconds | 95.93 s, 1 run | 38.94 GiB | These are offline measurements. See the [benchmark details](https://github.com/lishunyang12/vllm-omni/pull/63) for the measured environment and configuration. #### FastH3 V2: text to video and audio This eight-GPU deployment is experimental. Install the `cudnn-bsa` extra above, then start V2 in place of the Omni server: ```bash VLLM_WORKER_MULTIPROC_METHOD=spawn VLLM_OMNI_VIDEO_SYNC_TIMEOUT=7200 \ vllm serve FastVideo/FastVideo-FastH3-8-Step-V2 --omni \ --config recipes/FastVideo/fasth3-8step-8gpu.yaml ``` Wait for startup, then send a text-only request in a second terminal: ```bash curl --fail-with-body -sS http://localhost:8000/v1/videos/sync \ -F 'prompt=A train crosses a stone bridge at sunrise, with rhythmic wheels and distant birdsong.' \ -F 'size=1344x768' -F 'seconds=5' \ -o fasth3-v2.mp4 ``` Keep the model's default eight steps and sampling settings. ### FastH3 four-step Dense and VSA adapters Use the four-step adapter for text-to-video generation on eight GPUs. Choose the adapter at startup; per-request switching and CPU/layer offload are unsupported. Download the Dense and VSA adapters: ```bash hf download FastVideo/FastVideo-FastH3-4-step-Preview-v1-LoRA \ dense-datafree/adapter_model.safetensors \ vsa-datafree/adapter_model.safetensors \ --local-dir ./FastH3-LoRA ``` Start with the Dense adapter: ```bash CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \ VLLM_WORKER_MULTIPROC_METHOD=spawn VLLM_OMNI_VIDEO_SYNC_TIMEOUT=1800 \ vllm serve MiniMaxAI/MiniMax-H3 --omni \ --host 127.0.0.1 --port 8095 --trust-remote-code \ --task-type fl2va --served-model-name FastH3 \ --num-gpus 8 --tensor-parallel-size 1 --usp 8 --ring 1 \ --vae-patch-parallel-size 8 --vae-parallel-mode tile --vae-use-tiling \ --diffusion-attention-config.default.backend FLASH_ATTN \ --lora-path ./FastH3-LoRA/dense-datafree/adapter_model.safetensors ``` For the VSA adapter, install `uv pip install -e '.[vsa]'`, change `--lora-path` to `./FastH3-LoRA/vsa-datafree/adapter_model.safetensors`, and replace the attention option with: ```bash --diffusion-attention-config.default.backend FASTVIDEO_VSA \ --diffusion-attention-config.default.fastvideo_vsa_topk 64 ``` Keep pure Ulysses parallelism (`--usp 8 --ring 1`) for VSA. See [backend selection](https://github.com/lishunyang12/vllm-omni/blob/pr/fasth3-omni-upstream/recipes/FastVideo/FastH3.md#hardware-and-backend-selection) for hardware requirements. After the server starts, generate a video in a second terminal: ```bash curl --fail-with-body -sS http://127.0.0.1:8095/v1/videos/sync \ -F 'model=FastH3' \ -F 'prompt=A cinematic starship glides through a luminous nebula while its engines hum and distant radio chatter fills the cabin.' \ -F 'width=1344' -F 'height=768' -F 'aspect_ratio=16:9' -F 'fps=24' \ -F 'num_inference_steps=4' -F 'guidance_scale=1' -F 'seed=1101' \ -F 'extra_params={"task":"t2va","duration":10.0}' \ -o fasth3.mp4 ``` Use four steps and `task=t2va`; keep the default sampling settings. **Performance reference: eight B300 GPUs.** At 1344×768 and 24 FPS, server generation time including MP4 encoding was: | Output duration | Dense (TRTLLM) | VSA | |---|---:|---:| | 10 seconds | 9.84 s | 7.28 s | | 15 seconds | 14.20 s | 10.80 s | See [benchmark details](https://github.com/vllm-project/vllm-omni/pull/6909) for the measured environment and configuration. ### LightX2V Turbo few-step LoRA [LightX2V Turbo](https://huggingface.co/lightx2v/Minimax-h3-Turbo) is a family of few-step LoRA artifacts for H3. It reaches the same goal as FastH3 by the opposite route: instead of fusing full-rank deltas at startup, Turbo stays a standard LoRA that each request activates by name. One server can therefore hold several artifacts and let requests choose between them, at the cost of the LoRA matmuls on every denoise step. Eight Diffusers-layout artifacts are supported. The filename is the contract — the server parses the task family, step count and resolution out of it, so a renamed file is rejected rather than served on a guess. `audio_flow_shift` is `3.0` across the family. | Artifact | Tasks | `num_inference_steps` | `flow_shift` | alpha | |---|---|---:|---:|---:| | `minimax_h3_fl2v_turbo_4step_v0.1.safetensors` | T2VA / FL2VA | 5 | 12 | none → 8 | | `minimax_h3_fl2v_turbo_4step_v1.0_768p_bf16.safetensors` | T2VA / FL2VA | 5 | 6 | 128 | | `minimax_h3_fl2v_turbo_4step_v1.1_768p_bf16.safetensors` | T2VA / FL2VA | 5 | 6 | 128 | | `minimax_h3_fl2v_turbo_4step_v1.2_768p_bf16.safetensors` | T2VA / FL2VA | 5 | 6 | 8 | | `minimax_h3_fl2v_turbo_8step_v1.0_bf16.safetensors` | T2VA / FL2VA | 9 | 12 | 8 | | `minimax_h3_fl2v_turbo_8step_v1.0_768p_bf16.safetensors` | T2VA / FL2VA | 9 | 6 | 8 | | `minimax_h3_ref2v_turbo_4step_v0.1_bf16.safetensors` | Ref2VA | 5 | 12 | 8 | | `minimax_h3_ref2v_turbo_8step_v1.0_768p_bf16.safetensors` | Ref2VA | 9 | 6 | 8 | Requested sigma points are always one more than the artifact's forwards, which is why a four-step Turbo artifact takes `num_inference_steps=5` where FastH3 takes `4`. Every artifact is rank 128 and the delta is applied at `scale * alpha / 128`, with alpha read from the artifact's metadata — only `4step_v0.1` declares none and falls back to 8 with a warning. Download one artifact under the H3 root so the existing bind mount includes it. Each artifact except `4step_v0.1` also ships a `_comfyui_` export that fuses Q/K/V into a single projection; those are **not** supported and are refused by name. ```bash hf download lightx2v/Minimax-h3-Turbo \ minimax_h3_fl2v_turbo_4step_v1.2_768p_bf16.safetensors \ --local-dir /path/to/MiniMax-H3/Turbo-LoRA ``` Serve it on the same eight-GPU pure-Ulysses topology. `--lora-path` preloads the adapter; it accepts one artifact, or a directory holding exactly one — a directory with several recognized artifacts is rejected as ambiguous. ```bash CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \ VLLM_WORKER_MULTIPROC_METHOD=spawn \ VLLM_OMNI_VIDEO_SYNC_TIMEOUT=1800 \ vllm serve /path/to/MiniMax-H3 \ --omni \ --host 127.0.0.1 \ --port 8095 \ --trust-remote-code \ --task-type fl2va \ --served-model-name MiniMaxAI/MiniMax-H3 \ --num-gpus 8 \ --usp 8 \ --ring 1 \ --vae-patch-parallel-size 8 \ --vae-parallel-mode tile \ --vae-use-tiling \ --diffusion-attention-backend TRTLLM_ATTN \ --lora-backend peft \ --lora-path /path/to/MiniMax-H3/Turbo-LoRA/minimax_h3_fl2v_turbo_4step_v1.2_768p_bf16.safetensors ``` Unlike FastH3, the request must activate the adapter and carry that row's sampling contract. A request whose steps or shift do not match the loaded artifact is rejected, so a mismatch cannot silently sample the distilled model at noise levels it never saw. ```bash curl --fail-with-body -sS -X POST http://127.0.0.1:8095/v1/videos/sync \ -F 'prompt=A cinematic starship glides through a luminous nebula while its engines hum and distant radio chatter fills the cabin.' \ -F 'width=1344' \ -F 'height=768' \ -F 'aspect_ratio=16:9' \ -F 'fps=24' \ -F 'num_inference_steps=5' \ -F 'flow_shift=6' \ -F 'seed=1101' \ -F 'extra_params={"task":"t2va","duration":10.0,"audio_flow_shift":3.0}' \ -F 'lora={"name":"h3-turbo-v1.2","path":"/path/to/MiniMax-H3/Turbo-LoRA/minimax_h3_fl2v_turbo_4step_v1.2_768p_bf16.safetensors","scale":1.0}' \ -o turbo.mp4 ``` FL2VA artifacts serve `t2va` and `fl2va` on an FL2VA or combined server. The two `ref2v` rows need `--task-type ref2va`: a combined server answers `ref2va` from a second DiT the adapter cannot bind to, so loading one there is refused rather than running an undistilled model on a few-step ladder. The integration is dynamic-only — no prefusion, no LoRA composition. DLO works by keeping the request-switchable LoRA A/B buffers resident on the accelerator while DLO streams the base blocks; model-level and standard layerwise offload remain unsupported. Measured on 8×B300 with the command above, at 1344×768, 24 FPS, a 15 s FL2VA request with first and last keyframes (362 frames), fixed seed, one discarded warmup and three recorded runs. The baseline is the same server without an adapter at `num_inference_steps=50` and `flow_shift=12`: | Metric | Base H3, 49 forwards | Turbo v1.2, 4 forwards | Speedup | |---|---:|---:|---:| | Denoise loop | ~105 s | ~9 s | 11.7× | | Per denoise step | 2.14–2.16 s/it | 2.20–2.42 s/it | — | | Diffusion engine | 111.1–111.3 s | 15.0–15.1 s | 7.4× | | Server E2E incl. MP4 | 112.5–112.9 s | ~16.5 s | 6.8× | The ~10% higher per-step cost is the LoRA matmul the fused FastH3 route does not pay — irrelevant against a 12× cut in forwards, but the reason a single-artifact deployment can still prefer a fused student. These figures are for a server serving Turbo requests back to back. Alternating between Turbo and adapter-free requests on one server additionally pays a LoRA rebind on each switch — measured at about 24 s per switch in the same run (39.2 s diffusion engine against 15.1 s steady state), because every one of H3's 208 LoRA layers is torn down and re-uploaded. Give Turbo its own process if the workload mixes, or track vLLM-Omni #7195, which removes that rebind. ### RTX PRO 5000 Blackwell — four PCIe GPUs Select the `RTX PRO 5000 Blackwell 4x` hardware profile for the validated BF16, no-offload topology: TP2 × Ulysses2, Ring1, text-encoder TP4, VAE patch parallel 4 in tile mode, and `CUDNN_ATTN`. On the measured dual-socket host, `CUDA_VISIBLE_DEVICES=0,2,1,3` kept the TP pairs on local PXB paths; check `nvidia-smi topo -m` and adapt the order on other systems. This 72 GiB/GPU profile serves the root path but selects one DiT with `--task-type`. The task picker adds `fl2va` for T2VA/FL2VA and `ref2va` for Ref2VA; loading both DiTs together is not qualified on this memory-constrained profile. A five-step topology screen after warmup produced decodable H.264 video with stereo AAC: | Workload | Client E2E | Encode | Denoise | VAE decode | Peak/GPU | |---|---:|---:|---:|---:|---:| | T2VA, 124 frames, 1344×768 | 32.993 s | 0.034 s | 27.915 s | 4.307 s | 67.6 GiB | | FL2VA first frame, same output shape | 30.130 s | 0.276 s | 24.396 s | 4.339 s | 67.6 GiB | These are topology-screening results, not 50-step throughput claims. Nsight Systems aggregation for the same run was: | Workload | AllGather | Send/Recv | NCCL other | NCCL total | Dense FMHA | max-min/mean | |---|---:|---:|---:|---:|---:|---:| | T2VA | 0.93% | 8.53% | 14.63% | 24.09% | 31.15% | 1.11% | | FL2VA | 0.89% | 7.26% | 15.76% | 23.91% | 32.53% | 0.13% | `NCCL other` covers NCCL kernels outside recognized AllGather and point-to-point Send/Recv, including AllReduce, ReduceScatter, or Broadcast when present. ### AMD ROCm — four MI300X GPUs Verified on 4× AMD Instinct MI300X (gfx942) with the ROCm H3 image, AITER packed variable-length attention, Ulysses 4, text-encoder TP4, and native tiled VAE patch parallelism 4: ```bash docker pull vllm/vllm-omni-rocm:minimax-h3 HIP_VISIBLE_DEVICES=0,1,2,3 \ VLLM_ROCM_USE_AITER=1 \ VLLM_WORKER_MULTIPROC_METHOD=spawn \ VLLM_OMNI_VIDEO_SYNC_TIMEOUT=1800 \ vllm serve /path/to/MiniMax-H3 \ --omni \ --host 0.0.0.0 \ --port 8000 \ --trust-remote-code \ --num-gpus 4 \ --usp 4 \ --ring 1 \ --text-encoder-tp-size 4 \ --vae-patch-parallel-size 4 \ --vae-parallel-mode tile \ --vae-use-tiling \ --diffusion-attention-backend FLASH_ATTN ``` On ROCm, `FLASH_ATTN` resolves to AITER on supported Instinct architectures. Do not install the CUDA-only `[fa4]` extra or set `FLASHINFER_DISABLE_VERSION_CHECK`. The refreshed `vllm/vllm-omni-rocm:minimax-h3` tag includes the Ref2VA soundfile fallback, TorchCodec, and FFmpeg, so T2VA, FL2VA, and the complete Ref2VA path use the same official image. ### AMD ROCm — four MI355X GPUs Verified on 4× AMD Instinct MI355X. The official ROCm image includes the MiniMax H3 Ref2VA fixes, TorchCodec, and FFmpeg. We tested both Ulysses 4 with Ring 1 and Ulysses 1 with Ring 4. The command below shows the Ulysses 4 / Ring 1 configuration with tiled VAE patch parallelism degree 4, AITER, and CPU offload: ```bash docker pull vllm/vllm-omni-rocm:minimax-h3 HIP_VISIBLE_DEVICES=0,1,2,3 \ VLLM_ROCM_USE_AITER=1 \ VLLM_WORKER_MULTIPROC_METHOD=spawn \ VLLM_OMNI_VIDEO_SYNC_TIMEOUT=1800 \ vllm serve /path/to/MiniMax-H3 \ --omni \ --host 0.0.0.0 \ --port 8000 \ --trust-remote-code \ --num-gpus 4 \ --usp 4 \ --ring 1 \ --vae-patch-parallel-size 4 \ --vae-parallel-mode tile \ --vae-use-tiling \ --enable-cpu-offload \ --text-encoder-tp-size 4 ``` This root-model command accepts T2VA, FL2VA, and Ref2VA requests without restarting. Three constraints on this configuration: - **Do not add `--enforce-eager`.** Regional compile is the default (`--diffusion-compile-granularity regional`) and is worth ~9.5%. The first request includes compilation — warm the server once before measuring steady state. - **`--cfg-parallel-size` must stay 1.** H3 is CFG-distilled and has no negative branch; anything greater is rejected with an explicit error rather than silently duplicating a branch that does not exist. - **The VAE supports its native `tile` mode only** — not `spatial_shard_width` or `spatial_shard_height` (those belong to the distributed Wan autoencoder). Patch parallel size must be 1 or the full DiT group size. ## Launch — single GPU (accuracy and memory first) Model-level CPU offload keeps the Qwen3-VL encoder and the DiT from being resident simultaneously. This is the accuracy-qualified reference path; it trades PCIe/NVLink transfer latency for a much smaller GPU footprint, and needs enough system RAM for the offloaded components. ```bash CUDA_VISIBLE_DEVICES=0 \ VLLM_WORKER_MULTIPROC_METHOD=spawn \ VLLM_OMNI_VIDEO_SYNC_TIMEOUT=1800 \ vllm serve /path/to/MiniMax-H3 \ --omni \ --task-type fl2va \ --host 0.0.0.0 \ --port 8000 \ --trust-remote-code \ --num-gpus 1 \ --enable-cpu-offload \ --diffusion-attention-backend FLASH_ATTN ``` ## Launch — DGX Spark (GB10, 128 GB unified memory) GB10 is a **unified-memory** platform: the 128 GB LPDDR5X (≈121 GiB usable) is shared by CPU and GPU, so both offload strategies used elsewhere in this recipe stop being capacity levers. `--enable-cpu-offload` and distributed layerwise offload move weights from "VRAM" to "host RAM", which is the same physical pool here — nothing is freed. DLO is actively harmful: staging rank-local shards into pinned host memory creates a second copy alongside the resident one, and the process is killed by the OOM killer (`Exit code: -9`) right after `Enabling offloader backend`. One partition on disk is 135 GiB in BF16: | Component | Size | Quantized by `--quantization fp8` | |---|---:|---| | `text_encoder` (Qwen3-VL) | 63 GiB | no | | `transformer` (DiT) | 62 GiB | yes | | `video_vae` | 9.8 GiB | no | | `audio_vae` | 0.6 GiB | no | 135 GiB does not fit in 121 GiB, so **FP8 is mandatory** and only one DiT can be resident — always pass `--task-type fl2va` or `--task-type ref2va`. Online FP8 touches the DiT only, bringing the partition to roughly 104 GiB before activations. Weight quantization runs at load time and is slow on this platform; `--init-timeout 3600` keeps startup from being cut short. ```bash CUDA_VISIBLE_DEVICES=0 \ FLASHINFER_DISABLE_VERSION_CHECK=1 \ VLLM_WORKER_MULTIPROC_METHOD=spawn \ VLLM_OMNI_VIDEO_SYNC_TIMEOUT=7200 \ vllm serve /path/to/MiniMax-H3 \ --omni \ --task-type fl2va \ --host 0.0.0.0 \ --port 8000 \ --trust-remote-code \ --init-timeout 3600 \ --num-gpus 1 \ --usp 1 \ --ring 1 \ --quantization fp8 \ --vae-patch-parallel-size 1 \ --vae-parallel-mode tile \ --vae-use-tiling \ --enforce-eager \ --diffusion-attention-backend CUDNN_ATTN ``` Restart with `--task-type ref2va` to serve the Ref2VA family. Keep requests at a 576 px short edge (1024×576) and 5 s duration as the starting shape; GB10 has one GPU and no parallelism headroom, so wall-clock per request is far higher than the multi-GPU profiles above. ## Launch — single Ascend 950PR NPU (INT8) vLLM-Omni serves H3 on Ascend through the `vllm-ascend` platform plugin, on a CPU-only aarch64 PyTorch build with `torch_npu` — no CUDA runtime is present. One 950PR holds a single DiT partition, so this profile serves the `FL2VA` directory and uses online INT8 in place of the sharding levers the multi-GPU profiles rely on. Three Ascend-specific constraints come before the command: - **Attention backend.** `CUDNN_ATTN`, `FLASHINFER_ATTN`, `TRTLLM_ATTN`, and the SageAttention backends are CUDA-only; selecting one on NPU fails with a `NotImplementedError` out of `forward_npu`. The NPU platform default resolves to `FLASH_ATTN` when MindIE-SD is installed and `TORCH_SDPA` otherwise, and `FLASH_ATTN` is where H3's NPU memory work lives: a mask-free packed varlen path that never materializes the quadratic padding mask, plus K/V prefix slicing. Install MindIE-SD *before* passing `--diffusion-attention-backend FLASH_ATTN` — the backend imports `mindiesd` lazily and otherwise raises `ImportError` at the first denoise step. Once installed the platform default picks it up on its own. - **Shared memory.** Decoded video crosses the diffusion-worker process boundary through POSIX shared memory — roughly 0.9 GiB for a 5 s 1024×576 clip. The 64 MiB Docker default is large enough for the control-plane queues, so the server starts and the denoise loop reaches 100% before the write into `/dev/shm` raises SIGBUS and kills the worker with `Bus error (core dumped)`. Check `df -h /dev/shm` and start the container with `--shm-size=8g` or `--ipc=host`. It cannot be changed on a running container, and an in-container `mount -o remount` needs `CAP_SYS_ADMIN`. - **No `--enforce-eager`.** It disables regional compilation and only slows the steady state; the first request is a warmup either way. `--cfg-parallel-size` must stay 1, as everywhere else in this recipe. ```bash ASCEND_RT_VISIBLE_DEVICES=0 \ VLLM_WORKER_MULTIPROC_METHOD=spawn \ VLLM_OMNI_VIDEO_SYNC_TIMEOUT=14400 \ vllm serve /path/to/MiniMax-H3/FL2VA \ --omni \ --host 0.0.0.0 \ --port 8000 \ --trust-remote-code \ --num-gpus 1 \ --tensor-parallel-size 1 \ --usp 1 \ --ring 1 \ --text-encoder-tp-size 1 \ --vae-patch-parallel-size 1 \ --vae-parallel-mode tile \ --vae-use-tiling \ --init-timeout 14400 \ --stage-init-timeout 14400 \ --quantization int8 ``` ## Text-encoder tensor parallelism By default the whole Qwen3-VL encoder is resident on the DiT main rank, making it the peak-memory hotspot on multi-GPU no-offload runs. `--text-encoder-tp-size N` shards it across the first `N` DiT ranks using vLLM-style tensor-parallel layers on a dedicated encoder process group: ```bash vllm serve /path/to/MiniMax-H3 \ --omni --trust-remote-code \ --task-type fl2va \ --num-gpus 4 --usp 4 --ring 1 \ --text-encoder-tp-size 4 \ --vae-patch-parallel-size 4 --vae-parallel-mode tile --vae-use-tiling ``` - `N` must divide the Qwen3-VL head counts (64 attention heads / 8 KV heads) → 1, 2, 4 on a 4-GPU server; 1, 2, 4, 8 on 8 GPUs. - On 4 GPUs, `N=4` drops the per-rank engine peak from **133 GB to 103 GB** (~35.7 GB) at no measurable throughput cost, while each non-main rank gains ~51.5/N GB. - Numerically: `N=1` reproduces the HF reference path bit-exactly (`max_abs = 0`); `N=4` introduces bounded BF16 rounding only (row-parallel all-reduce in FP32), giving 31.11 dB PSNR / 0.9566 SSIM end-to-end against `N=1` with identical structure. - Combining `N>1` with regional compile used to produce a `ValueError: v must be finite` on the *second* request. That is fixed — the token refiner's attention runs on replicated rows before `sp_prepare` and was wrongly getting a Ulysses SP all-to-all. ## Key request parameters | Parameter | Recommended | Notes | |---|---|---| | `task` | `t2va`, `fl2va`, `ref2va` | Passed inside `extra_params`; must be compatible with `--task-type` when one is set | | `duration` | 4–15 s | Decimal seconds in `extra_params`; snapped to H3's legal `17n+5` frame count (15 s → 362 frames / 15.083 s) | | `fps` | `24` | Output FPS is fixed | | `num_inference_steps` | `50` | Matches the reference accuracy workloads | | `flow_shift` | `12` | Video sigma shift | | `audio_flow_shift` | `3` | Audio sigma shift, in `extra_params` | | `width`, `height` | see below | Server-side validation: multiples of 32, aspect ratio 1:4 to 4:1 | | `seed` | fixed value | Output is deterministic per request at a fixed seed | On `width`/`height`, the serving-side check is broader than the documented product modes — pick a **short edge of 768 or 1440** in one of the supported ratios (21:9, 16:9, 4:3, 1:1, 3:4, 9:16) to stay on shapes the model was trained and validated for. FL2VA can omit both and inherit the first frame's aspect ratio at a 768 px short edge. Prompts cap at 7,000 characters. Use `POST /v1/videos/sync` when you want the MP4 in the response body; `POST /v1/videos` is the async job-polling variant. ## Validated performance (4×B300) These measurements used task-specific single-DiT processes before the modular combined-service merge; they remain the validated per-task operating points. | Workload | Config | Result | |---|---|---| | FL2VA, 209 frames, 1248×768 (8.7 s) | no offload, U4, VPP4 tile, regional compile | **86.96 s** client E2E (±0.23 s) | | Two-video Ref2VA, 362 frames, 1344×768 (15 s) | same | **784.4 s** accounted model-stage mean | Stage split for the FL2VA case: text encoder 0.21 s, visual encoder 0.20 s, DiT 79.1 s (**88%** of the request), video+audio VAE decode 2.40 s. VAE patch parallelism is the cheapest win available — it cuts decode 3.4–3.5× (8.24 s → 2.40 s). ### Validated performance (4×MI300X) Measured with `vllm/vllm-omni-rocm:minimax-h3` (`sha256:29d1946af9c69e3e0a7128c247bdc8c82437ead43fbc366fed443533cf6ce9e8`), BF16, AITER, U4, text-encoder TP4, VPP4 tile, regional compile, one warmup, and the synchronous Video API: | Workload | Model stages | Client E2E | |---|---:|---:| | T2VA, 209 frames, 1344×768 (8.7 s) | encode 0.09 s, denoise 244.04 s, decode 4.15 s | **267.42 s** | | FL2VA, 209 frames, 1344×768 (8.7 s) | encode 13.98 s, denoise 257.58 s, decode 4.11 s | **287.07 s** | Both outputs were validated as 209-frame H.264 at 24 FPS with 32 kHz stereo AAC. A v0.26.0 source build additionally completed image+audio and video-reference Ref2VA, plus a 2048×1088 T2VA run. These measurements describe tested shapes, not a general latency guarantee. A `with_stack` profile of the two-video Ref2VA case shows the pipeline is **GPU-attention-bound, not host-starved**: FlashAttention-4 is ~76% of `diffuse` device time, Ulysses send/recv is secondary, GPU utilization is 92.8–94.3% and the CPU-idle union inside the transformer forwards is 3.15%. Reducing DiT time further needs a change in attention cost, which is not lossless; [SAGE + Skip-Softmax](#optional-lossy-attention-sage--skip-softmax) above is the supported way to trade attention precision and sparsity for speed on datacenter Blackwell. Against the checkpoint's own reference implementation, the accuracy-qualified path scores SSIM 0.9873–0.9896, PSNR 39–42 dB, pixel cosine 0.9996+, and audio log-mel cosine 0.977–0.996 on T2VA/FL2VA. The two-video Ref2VA composite scores lower on raw pixels (SSIM 0.628) largely because of H.264 bitrate differences, while CLIP cosine is 0.9816 and audio log-mel cosine 0.9869 — content and motion stay aligned. ### Cache-DiT Opt-in only. A 50-step T2VA probe cut denoise time **30.2%** (121.01 s → 84.47 s) but dropped SSIM to **0.831** against the uncached output. H3's video trajectory is sensitive to cached denoise steps — choose the cache budget against your own quality target. ### Offload and accuracy On a **single GPU**, model-level CPU offload is the accuracy-qualified path. On four GPUs, no-offload is both faster and accuracy-clean once the encoder stays resident: an earlier no-offload build performed a ~63 GiB DiT GPU→CPU→GPU round trip inside `encode_prompt`, which was pure overhead (encoder 30.38 s → 0.207 s once removed, ~146×). ## Known limitations - The modular root service loads both DiTs by default. Capacity profiles use `--task-type fl2va` or `--task-type ref2va` and expose only that task family. - H3 executes **one generation request per diffusion batch** today. - The first request is a compile warmup only on the resident regional-compile profiles. The two-GPU DLO recipes use `--enforce-eager` and do not compile the DiT. - **The serving path accepts fewer references than the model supports.** H3 documents up to 9 images, 3 video clips, and 3 audio clips (12 files) per Omni Reference request; the current vLLM-Omni path takes exactly one image plus one audio reference, **or** one or more videos with **no** separate `audio_reference` (it uses the source soundtracks). - The 768 px short-edge mode is available for T2VA and FL2VA; 1344×768 is the documented 16:9 request shape. The 24 GB DLO recipe uses 1024×576 as its lower-memory starting point. - `--cfg-parallel-size > 1` is rejected by design (CFG-distilled, no negative branch). - The VAE supports the native `tile` parallel mode only. - A `U2 × Ring2` hybrid currently fails with an attention-mask length mismatch; use pure Ulysses. - FastH3 preview v1 serves T2VA only, is fused at startup, and cannot use model-level, layerwise, or distributed layerwise offload. Its VSA variants additionally require CUDA, `fastvideo-kernel`, `FASTVIDEO_VSA`, and pure local/Ulysses attention; Ring and AllGather sequence parallelism are rejected. - Online FP8 quantizes the DiT only — the Qwen3-VL text encoder and both VAEs stay BF16. It is incompatible with layerwise offload, whose weight stride is rejected by the Cutlass FP8 kernel. - On unified-memory platforms (DGX Spark / GB10) CPU offload and DLO give no capacity benefit, because host RAM and VRAM are one pool; DLO additionally doubles peak usage while staging. Use resident FP8 with a single `--task-type` partition instead. - Pure Ulysses still replicates the 66.3 GB DiT on every rank, so 64 GB GPUs cannot use `--usp N --tp 1` as a resident capacity path. Use DiT TP plus DLO (the two-GPU recipes above), or model-level CPU offload; text-encoder TP alone is not sufficient. - DLO is a capacity path, not a free speedup: it needs substantial host RAM and streams weights over PCIe on every denoise step. Increase `--dlo-resident-layers` only after checking peak HBM on the exact request shape. ## References - [Model card](https://huggingface.co/MiniMaxAI/MiniMax-H3) - [Modular FL2VA/Ref2VA pipeline (vLLM-Omni #5720)](https://github.com/vllm-project/vllm-omni/pull/5720) - [vLLM-Omni Video API](https://docs.vllm.ai/projects/vllm-omni/en/latest/serving/videos_api/) - [vLLM-Omni diffusion parallelism](https://docs.vllm.ai/projects/vllm-omni/en/latest/user_guide/diffusion/parallelism/overview/) - [vLLM-Omni diffusion attention backends](https://docs.vllm.ai/projects/vllm-omni/en/latest/user_guide/diffusion/attention_backends/) - [FastVideo VSA attention guide](https://docs.vllm.ai/projects/vllm-omni/en/latest/user_guide/diffusion/attention_backends/fastvideo_vsa/) - [FastH3 preview adapters](https://huggingface.co/FastVideo/FastVideo-FastH3-4-step-Preview-v1-LoRA) - [FastH3 family and deployment guides](https://github.com/lishunyang12/vllm-omni/blob/pr/fasth3-omni-upstream/recipes/FastVideo/README.md) - [FastH3 Omni and V2 integration (vLLM-Omni #9167)](https://github.com/vllm-project/vllm-omni/pull/9167) - [MiniMax-H3 VSA and Ulysses support (vLLM-Omni #6909)](https://github.com/vllm-project/vllm-omni/pull/6909) - [vLLM-Omni TRTLLM attention guide (SAGE, Skip-Softmax)](https://docs.vllm.ai/projects/vllm-omni/en/latest/user_guide/diffusion/attention_backends/trtllm/) - [MiniMax-H3 production serving on vLLM-Omni (blog)](https://vllm.ai/blog/2026-09-01-minimax-h3-production-serving) - [vLLM-Omni supported models](https://docs.vllm.ai/projects/vllm-omni/en/latest/models/supported_models/) - [vLLM-Omni GPU installation](https://docs.vllm.ai/projects/vllm-omni/en/latest/getting_started/installation/gpu/) - [ROCm H3 tracking issue](https://github.com/vllm-project/vllm-omni/issues/5697) --- # RecursiveSI/flint on vLLM Source: https://recipes.vllm.ai/RecursiveSI/flint Provider: RecursiveSI Released: 2026-10-21T14:20:58.000Z Updated: 2026-10-21 Architecture: moe Parameters: 39.8B Context length: 262,144 Min vLLM: nightly Tasks: text Verified hardware: GB200 NVL4 Summary: 39.8B-parameter (2.5B active) MoE with hybrid sliding-window / full attention, reasoning and tool calling; BF16 and NVFP4 checkpoints on a single Blackwell GPU. ## Variants - default: precision=bf16, vram_minimum_gb=96 BF16 safetensors (79.6 GB of weights). - nvfp4: precision=nvfp4, vram_minimum_gb=29, model_id=RecursiveSI/flint-nvfp4-gguf NVFP4 GGUF served weight-only (W4A16); 23.5 GB resident after load (the LM head is dequantized to BF16). Config and tokenizer come from the BF16 repo. ## Guide ## Overview Flint is a 39.8B-parameter mixture-of-experts model (256 experts, top-6 sigmoid routing plus a shared expert; ~2.5B parameters active per token) with 49 layers: 35 sliding-window layers (window 513, RoPE) and 14 full-attention layers (no RoPE), GQA 32/4, attention sinks, and a 262,144-token context. Two checkpoints are served: - [RecursiveSI/flint](https://huggingface.co/RecursiveSI/flint): BF16 safetensors. - [RecursiveSI/flint-nvfp4-gguf](https://huggingface.co/RecursiveSI/flint-nvfp4-gguf): a single NVFP4 GGUF (repacked losslessly at load and run as W4A16, like the reference runtime). It ships no config or tokenizer; both come from the BF16 repo. Tested end to end on one GB200 (TP=1, BF16 KV cache, CUDA graphs, no torch.compile) for both checkpoints. On a single GPU the Flint kernels are always used (fused MoE, small-batch CuTe GEMV, CuTe decode attention, low-latency decode path for <= 8 requests); TP > 1 and expert parallelism fall back to stock vLLM layers and are not validated. ## Launch BF16: ```bash vllm serve RecursiveSI/flint --served-model-name flint \ --enable-auto-tool-choice --tool-call-parser glm47 --reasoning-parser glm47 \ --exclude-tools-when-tool-choice-none ``` NVFP4 (requires `gguf>=0.19.0`): ```bash vllm serve RecursiveSI/flint-nvfp4-gguf \ --hf-config-path RecursiveSI/flint --tokenizer RecursiveSI/flint \ --served-model-name flint \ --enable-auto-tool-choice --tool-call-parser glm47 --reasoning-parser glm47 \ --exclude-tools-when-tool-choice-none ``` ## Client usage Use `/v1/chat/completions`. Sampling defaults from `generation_config.json`: temperature 1.0, top_p 0.95, top_k 20 (greedy decoding is not validated). Reasoning is returned in `reasoning` and counts toward `max_tokens`; use `max_tokens` of at least 32768 for hard problems. ```bash curl localhost:8000/v1/chat/completions -H 'Content-Type: application/json' -d '{ "model": "flint", "messages": [{"role": "user", "content": "What is 17*23? Answer briefly."}], "max_tokens": 1024 }' ``` Accuracy on GB200 with this recipe (BF16 / NVFP4): GSM8K 0.950 / 0.950, GPQA-Diamond (4 repeats) 0.721 / 0.696, MRCR (48 samples, up to 114K-token prompts) ~0.47 / ~0.46, tool-calling suite (19 cases x 10: typed args, tool selection, parallel calls, multi-turn, tool_choice none, escaping, streaming) 10/10 on every case. ## Benchmarking Forward time on one GB200 (`vllm bench forward`, batch x context, us): | shape | BF16 | NVFP4 | |---|---|---| | decode 1 x 100K | 2341 | 1783 | | decode 8 x 100K | 6836 | 5703 | | decode 32 x 100K | 19580 | 16716 | | prefill 8K @ 100K | 145617 | 144765 | Serving (NVFP4, `vllm bench serve`, random 1000 in / 1000 out): TPOT 2.0 ms at concurrency 1, 2.8 ms at 8, 4.9 ms at 32, 8.5 ms at 128; median inter-token latency 2.06 ms at batch 1 with 100K context. Prefill runs eagerly and is CPU-bound on kernel-launch overhead: TTFT is ~60 ms for a short prompt and ~125 ms for an 8K-token prompt. ## Troubleshooting - Rust frontend: not supported for this recipe. It rejects `--exclude-tools-when-tool-choice-none`, and for NVFP4 also `--hf-config-path` / `--tokenizer`; keep the default Python frontend. - Reasoning parser: the reference runtime's `deepseek-v3` name maps to vLLM's `deepseek_v3` parser, which only extracts reasoning when the request enables `thinking`; with Flint the reasoning then stays in `content`. Use `glm47`. - If `content` contains literal `` or `` text, the server was started without the parsers above. - Batch-1 decode time varies by ~3-4% between sessions (a GPU-side state, not configuration). ## References - [Flint (BF16) model card](https://huggingface.co/RecursiveSI/flint) - [Flint NVFP4 model card](https://huggingface.co/RecursiveSI/flint-nvfp4-gguf) --- # zai-org/GLM-5.3-Flash on vLLM Source: https://recipes.vllm.ai/zai-org/GLM-5.3-Flash Provider: GLM (Z-AI) Released: 2026-08-25T06:43:14.000Z Updated: 2026-10-19 Architecture: moe Parameters: 321B Context length: 1,048,576 Min vLLM: 0.29.0 Tasks: text, multimodal Verified hardware: H100, B200, GB200 NVL4, MI355X, Ascend 950PR 8x Summary: GLM-5.3-Flash is a 320B-total / 18B-active multimodal MoE with hybrid KDA and sparse MLA attention, native FP8 weights, MTP, and a 1M-token context window. ## Variants - default: precision=fp8, vram_minimum_gb=386 Default native FP8 checkpoint (zai-org/GLM-5.3-Flash) - nvfp4: precision=nvfp4, vram_minimum_gb=229, model_id=RedHatAI/GLM-5.3-Flash-NVFP4 NVFP4 checkpoint (RedHatAI/GLM-5.3-Flash-NVFP4) — MoE experts quantized to 4-bit; requires Blackwell - bf16: precision=bf16, vram_minimum_gb=772, model_id=zai-org/GLM-5.3-Flash-BF16 Official BF16 checkpoint; requires substantially more GPU memory ## Recommended base flags ```bash vllm serve 'zai-org/GLM-5.3-Flash' \ --tokenizer-mode \ glm5_next ``` Environment: ```bash VLLM_ENGINE_READY_TIMEOUT_S='3600' ``` ## Guide ## Overview GLM-5.3-Flash is a multimodal mixture-of-experts model with approximately 321B total parameters and 18B active parameters per token. Its 45-layer language model combines KDA linear-attention layers with NoPE sparse MLA layers, routes each token through 8 of 288 experts, and supports image and video inputs. The checkpoint declares a maximum context length of 1,048,576 tokens and includes one MTP draft layer. The implementation supports NVIDIA Hopper and newer GPUs, AMD Instinct gfx950 via ROCm, and Huawei Ascend 950PR (8 NPUs, one node) via vLLM Ascend. On NVIDIA/AMD it can scale with tensor, pipeline, expert, or data+expert parallelism. The Ascend path is validated as single-node TP=8 only. ## Prerequisites - **Weights:** about 306 GiB for the default native FP8 checkpoint before runtime and KV-cache overhead; the BF16 variant requires roughly twice the weight memory - **vLLM:** an image containing [native GLM5Next support (#62424)](https://github.com/vllm-project/vllm/pull/62424), including the matching Rust frontend binary. Older images need an updated build for the commands below. - **FlashInfer:** 0.6.17 or newer is required for NoPE sparse MLA The default model ID, `zai-org/GLM-5.3-Flash`, is the FP8 checkpoint. Select the BF16 variant to serve `zai-org/GLM-5.3-Flash-BF16`, or the NVFP4 variant to serve `RedHatAI/GLM-5.3-Flash-NVFP4`. Its MoE experts are quantized to 4-bit (NVFP4), and it requires NVIDIA Blackwell GPUs. ## Frontends The command builder defaults to **Rust** and adds `--tokenizer-mode glm5_next`. With `VLLM_USE_RUST_FRONTEND=1`, this selects native GLM5Next chat rendering for the GLM-5.3-Flash template, including tool calls and reasoning options. Image and video inputs are not yet supported on the Rust frontend; use the Python frontend for multimodal requests. Select **Python** in the command builder, or set `VLLM_USE_RUST_FRONTEND=0`, to use the Hugging Face renderer and processor on the same build. On older builds that do not recognize `glm5_next`, also replace `--tokenizer-mode glm5_next` with `--tokenizer-mode auto`. ## Launching the server ### FP8 with TP4 and MTP on one GB200 tray ```bash VLLM_USE_RUST_FRONTEND=1 vllm serve zai-org/GLM-5.3-Flash \ --tokenizer-mode glm5_next \ --tensor-parallel-size 4 \ --kv-cache-dtype fp8 \ --speculative-config '{"method":"mtp","num_speculative_tokens":5}' \ --tool-call-parser glm47 \ --reasoning-parser glm47 \ --enable-auto-tool-choice \ --served-model-name zai-org/GLM-5.3-Flash ``` ### Prefill/Decode disaggregation (single node, TP4 + TP4) Split one 8-GPU node into a prefill pool (GPUs 0-3) and a decode pool (GPUs 4-7), bridged by NIXL KV transfer. The KDA conv-state and KV-cache layouts must be pinned identically on both pools, and MTP drafts run on both sides. On Blackwell you can add `--kv-cache-dtype fp8` to both pools; Hopper does not support FP8 KV cache for this model and must run BF16 KV. > Note: `num_speculative_tokens` must be the same in prefill and decode instance ```bash # Prefill pool CUDA_VISIBLE_DEVICES=0,1,2,3 \ VLLM_SSM_CONV_STATE_LAYOUT=DS \ VLLM_KV_CACHE_LAYOUT=HND \ UCX_NET_DEVICES=all \ VLLM_NIXL_SIDE_CHANNEL_PORT=5557 \ VLLM_USE_RUST_FRONTEND=1 \ vllm serve zai-org/GLM-5.3-Flash \ --tokenizer-mode glm5_next \ --port 8001 \ --tensor-parallel-size 4 \ --kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_producer"}' \ --speculative-config '{"method":"mtp","num_speculative_tokens":5}' \ --compilation-config '{"cudagraph_mm_encoder": true}' \ --enforce-eager \ --no-disable-hybrid-kv-cache-manager \ --served-model-name zai-org/GLM-5.3-Flash # Decode pool CUDA_VISIBLE_DEVICES=4,5,6,7 \ VLLM_SSM_CONV_STATE_LAYOUT=DS \ VLLM_KV_CACHE_LAYOUT=HND \ UCX_NET_DEVICES=all \ VLLM_NIXL_SIDE_CHANNEL_PORT=5558 \ VLLM_USE_RUST_FRONTEND=1 \ vllm serve zai-org/GLM-5.3-Flash \ --tokenizer-mode glm5_next \ --port 8002 \ --tensor-parallel-size 4 \ --kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_consumer"}' \ --speculative-config '{"method":"mtp","num_speculative_tokens":5}' \ --max-num-seqs 512 \ --no-disable-hybrid-kv-cache-manager \ --served-model-name zai-org/GLM-5.3-Flash # Router vllm-router --policy round_robin --vllm-pd-disaggregation \ --prefill http://127.0.0.1:8001 \ --decode http://127.0.0.1:8002 \ --host 0.0.0.0 --port 8000 \ --intra-node-data-parallel-size 1 ``` ## Reasoning modes Thinking is always on — the generation prompt opens a `` block unconditionally. GLM-5.3-Flash offers **three reasoning effort levels** driven by the `reasoning_effort` field; the default is **`max`**: | Mode | How to request | Behavior | |------|----------------|----------| | **Think Max** (default) | omit `reasoning_effort`, or set `"max"` | Deepest reasoning — hard math, multi-step planning, agentic tasks. Highest token cost. | | **Think High** | `"reasoning_effort": "high"` | Balanced depth and latency. | | **Think Low** | `"reasoning_effort": "low"` | Lightest reasoning — simple Q&A, lowest latency and token cost. | The chat template resolves effort to **`max` unless `reasoning_effort` is explicitly `"low"` or `"high"`** (any other value falls back to `max`), then injects `Reasoning Effort: Low|High|Max` into the system prompt. Pass it through `chat_template_kwargs` or the top-level OpenAI `reasoning_effort` field. ```python from openai import OpenAI client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1") msgs = [{"role": "user", "content": "Summarize sparse attention in one sentence."}] # Think Max (default) — just omit reasoning_effort client.chat.completions.create(model="zai-org/GLM-5.3-Flash", messages=msgs, max_tokens=4096) # Think High / Think Low — explicitly request the effort level client.chat.completions.create( model="zai-org/GLM-5.3-Flash", messages=msgs, max_tokens=4096, extra_body={"chat_template_kwargs": {"reasoning_effort": "low"}}, ) ``` ```bash curl http://localhost:8000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "zai-org/GLM-5.3-Flash", "messages": [{"role": "user", "content": "Summarize sparse attention in one sentence."}], "temperature": 1, "max_tokens": 4096, "chat_template_kwargs": {"reasoning_effort": "high"} }' ``` ## Client usage ```python from openai import OpenAI client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1") response = client.chat.completions.create( model="zai-org/GLM-5.3-Flash", messages=[{"role": "user", "content": "Summarize sparse attention in one sentence."}], temperature=1.0, max_tokens=256, ) print(response.choices[0].message.content) ``` ### Multimodal input (image / video) Standard OpenAI multi-part content — list `image_url` / `video_url` items alongside the text. The chat template expands each into its `<|begin_of_image|>…` / `<|begin_of_video|>…` placeholder tokens, and the same image token is reused for video frames (frame spans are delimited by the video start/end tokens). ```python from openai import OpenAI client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1") # Image client.chat.completions.create( model="zai-org/GLM-5.3-Flash", messages=[{ "role": "user", "content": [ {"type": "image_url", "image_url": {"url": "https://example.com/image.png"}}, {"type": "text", "text": "Describe the image."}, ], }], max_tokens=512, ) ``` ```python from openai import OpenAI client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1") # Video client.chat.completions.create( model="zai-org/GLM-5.3-Flash", messages=[{ "role": "user", "content": [ {"type": "video_url", "video_url": {"url": "https://example.com/clip.mp4"}}, {"type": "text", "text": "What happens in this video?"}, ], }], max_tokens=512, ) ``` ```bash # Image curl http://localhost:8000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "zai-org/GLM-5.3-Flash", "messages": [{ "role": "user", "content": [ {"type": "image_url", "image_url": {"url": "https://example.com/image.png"}}, {"type": "text", "text": "Describe the image."} ] }], "max_tokens": 512 }' ``` ```bash # Video curl http://localhost:8000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "zai-org/GLM-5.3-Flash", "messages": [{ "role": "user", "content": [ {"type": "video_url", "video_url": {"url": "https://example.com/clip.mp4"}}, {"type": "text", "text": "What happens in this video?"} ] }], "max_tokens": 512 }' ``` ## Running on AMD Instinct Use a `vllm/vllm-openai-rocm:nightly` build containing #62424 for native Rust rendering. Model support landed in [vLLM #53906](https://github.com/vllm-project/vllm/pull/53906). ```bash export VLLM_ROCM_USE_AITER=1 # gfx950 (MI350X/MI355X) only export VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS=1 export VLLM_ROCM_USE_AITER_TRITON_SPARSE_MLA=1 VLLM_USE_RUST_FRONTEND=1 vllm serve zai-org/GLM-5.3-Flash \ --tokenizer-mode glm5_next \ --served-model-name zai-org/GLM-5.3-Flash \ --tensor-parallel-size 4 \ --max-num-seqs 512 \ --tool-call-parser glm47 \ --reasoning-parser glm47 \ --enable-auto-tool-choice \ --attention-backend ROCM_AITER_MLA_SPARSE ``` At TP=4 the FP8 checkpoint reports a 14.92M-token KV pool and ~113.81x max concurrency at 128K context; the BF16 checkpoint reports an 8.87M-token KV pool at the same TP and context. MTP speculative decoding is supported on MI355X by vLLM builds containing [vLLM #55239](https://github.com/vllm-project/vllm/pull/55239), validated on 4×MI355X with TP=4 and MTP5. Enable **Spec Decoding** and select **MTP** in the command builder to add the required configuration. ### Long-context indexer top-k (gfx950) If your traffic is mostly long context (>=64k tokens) and using gfx950 (MI350X/MI355X), using the topk indexer backend from AITER is recommended: ```bash VLLM_USE_RUST_FRONTEND=1 vllm serve zai-org/GLM-5.3-Flash \ --tokenizer-mode glm5_next \ --attention-backend ROCM_AITER_MLA_SPARSE \ --sparse-indexer-topk-backend aiter ``` ## Benchmarking ```bash vllm bench serve \ --backend vllm \ --model zai-org/GLM-5.3-Flash \ --served-model-name zai-org/GLM-5.3-Flash \ --dataset-name random \ --random-input-len 8192 \ --random-output-len 1024 \ --max-concurrency 16 \ --num-prompts 64 ``` ## Troubleshooting - **Sparse-MLA initialization error:** verify that the image contains FlashInfer 0.6.18 or newer. ## KV cache offloading Offloading requires a build containing [vllm-project/vllm#57145](https://github.com/vllm-project/vllm/pull/57145). Validation used [zai-org/GLM-5.3-Flash](https://huggingface.co/zai-org/GLM-5.3-Flash) on 4×H200 (TP4), vLLM commit `d1b4028d7e`, the Python frontend, `--enforce-eager`, `--max-model-len 16384`, `--max-num-seqs 16` and `--max-num-batched-tokens 8192`. Validation used `TieringOffloadingSpec`: 8 GiB total CPU tier, `blocks_per_chunk: 1`, a persistent filesystem `root_dir`, eight read/write threads and `PYTHONHASHSEED=0`. Three CPU-tier reloads and three filesystem reloads after restarting the server passed. GLM returned matching final answers; its arithmetic reasoning field varied on filesystem reload. The five-shot GSM8K diagnostic scored 31/32; this is a small sample, not the full benchmark. These results cover this text-only configuration, not the builder's larger tier/chunk defaults or other variants. ## References - [Model card](https://huggingface.co/zai-org/GLM-5.3-Flash) - [vLLM](https://github.com/vllm-project/vllm) - [ROCm MTP sparse-MLA fix](https://github.com/vllm-project/vllm/pull/55239) - **FlashInfer:** 0.6.17 or newer is required for NoPE sparse MLA - **Ascend 950PR:** native FP8 checkpoint (~306 GiB). On 950 the block scales are re-grouped to MXFP8 at load (weights stay 1 byte/element). Do not pass `--quantization ascend`. `--block-size 512` is required (kpool indexer). A2/A3 and the official BF16 checkpoint are not validated in this recipe. --- # zai-org/GLM-5.3 on vLLM Source: https://recipes.vllm.ai/zai-org/GLM-5.3 Provider: GLM (Z-AI) Released: 2026-08-25T06:42:50.000Z Updated: 2026-10-19 Architecture: moe Parameters: 743B Context length: 1,048,576 Min vLLM: 0.29.0 Tasks: text Verified hardware: B300, H200, MI325X, MI350X, Ascend 950PR 8x, DGX Station (GB300) Summary: GLM-5.3 — Frontier Coding with Emergent Cyber Capabilities ## Variants - default: precision=fp8, vram_minimum_gb=893 Native FP8 checkpoint (zai-org/GLM-5.3) — 8xH200/H20 (141GB × 8) single-node serving - nvfp4: precision=nvfp4, vram_minimum_gb=558, model_id=Inferact/GLM-5.3-NVFP4 Inferact NVFP4 checkpoint — only MoE expert linears are quantized to NVFP4; shared experts, attention, embeddings, and the early dense layers stay FP8/BF16, with FP8 KV cache. Blackwell GPUs (B200/B300) only. On DGX Station, weights CPU-offload into unified system RAM. - w4a8: precision=int4, vram_minimum_gb=447, model_id=gpustack/GLM-5.3-W4A8 GPUStack Hopper-native W4A8 (gpustack/GLM-5.3-W4A8). Routed experts are INT4 group-128; non-expert layers stay FP8. Weights drop from 704 GB to 372 GB, roughly doubling KV cache on 8×H200/H20 with no measurable accuracy drop vs official FP8. CUTLASS W4A8 is SM90-only — H100/H200/H20; Blackwell and AMD are unsupported. - bf16: precision=bf16, vram_minimum_gb=1786, model_id=zai-org/GLM-5.3-BF16 Full precision BF16 — requires multi-node deployment ## Recommended base flags ```bash vllm serve 'zai-org/GLM-5.3' \ --tokenizer-mode \ glm5_next \ --kv-cache-dtype \ fp8_e4m3 ``` ## Guide ## Overview **GLM-5.3** is the newest model in the GLM-5 series — a ~743B-parameter MoE (39B active) from **Z-AI**. The **default `zai-org/GLM-5.3` checkpoint is native FP8** — BF16 weights live under the suffixed `zai-org/GLM-5.3-BF16` repo. The FP8 checkpoint fits on a single 8xH200 / 8xH20 node and — with FP8 KV cache — reaches the full 1M-token context on 8xB200. An NVFP4 checkpoint for Blackwell is published by Inferact (`Inferact/GLM-5.3-NVFP4`). ## Prerequisites - **vLLM:** a build containing [native GLM5Next support (#62424)](https://github.com/vllm-project/vllm/pull/62424), including the matching Rust frontend binary. - **Transformers:** 5.15.0 or newer. - **GPU:** 8xH200 or 8xH20 (141 GB each) for single-node FP8; 8xB200 (180 GB each) for the full 1M context. ## Installation Use a wheel or container built with #62424 for the Rust commands below. The older release and hardware-specific image pins record prior model validation; they need an updated build to use `--tokenizer-mode glm5_next`. ## Frontends The command builder defaults to **Rust** and adds `--tokenizer-mode glm5_next`. With `VLLM_USE_RUST_FRONTEND=1`, this selects native GLM5Next chat rendering for GLM-5.3, including tool calls and reasoning options. Select **Python** in the command builder, or set `VLLM_USE_RUST_FRONTEND=0`, to use the Hugging Face renderer on the same build. On older builds that do not recognize `glm5_next`, also replace `--tokenizer-mode glm5_next` with `--tokenizer-mode auto`. ## Launching the server ### FP8 on 8xH200 (standard) ```bash VLLM_USE_RUST_FRONTEND=1 vllm serve zai-org/GLM-5.3 \ --tokenizer-mode glm5_next \ --kv-cache-dtype fp8 \ --tensor-parallel-size 8 \ --speculative-config.method mtp \ --speculative-config.num_speculative_tokens 5 \ --tool-call-parser glm47 \ --reasoning-parser glm47 \ --enable-auto-tool-choice \ --served-model-name glm-5.3 ``` ### FP8 on AMD MI300X/MI325X/MI350X/MI355X ```bash VLLM_ROCM_USE_AITER=1 \ VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS=1 \ VLLM_USE_RUST_FRONTEND=1 \ vllm serve zai-org/GLM-5.3 \ --tokenizer-mode glm5_next \ --kv-cache-dtype fp8_e4m3 \ --tensor-parallel-size 8 \ --speculative-config.method mtp \ --speculative-config.num_speculative_tokens 5 \ --tool-call-parser glm47 \ --enable-auto-tool-choice \ --reasoning-parser glm47 \ --gpu-memory-utilization 0.80 \ --max-model-len 524288 \ --max-num-seqs 32 \ --linear-backend aiter \ --moe-backend aiter ``` Earlier validation on 8x MI325X with `vllm/vllm-openai-rocm:v0.29.0` served `--max-model-len 524288` at `--gpu-memory-utilization 0.80`, reporting a 1,477,168-token KV cache and 2.82x maximum concurrency. MTP is healthy there: a mean acceptance length of 2.77 with 5 tokens drafted per step on a ~500-token greedy generation. On 8x MI350X with the same image and settings, it reports a 2,013,072-token KV cache and 3.84x maximum concurrency, and MTP's mean acceptance length is 3.19 with 5 tokens drafted per step on a 512-token greedy generation. ### FP8 on 8xB200 (full 1M context) GLM-5.3 has a native 1M-token window. Whether the full window fits is a KV-cache VRAM question, so the lever is `--max-num-seqs` — it bounds how many sequences share the KV budget at once. **Start at 32** and scale with your node's VRAM. FP8 KV cache (`--kv-cache-dtype fp8_e4m3`, already in the base flags) roughly halves that budget. ```bash VLLM_USE_RUST_FRONTEND=1 vllm serve zai-org/GLM-5.3 \ --tokenizer-mode glm5_next \ --kv-cache-dtype fp8_e4m3 \ --tensor-parallel-size 8 \ --speculative-config.method mtp \ --speculative-config.num_speculative_tokens 5 \ --max-num-seqs 32 \ --tool-call-parser glm47 \ --reasoning-parser glm47 \ --enable-auto-tool-choice \ --served-model-name glm-5.3 ``` - **`--max-num-seqs 32`** — the single knob for fitting 1M context; start at 32 and tune it to your VRAM (up on headroom, down on OOM). - **BF16** weights are served from `zai-org/GLM-5.3-BF16` and need multi-node deployment. ### NVFP4 on Blackwell (B200/B300) The `Inferact/GLM-5.3-NVFP4` variant is Inferact's NVFP4 re-quantization : only the MoE expert linears drop to NVFP4 while shared experts, attention, embeddings, and the early dense layers stay BF16. The ~465 GB checkpoint fits comfortably on Blackwell Select the **NVFP4** variant above (Blackwell-only) or run: ```bash VLLM_USE_RUST_FRONTEND=1 vllm serve Inferact/GLM-5.3-NVFP4 \ --tokenizer-mode glm5_next \ --tensor-parallel-size 8 \ --enable-expert-parallel \ --reasoning-parser glm47 \ --tool-call-parser glm47 \ --enable-auto-tool-choice \ --kv-cache-dtype fp8_e4m3 \ --served-model-name glm-5.3-nvfp4 ``` ### DSpark (opt-in) DSpark is an external confidence-scheduled drafter. Serve `zai-org/GLM-5.3` and attach [RedHatAI/GLM-5.3-speculator.dspark](https://huggingface.co/RedHatAI/GLM-5.3-speculator.dspark) with eight speculative tokens: ```bash VLLM_USE_RUST_FRONTEND=1 vllm serve zai-org/GLM-5.3 \ --tokenizer-mode glm5_next \ --tensor-parallel-size 8 \ --speculative-config '{"method":"dspark","model":"RedHatAI/GLM-5.3-speculator.dspark","num_speculative_tokens":8}' ``` ## Reasoning modes Thinking is always on — the generation prompt opens a `` block unconditionally. GLM-5.3 offers **three reasoning effort levels** driven by the `reasoning_effort` field; the default is **`max`**: | Mode | How to request | Behavior | |------|----------------|----------| | **Think Max** (default) | omit `reasoning_effort`, or set `"max"` | Deepest reasoning — hard math, multi-step planning, agentic tasks. Highest token cost. | | **Think High** | `"reasoning_effort": "high"` | Balanced depth and latency. | | **Think Low** | `"reasoning_effort": "low"` | Lightest reasoning — simple Q&A, lowest latency and token cost. | The chat template resolves effort to **`max` unless `reasoning_effort` is explicitly `"low"` or `"high"`** (any other value falls back to `max`), then injects `Reasoning Effort: Low|High|Max` into the system prompt. Pass it through `chat_template_kwargs` or the top-level OpenAI `reasoning_effort` field. ```python from openai import OpenAI client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1") msgs = [{"role": "user", "content": "Summarize GLM-5.3 in one sentence."}] # Think Max (default) — just omit reasoning_effort client.chat.completions.create(model="glm-5.3", messages=msgs, max_tokens=4096) # Think High / Think Low — explicitly request the effort level client.chat.completions.create( model="glm-5.3", messages=msgs, max_tokens=4096, extra_body={"chat_template_kwargs": {"reasoning_effort": "low"}}, ) ``` ```bash curl http://localhost:8000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "glm-5.3", "messages": [{"role": "user", "content": "Summarize GLM-5.3 in one sentence."}], "temperature": 1, "max_tokens": 4096, "chat_template_kwargs": {"reasoning_effort": "high"} }' ``` ## Client usage ```python from openai import OpenAI client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1") response = client.chat.completions.create( model="glm-5.3", messages=[{"role": "user", "content": "Summarize GLM-5.3 in one sentence."}], temperature=1.0, max_tokens=256, ) print(response.choices[0].message.content) ``` ## Benchmarking Add `--no-enable-prefix-caching` to the server command for a clean measurement. ```bash vllm bench serve \ --model zai-org/GLM-5.3 \ --dataset-name random \ --random-input-len 8000 \ --random-output-len 1024 \ --request-rate 10 \ --num-prompts 32 \ --ignore-eos ``` > **Note:** pure throughput benchmarks tend to under-report real speed, because MTP's > acceptance rate is usually low in synthetic runs. ## Troubleshooting - **FP8 performance:** DeepGEMM is required — install via `install_deepgemm.sh`. ## References - [Model card](https://huggingface.co/zai-org/GLM-5.3) - [BF16 checkpoint](https://huggingface.co/zai-org/GLM-5.3-BF16) - [NVFP4 checkpoint (Inferact)](https://huggingface.co/Inferact/GLM-5.3-NVFP4) - [DSpark draft](https://huggingface.co/RedHatAI/GLM-5.3-speculator.dspark) --- # OpenMOSS-Team/MOSS-Image-Base on vLLM Source: https://recipes.vllm.ai/OpenMOSS-Team/MOSS-Image-Base Provider: OpenMOSS Released: 2026-10-11T06:49:17.000Z Updated: 2026-10-16 Architecture: dense Parameters: 5B Min vLLM: 0.31.0 Tasks: omni Summary: Serve MOSS-Image-Base and MOSS-Image-SFT with vLLM-Omni for text-to-image generation, Chinese text rendering, and JSON layout prompts. ## Variants - default: precision=bf16, vram_minimum_gb=33 MOSS-Image-Base; BF16 DiT/text encoder and FP32 VAE. The memory estimate is for batch 1. - sft: precision=bf16, vram_minimum_gb=33, model_id=OpenMOSS-Team/MOSS-Image-SFT Supervised fine-tune with the same architecture and pipeline. ## Recommended base flags ```bash vllm serve 'OpenMOSS-Team/MOSS-Image-Base' \ --enforce-eager ``` ## Guide ## Overview MOSS-Image supports natural-language prompts, Chinese and English text rendering, and structured JSON layouts. This recipe serves [MOSS-Image-Base](https://huggingface.co/OpenMOSS-Team/MOSS-Image-Base) and [MOSS-Image-SFT](https://huggingface.co/OpenMOSS-Team/MOSS-Image-SFT) through vLLM-Omni's `/v1/images/generations` API. MOSS-Image is a 5B diffusion transformer paired with a Qwen3.5 text encoder and a FLUX.2 VAE. DiT and text-encoder weights use BF16; the VAE, noise, guidance arithmetic, and Euler updates use FP32. ## Choose a Docker image Install Docker and the NVIDIA Container Toolkit on a Linux GPU host. | Image | CUDA runtime | Suggested Linux driver | | --- | --- | --- | | `vllm/vllm-omni:mosimage` | 13.0.2 | R580 or newer | | `vllm/vllm-omni:mosimage-cu129` | 12.9.1 | 575.57.08 or newer | The container supplies CUDA libraries; a separate host CUDA Toolkit installation is not required. See NVIDIA's [driver compatibility table](https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/index.html#cuda-driver). Use image builds containing the `MOSSImagePipeline` registration from [vLLM-Omni PR #8926](https://github.com/vllm-project/vllm-omni/pull/8926). The pipeline is detected automatically for both Base and SFT checkpoints. ## Start a single-GPU server ```bash export IMAGE=vllm/vllm-omni:mosimage # For CUDA 12.9: export IMAGE=vllm/vllm-omni:mosimage-cu129 export MODEL=OpenMOSS-Team/MOSS-Image-Base # For SFT: export MODEL=OpenMOSS-Team/MOSS-Image-SFT docker pull "$IMAGE" docker run --rm --name moss-image \ --gpus 'device=0' --ipc=host \ -p 8000:8000 \ -v "$HOME/.cache/huggingface:/root/.cache/huggingface" \ -e HF_TOKEN \ "$IMAGE" \ vllm serve "$MODEL" --omni \ --enforce-eager \ --port 8000 ``` `-e HF_TOKEN` forwards an existing host token if one is set. For an existing Hugging Face cache, replace the host side of the mount with its cache root. For a standalone model directory, mount it with `-v /path/to/model:/model:ro`, use `vllm serve /model`, and add `--served-model-name "$MODEL"`. These images have no default entrypoint: include `vllm serve` after the image name. Wait for startup to finish, then check the server from another terminal: ```bash curl --fail-with-body -sS http://localhost:8000/health curl --fail-with-body -sS http://localhost:8000/v1/models ``` ## Inference Parameters MOSS-Image-Base has no post-training or CFG distillation, preserving its training information for community adaptation such as LoRA. Inference settings therefore matter for image quality. The official [Inference Parameters guide](https://github.com/OpenMOSS/MOSS-Image/blob/main/USERGUIDE.md#inference-parameters) ([中文](https://github.com/OpenMOSS/MOSS-Image/blob/main/USERGUIDE_ZH.md#推理参数配置)) recommends: | Parameter | Official guidance | | --- | --- | | Resolution (`size`) | Target 1M, 1.5M, or 2M total pixels. The model also supports extrapolation from 0.25M (512 × 512) to 4M (2048 × 2048). | | Aspect ratio | Any ratio; both dimensions must be positive multiples of 16. | | Steps (`num_inference_steps`) | Use at least 50 for best quality. | | CFG (`guidance_scale`) | Default 3.0; recommended range 3.0–6.0. Long prompts generally work at 3.0; increase it as needed for shorter prompts. | | Shift (`flow_shift`) | Default 5.0. | | Sampler | Euler is recommended. | The guide describes two forms of **Skip Layer Guidance (SLG)**: - **Standard SLG:** use scale 3 over the 0.01–1.0 sampling interval. - **Two-pass sampling SLG:** the recommended `uncond skip` mode skips layer 22 (zero-based) in the negative branch. Layers 24 and 26 are also useful alternatives. The authors report that disabling SLG can increase realism while degrading details. In vLLM-Omni, SLG options belong in `extra_params`. Standard SLG uses `slg_layers` and `slg_scale`; uncond skip uses `uncond_skip_layers` without requiring `slg_scale`. Both use `slg_start` and `slg_stop` to select the sampling interval. ## Generate an image This example uses 1024 × 1024, 50 steps, CFG 3.0, shift 5.0, and seed 0, with `uncond_skip_layers=[22]` active over `slg_start=0.01` to `slg_stop=1.0`. Natural-language descriptions, tag lists, and JSON layouts are supported. English is recommended for scene descriptions; keep visible Chinese text in Chinese and insert explicit line breaks to control its layout. This kitchen example follows the guide's approach to multilingual text rendering: ```bash cat > request.json <<'JSON' { "prompt": "A sunlit kitchen with a mint-green refrigerator beside a wooden counter. On the refrigerator, a yellow note has three clearly handwritten Chinese lines:\n记得买牛奶和鸡蛋！\n牛奶两盒，鸡蛋一打。\n顺便带一袋面包。", "size": "1024x1024", "n": 1, "num_inference_steps": 50, "guidance_scale": 3.0, "flow_shift": 5.0, "seed": 0, "negative_prompt": "nsfw, worst quality, bad anatomy, watermark, jpeg artifact", "response_format": "b64_json", "extra_params": { "uncond_skip_layers": [22], "slg_start": 0.01, "slg_stop": 1.0 } } JSON curl --fail-with-body -sS http://localhost:8000/v1/images/generations \ -H 'Content-Type: application/json' \ --data-binary @request.json -o response.json && \ python3 - <<'PY' import base64 import json from pathlib import Path response = json.loads(Path("response.json").read_text()) Path("moss-image.png").write_bytes(base64.b64decode(response["data"][0]["b64_json"])) print("Saved moss-image.png") PY ``` The request uses the server's loaded model, so it also works with SFT. To disable negative-branch skipping, set `uncond_skip_layers` to `[]`. ### Layout prompts For precise text and object placement, use the guide's experimental Layout format: `summary` describes the scene, and `items` supplies captions, visible `text`, and bounding boxes. Coordinates use `[y_min, x_min, y_max, x_max]` with the canvas center as origin; a 1024 × 1024 canvas spans −512 to +512 on each axis. Encode the layout as the API's `prompt` **string** and use the matching canvas size: ```bash python3 - <<'PY' import json from pathlib import Path layout = { "summary": {"caption": "A travel poster on ivory paper with teal mountains and a Chinese heading."}, "items": [ {"label": "text", "caption": "Large teal heading at the top.", "text": "向山而行", "bbox": [-432, -368, -288, 368]}, {"label": "object", "caption": "Layered mountain silhouettes below the heading.", "bbox": [-176, -336, 400, 336]} ], "bbox_coord_space": "abs_centralized", "bbox_format": "yxyx" } request = json.loads(Path("request.json").read_text(encoding="utf-8")) request["prompt"] = json.dumps(layout, ensure_ascii=False, separators=(",", ":")) request["size"] = "1024x1024" Path("request-layout.json").write_text(json.dumps(request, ensure_ascii=False), encoding="utf-8") PY curl --fail-with-body -sS http://localhost:8000/v1/images/generations \ -H 'Content-Type: application/json' \ --data-binary @request-layout.json -o response.json ``` Decode `response.json` with the image-saving snippet above. ## Improve prompts with the official skill The official [MOSS-Image Prompt Enhancer](https://github.com/OpenMOSS/MOSS-Image/tree/main/skills/moss-image-prompt-enhancer) turns an image idea into a detailed caption or a spatial Layout prompt. Choose a caption for scenes and illustrations, or Layout for posters, panels, and precise text placement. The skill prepares prompts; vLLM-Omni generates the image. Copy the complete `skills/moss-image-prompt-enhancer` directory, including its `scripts` and `references`, into your agent's skills directory. See the official [SKILL.md](https://github.com/OpenMOSS/MOSS-Image/blob/main/skills/moss-image-prompt-enhancer/SKILL.md) for its workflow. For example, ask the agent: ```text Use $moss-image-prompt-enhancer to create a 1024×1024 Layout prompt for a travel poster with teal mountains on ivory paper. Put the exact Chinese heading “向山而行” at the top. Save the draft as draft.json and compile it to prompt.txt. ``` The bundled `compile_prompt.py` converts top-left pixel boxes in the draft into MOSS-Image's centered layout coordinates. It also supports caption drafts and needs only Python's standard library. Send the compiled `prompt.txt` as a string, keeping the requested dimensions the same as the draft: ```bash python3 - <<'PY' import json from pathlib import Path request = json.loads(Path("request.json").read_text(encoding="utf-8")) request["prompt"] = Path("prompt.txt").read_text(encoding="utf-8").strip() request["size"] = "1024x1024" Path("request-enhanced.json").write_text(json.dumps(request, ensure_ascii=False), encoding="utf-8") PY curl --fail-with-body -sS http://localhost:8000/v1/images/generations \ -H 'Content-Type: application/json' \ --data-binary @request-enhanced.json -o response.json ``` ## Troubleshooting - **Unexpected sampling behavior:** send `flow_shift: 5.0` explicitly to match the checkpoint's scheduler configuration. - **CUDA driver mismatch:** select the CUDA 12.9 image or update the host driver to the matching supported version. ## References - [MOSS-Image-Base model card](https://huggingface.co/OpenMOSS-Team/MOSS-Image-Base) - [MOSS-Image-SFT model card](https://huggingface.co/OpenMOSS-Team/MOSS-Image-SFT) - [MOSS-Image project and user guide](https://github.com/OpenMOSS/MOSS-Image) - [Official Chinese user guide](https://github.com/OpenMOSS/MOSS-Image/blob/main/USERGUIDE_ZH.md) - [Official Prompt Enhancer skill](https://github.com/OpenMOSS/MOSS-Image/blob/main/skills/moss-image-prompt-enhancer/SKILL.md) - [vLLM-Omni implementation PR #8926](https://github.com/vllm-project/vllm-omni/pull/8926) - [vLLM-Omni Docker images](https://hub.docker.com/r/vllm/vllm-omni/tags?name=mosimage) --- # meta-models/Muse-Spark-Open-816B-A42B on vLLM Source: https://recipes.vllm.ai/meta-models/Muse-Spark-Open-816B-A42B Provider: Muse (Meta) Released: 2026-09-15T19:23:19.000Z Updated: 2026-10-16 Architecture: moe Parameters: 816B Context length: 1,048,576 Min vLLM: 0.33.0 Tasks: text, multimodal Verified hardware: H200, GB200 NVL4, GB300 NVL4, MI350X, MI355X Summary: Sparse MoE reasoning model with 816B total / 42B active parameters and a 1M-token context, built for long-horizon agentic coding. Native image and video perception, channel-scoped reasoning and tool calls via the `muse` parsers, and an optional DSpark draft model for speculative decoding. ## Variants - bf16: precision=bf16, vram_minimum_gb=1959, model_id=meta-models/Muse-Spark-Open-816B-A42B BF16 reference weights for NVIDIA and AMD; the ROCm path is validated on gfx950 (MI350X/MI355X). - mxfp4: precision=mxfp4, vram_minimum_gb=611, model_id=amd/Muse-Spark-Open-816B-A42B-Quark-MXFP4 AMD Quark OCP MXFP4 W4A4 weights for gfx950 on the AITER MXFP4 MoE kernels. - default: precision=nvfp4, vram_minimum_gb=640, model_id=meta-models/Muse-Spark-Open-816B-A42B-NVFP4 NVFP4 weights for Blackwell. The primary NVIDIA serving path. ## Guide ## Overview Muse Spark Open is a sparse-MoE reasoning model — 816B total parameters, 42B active — with a 1M-token context window, aimed at long-horizon agentic coding. It is the model Muse Glimmer 30B was distilled from. ## Architecture 62 layers, hidden size 8192, and **every layer is a mixture-of-experts layer** — there are no dense layers. Each holds 256 experts and uses 8 per token. The experts are 799B of the 816B; everything else is small. **The experts work narrow.** A learned matrix squeezes each token from 8192 to 4096, all 256 experts run at that width, and a second matrix widens the result back. Both stay in full precision even when the experts are 4-bit. The router reads the token *before* the squeeze, and in fp32 — picking the wrong expert costs more than running a smaller one. There is no shared expert. **Attention is 128 query heads sharing 16 key-value heads**, head dim 64, so the KV cache is eight times smaller than with one key-value head per query head. **Layers alternate between a narrow and a wide view**, on a cycle of four: ``` sliding_window: 2048 layer_types: [sliding_attention, full_attention, sliding_attention, sliding_attention, ...] ``` Three layers in each cycle look back 2048 tokens; the other (layers 1, 5, …, 61) sees the whole context and drops rotary position (`layer_rope_theta: 0`), because rotary becomes unreliable over very long distances. Across 62 layers that is **46 narrow layers and 16 wide ones**, and it is what makes a 1M-token context affordable — only the 16 wide layers hold KV for the whole sequence. **Normalization appears in more places than usual**, each doing a job: a norm before *and* after each block (the ones after carry no learnable weight), on queries and keys before rotary position, on each attention head before its sigmoid gate, and on every expert's output before the weighted blend. The residual stream is carried in fp32. Logits are scaled and then squashed through tanh (`final_logit_softcapping: 20.0`) before the softmax. ## Prerequisites - **vLLM**: Use Docker image for now until the code is in vLLM main repo. - **Hardware**: the checkpoints hold ~1.64 TB of weights in BF16 and ~509 GB in Quark MXFP4, before KV cache. NVFP4 (~533 GB) is the primary NVIDIA Blackwell path and fits one 8-GPU node. On AMD, all three AMD variants are validated on gfx950 (MI350X/MI355X). ## NVIDIA Meta ships BF16 and NVFP4 checkpoints. On Blackwell, serve `meta-models/Muse-Spark-Open-816B-A42B-NVFP4` on one 8-GPU node with **Tensor + Expert Parallel** (TP8 + `--enable-expert-parallel`), the default strategy here. The command builder turns DSpark on by default for NVFP4; see [Speculative decoding](#speculative-decoding-dspark). NVIDIA supports every parallelism this recipe offers: TP, TP + PP, TP + expert parallel, data + expert parallel, and prefill/decode disaggregation. AMD supports only TP and TP + expert parallel. ## AMD ROCm variants Quark MXFP4 is the default AMD variant. | Variant | Checkpoint | AMD hardware | Experts / attention | | --- | --- | --- | --- | | MXFP4 | `amd/Muse-Spark-Open-816B-A42B-Quark-MXFP4` | MI350X, MI355X | AITER MXFP4 MoE / AITER FA | | BF16 → MXFP4 | `meta-models/Muse-Spark-Open-816B-A42B` | MI350X, MI355X | experts of layers 1-60 rounded to MXFP4 at load, AITER MXFP4 MoE / AITER FA | | BF16 | `meta-models/Muse-Spark-Open-816B-A42B` | MI350X, MI355X | Triton BF16 MoE / AITER FA | In both MXFP4 variants the experts of the first and last layers stay BF16 and run on Triton BF16 MoE. On AMD, the recipe's commands are text-only (`--language-model-only`), use the V2 model runner, disable prefix caching, enable chunked prefill, and request breakable `FULL_AND_PIECEWISE` graphs. The ROCm MoE path supports the Tensor Parallel and Tensor + Expert Parallel strategies; data-parallel strategies are not available on AMD. ### Choosing TP or TP + expert parallel (MXFP4 variants) Use plain **TP** for latency-bound serving (a handful of concurrent requests) and the **Tensor + Expert Parallel** strategy (TP8 + `--enable-expert-parallel`, 32 of 256 experts per GPU) for throughput at 32 or more concurrent requests, where it also lowers TTFT. Accuracy is the same under both. Keep **TP** for BF16: its Triton experts are slower with expert parallelism. With DSpark on, expert parallelism pays off from a few concurrent requests; see the DSpark section. The AMD commands use `--max-model-len 32768`. For the full 1M-token context, the MXFP4 variant is validated on TP with `--max-model-len 1048576` and `--max-num-seqs 8`, with either the default or `--kv-cache-dtype fp8` KV cache. ## Launching the Server NVIDIA Blackwell, NVFP4, TP8 + expert parallel, with DSpark: ```bash export VLLM_USE_RUST_FRONTEND=1 vllm serve meta-models/Muse-Spark-Open-816B-A42B-NVFP4 \ --tensor-parallel-size 8 \ --enable-expert-parallel \ --max-model-len 1048576 \ --reasoning-parser muse \ --tool-call-parser muse \ --enable-auto-tool-choice \ --speculative-config '{"method":"dspark","model":"Inferact/Muse-Spark-Open-DSpark","num_speculative_tokens":7,"draft_sample_method":"probabilistic","rejection_sample_method":"block","enable_adaptive_verification":true}' ``` ## Frontends Commands use the Rust frontend by default on NVIDIA and AMD ROCm. To use the Python frontend, select Python in the command builder or replace `VLLM_USE_RUST_FRONTEND=1` with `VLLM_USE_RUST_FRONTEND=0` in the generated command. ## Reasoning effort Set the strength per request with the top-level `reasoning_effort` field: | `reasoning_effort` | Strength hint | | --- | --- | | `low` | 8 | | `medium` | 32 | | `high` | 256 | | `xhigh` | 512 (default) | ```bash curl http://localhost:8000/v1/chat/completions \ -H 'Content-Type: application/json' \ -d '{ "model": "meta-models/Muse-Spark-Open-816B-A42B-NVFP4", "messages": [{"role": "user", "content": "Refactor this module."}], "reasoning_effort": "xhigh" }' ``` ### Multimodal Muse Spark is a native multimodal model that understands image, audio and video. Video is processed through a series of sampled video frames at load time. On AMD, the recipe serves text only. ```bash curl http://localhost:8000/v1/chat/completions \ -H 'Content-Type: application/json' \ -d '{ "model": "meta-models/Muse-Spark-Open-816B-A42B-NVFP4", "messages": [{"role": "user", "content": [ {"type": "video_url", "video_url": {"url": "https://example.com/demo.mp4"}}, {"type": "text", "text": "What happens, and when?"} ]}] }' ``` The audio track of a video is not processed. To include it, extract the audio and send it as a separate audio part in the same request. Video sampling rate can be changed when starting the server. ```bash --media-io-kwargs '{"video": {"fps": 1}}' ``` ## Speculative decoding (DSpark) Inferact trained a DSpark speculator for this model. The draft is a small sliding-window model that reads the target's residual stream at selected layers (`target_layer_ids`) and proposes a block per forward. On NVIDIA, DSpark is on by default for NVFP4. It drafts up to seven tokens and uses **adaptive verification** (`enable_adaptive_verification: true`), which uses draft confidence to verify only the slots likely to be accepted. Because of that it keeps paying off at large batch sizes, not just for single-request latency. Turn off **Spec Decoding** in the builder to serve without the draft. On AMD, selecting **DSpark** emits the fixed seven-token profile with AITER FA attention on every variant. Its causal multi-token decode is handled by AITER unified attention, with the gfx950 short-query 3D schedule enabled. DSpark gains the most when only a few requests are in flight, and still stays ahead of plain TP at 32 concurrent requests. It also runs under the **Tensor + Expert Parallel** strategy on the MXFP4 variants (TP8 + `--enable-expert-parallel`), which is the faster combination from a few concurrent requests up to about 64 at the same accuracy; plain TP + DSpark remains the lowest-latency choice for a single request. With expert parallelism, DSpark is validated up to `--max-num-seqs 128`; with plain TP, keep `--max-num-seqs 32`. For throughput beyond about 64 concurrent requests, use expert parallelism without DSpark. ### Adaptive DSpark (manual opt-in, Triton attention) Adaptive verification captures variable-length decode CUDA graphs, which the AITER FA attention backend does not support; with the AMD MXFP4 command above it fails at startup. To use it on MI350X/MI355X, switch the target to Triton attention and use the adaptive config: ```bash export VLLM_ADAPTIVE_VERIFICATION_PROFILE_CONTEXT_LEN=8192 --attention-backend TRITON_ATTN \ --speculative-config '{"method":"dspark","model":"Inferact/Muse-Spark-Open-DSpark","num_speculative_tokens":7,"draft_sample_method":"probabilistic","rejection_sample_method":"standard","enable_adaptive_verification":true,"attention_backend":"TRITON_ATTN"}' ``` Seven tokens remain the maximum draft depth. At startup, adaptive verification profiles the DSpark and target costs, then uses draft confidence to keep only the slots whose expected accepted-token benefit justifies their verification cost. There is no hand-tuned acceptance threshold. Keep the profile context representative of production traffic. ## References - [Muse Glimmer 30B recipe](/meta-models/Muse-Glimmer-30B) — the distilled 30B sibling --- # reflection/Beam-1.0-501B-A23B on vLLM Source: https://recipes.vllm.ai/reflection/Beam-1.0-501B-A23B Provider: Reflection Released: 2026-10-07T19:52:01.000Z Updated: 2026-10-16 Architecture: moe Parameters: 501B Context length: 1,048,576 Min vLLM: 0.33.0 Tasks: text Verified hardware: B300, H200 Summary: Reflection's Beam — a sparse MoE language model (501B total / 23B active) with hybrid sliding-window + NoPE full attention, a 1M-token window, Harmony-format reasoning and tool calling, and an optional DFlash2 speculative drafter. Available in NVFP4, FP8 and BF16. ## Variants - default: precision=nvfp4, vram_minimum_gb=353, model_id=reflection/Beam-1.0-501B-A23B-NVFP4 ModelOpt NVFP4 on the routed experts; all other weights BF16. W4A4 on Blackwell; W4A16 through Marlin on Hopper. Recommended on Blackwell and H100. - fp8: precision=fp8, vram_minimum_gb=604, model_id=reflection/Beam-1.0-501B-A23B-FP8 FP8 per-channel weights (compressed-tensors); router, embeddings and lm_head unquantized. Recommended on H200. - bf16: precision=bf16, vram_minimum_gb=1204, model_id=reflection/Beam-1.0-501B-A23B Native BF16 checkpoint (FP32 router weights and expert biases). Requires one 8x B200 or B300 node. ## Recommended base flags ```bash vllm serve 'reflection/Beam-1.0-501B-A23B' \ --kv-cache-dtype \ fp8 ``` Environment: ```bash VLLM_ENGINE_READY_TIMEOUT_S='3600' ``` ## Guide ## Overview [Beam](https://huggingface.co/reflection/Beam-1.0-501B-A23B) is Reflection's sparse mixture-of-experts language model, with 501B total parameters and 23B active parameters per token. Its 52 layers combine 512-token sliding-window attention with NoPE full-attention layers, route each token through 8 of 256 experts (layers 0-2 use dense MLPs), and support a context length of 1,048,576 tokens. Beam uses the Harmony message format, with an `analysis` channel for reasoning and a `final` channel for the answer. Generation stops on `<|return|>` at the end of an answer and on `<|call|>` at the end of a tool call. ## Variants | Variant | Checkpoint | Weights on disk | Minimum GPU memory | Recommended hardware | |---|---|---|---|---| | NVFP4 (default) | [`reflection/Beam-1.0-501B-A23B-NVFP4`](https://huggingface.co/reflection/Beam-1.0-501B-A23B-NVFP4) | 294 GB | 353 GB | 4x B200/B300 or one GB200/GB300 tray (TP4); 8x H100 (TP8, Marlin) | | FP8 | [`reflection/Beam-1.0-501B-A23B-FP8`](https://huggingface.co/reflection/Beam-1.0-501B-A23B-FP8) | 503 GB | 604 GB | 8x H200 (TP8) | | BF16 | [`reflection/Beam-1.0-501B-A23B`](https://huggingface.co/reflection/Beam-1.0-501B-A23B) | 1,003 GB | 1,204 GB | 8x B200/B300 (TP8) | *Minimum GPU memory* is the weights plus 20% headroom for activations, CUDA graphs and a working KV cache. It is the total across all GPUs, and it is the figure the variant selector above shows. The NVFP4 checkpoint quantizes only the routed experts; attention, dense MLPs, the router, embeddings and the LM head stay in BF16. On Blackwell it runs W4A4; on Hopper, vLLM serves the FP4 experts through Marlin with BF16 activations. ## Prerequisites - **vLLM:** the `vllm/vllm-openai:reflection-beam` Docker image. Beam support has not shipped in a vLLM release yet. - **Hardware:** NVIDIA GPUs. Beam's kernels, including a custom all-reduce used with expert parallelism, are CUDA-only. - **Transformers:** the tokenizer uses the Transformers 5 format (`TokenizersBackend`). ### Install vLLM ```bash docker pull vllm/vllm-openai:reflection-beam ``` The `vllm serve` commands below run inside this image; the command builder above renders the full `docker run` line. ## Launching the server ### FP8 with TP8 on H200 or Blackwell ```bash vllm serve reflection/Beam-1.0-501B-A23B-FP8 \ --tensor-parallel-size 8 \ --moe-backend triton \ --kv-cache-dtype fp8 \ --enable-auto-tool-choice \ --tool-call-parser beam \ --reasoning-parser beam ``` This is the best-performing FP8 configuration measured on both H200 and B300. Set `--moe-backend triton` explicitly; vLLM otherwise selects CUTLASS for the FP8 MoE. vLLM's default attention backends, FlashAttention 3 on Hopper and FlashInfer with TRT-LLM kernels on Blackwell, are already the fastest measured. ### NVFP4 with TP4 on Blackwell ```bash vllm serve reflection/Beam-1.0-501B-A23B-NVFP4 \ --tensor-parallel-size 4 \ --kv-cache-dtype fp8 \ --enable-auto-tool-choice \ --tool-call-parser beam \ --reasoning-parser beam ``` At TP4 the weights take about 73 GB per GPU, leaving room for the full context window many times over; an 8-GPU node can host two replicas. The first start takes about 10 minutes because FlashInfer autotunes the MoE kernel. That is close to vLLM's default 600 s engine-ready timeout, so the generated command sets `VLLM_ENGINE_READY_TIMEOUT_S=3600`. ### NVFP4 with TP8 on H100 ```bash vllm serve reflection/Beam-1.0-501B-A23B-NVFP4 \ --tensor-parallel-size 8 \ --moe-backend marlin \ --kv-cache-dtype fp8 \ --enable-auto-tool-choice \ --tool-call-parser beam \ --reasoning-parser beam ``` Set `--moe-backend marlin` explicitly; it is faster than the backend vLLM selects by default on Hopper. The FP8 checkpoint also fits on 8x H100 but leaves much less memory for the KV cache. ### BF16 with TP8 on Blackwell ```bash vllm serve reflection/Beam-1.0-501B-A23B \ --tensor-parallel-size 8 \ --enable-auto-tool-choice \ --tool-call-parser beam \ --reasoning-parser beam ``` ### Tuning - **KV cache dtype:** the NVFP4 and FP8 commands use `--kv-cache-dtype fp8`, which halves KV memory per token (on 8x H200 with the FP8 checkpoint, about 14M to 28M tokens). The BF16 command keeps the KV cache in BF16. - **Parallelism:** tensor parallelism is the recommended single-node layout. TEP (`--enable-expert-parallel`) and DEP (`--data-parallel-size 8 --enable-expert-parallel`) are also supported; DEP is better suited to very large batch sizes. ### MegaMoE with attention DP (NVFP4, Blackwell) For prefill-heavy, high-concurrency traffic, run attention data-parallel with expert parallelism on FlashInfer's CuTe DSL NVFP4 MegaMoE kernel, which overlaps the expert all-to-all with the expert GEMMs. Start from the NVFP4 command and replace the parallel and MoE settings: ```bash vllm serve reflection/Beam-1.0-501B-A23B-NVFP4 \ --data-parallel-size 4 \ --enable-expert-parallel \ --moe-backend flashinfer_moe_ep_cutedsl \ --kv-cache-dtype fp8 \ --enable-auto-tool-choice \ --tool-call-parser beam \ --reasoning-parser beam ``` - Only for DEP (`--data-parallel-size` with `--enable-expert-parallel`). It does not support TEP or pure TP. - Use `--data-parallel-size 4` on a GB200/GB300 tray or four B200/B300 GPUs, and `8` across a full 8-GPU node. - For low-latency serving, stay on TP4 with `flashinfer_trtllm`. ### DFlash2 speculative decoding The DFlash2 drafter ([`reflection/Beam-1.0-501B-A23B-DFlash2`](https://huggingface.co/reflection/Beam-1.0-501B-A23B-DFlash2)) shares the target model's embeddings and LM head: ```bash --speculative-config '{"method":"dflash","model":"reflection/Beam-1.0-501B-A23B-DFlash2","num_speculative_tokens":7,"attention_backend":"FLASH_ATTN","enable_adaptive_verification":true}' ``` With `enable_adaptive_verification`, vLLM predicts how likely each draft token is to be accepted and verifies only the ones worth the compute, rather than all 7 every step. It makes little difference at low concurrency and raises throughput under load. Remove it to verify the full draft every step. The drafter supports up to 15 speculative tokens; 7 is recommended. It speeds up decoding most at low concurrency and adds about 1.7 GB per GPU, which reduces KV-cache capacity. Acceptance depends on natural-language input, so benchmark it with a real-text dataset rather than `--dataset-name random`. ### Output constraints By default, the `beam` output grammar constrains every assistant turn from its first token (reasoning, then an answer or a tool call). Set `VLLM_BEAM_ALWAYS_CONSTRAIN_OUTPUT=0` to disable it. ## Client usage ```bash curl http://localhost:8000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "reflection/Beam-1.0-501B-A23B-NVFP4", "messages": [{"role": "user", "content": "Hello."}] }' ``` Set `model` to the checkpoint being served: `reflection/Beam-1.0-501B-A23B-NVFP4`, `reflection/Beam-1.0-501B-A23B-FP8`, or `reflection/Beam-1.0-501B-A23B` for BF16. `/v1/models` lists the served name. ### Reasoning effort Beam reasons at the maximum effort, `max` (512), unless the request sets `reasoning_effort`: ```bash curl http://localhost:8000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "reflection/Beam-1.0-501B-A23B-NVFP4", "messages": [{"role": "user", "content": "What is 17 * 19?"}], "reasoning_effort": "low" }' ``` The Rust frontend (`VLLM_USE_RUST_FRONTEND=1`) serves `/v1/chat/completions` and `/v1/completions`, but not yet `/v1/responses` or the Anthropic-compatible `/v1/messages` (both return 404). Use the default Python frontend for those APIs. ## References - [Beam model card](https://huggingface.co/reflection/Beam-1.0-501B-A23B) - [Beam FP8](https://huggingface.co/reflection/Beam-1.0-501B-A23B-FP8) - [Beam NVFP4](https://huggingface.co/reflection/Beam-1.0-501B-A23B-NVFP4) - [DFlash2 drafter](https://huggingface.co/reflection/Beam-1.0-501B-A23B-DFlash2) --- # Qwen/Qwen3.8-27B on vLLM Source: https://recipes.vllm.ai/Qwen/Qwen3.8-27B Provider: Qwen Released: 2026-08-05T08:22:59.000Z Updated: 2026-10-15 Architecture: dense Parameters: 27B Context length: 262,144 Min vLLM: 0.17.0 Tasks: multimodal, text Verified hardware: GB300 NVL4, B200, RTX 5090, RTX 5090, RTX Pro 6000, DGX Spark (GB10), Ascend 950PR, Arc Pro B60, Arc Pro B70 Summary: 27B-parameter dense hybrid-attention model with linear attention on 48 of 64 layers, a vision tower, a built-in MTP draft head, 262K native context window and extensible to 1M context ## Variants - default: precision=bf16, vram_minimum_gb=67 Full-precision BF16 — 51.7 GiB of weights: 1 GPU. - fp8: precision=fp8, vram_minimum_gb=38, model_id=Qwen/Qwen3.8-27B-FP8 Official block-scaled FP8 checkpoint — 28.7 GiB: 1 GPU. - nvfp4: precision=nvfp4, vram_minimum_gb=32, model_id=Inferact/Qwen3.8-27B-NVFP4 NVFP4 build for NVIDIA Blackwell — 24.6 GiB, the smallest footprint: 1 GPU. - nvfp4_nvidia: precision=nvfp4, vram_minimum_gb=32, model_id=nvidia/Qwen3.8-27B-NVFP4 NVFP4 build for NVIDIA Blackwell — 21.9 GiB, the smallest footprint: 1 GPU. - int4: precision=int4, vram_minimum_gb=24, model_id=RedHatAI/Qwen3.8-27B-INT4 Red Hat AI compressed-tensors W4A16 INT4 build — 19.5 GB: 1 GPU, runs on Hopper too. - mxfp4: precision=mxfp4, vram_minimum_gb=24, model_id=RedHatAI/Qwen3.8-27B-MXFP4 Red Hat AI compressed-tensors MXFP4 build — 19.8 GB: 1 GPU. True W4A4 on Blackwell (SM100+); W4A16 via Marlin on Hopper and elsewhere. - ascend_w8a8: precision=int8, vram_minimum_gb=39, model_id=Eco-Tech/Qwen3.8-27B-w8a8 ModelSlim INT8 W8A8 for Ascend NPU — 32.1 GB: 1 NPU. Requires --quantization ascend. ## Guide ## Overview Qwen3.8-27B is the 27-billion-parameter dense member of the Qwen3.8 family, on the same hybrid-attention backbone as the 2.4T MoE flagship. **The layer mix is the interesting part.** Only 16 of the 64 layers run full attention (`full_attention_interval: 4`); the other 48 run linear attention with a constant recurrent state. Unlike the 2.4T this is a multimodal model: the architecture is `Qwen3_5ForConditionalGeneration` and `config.json` carries a `vision_config`. Text serving is what this recipe covers and what has been verified. Quantized builds (NVFP4, FP8, INT4, and Red Hat AI's MXFP4) are offered as variants alongside the base checkpoint. ## Prerequisites - **transformers >= 5.8.0**, matching the version `config.json` was written by. vLLM parses the config with its own `Qwen3_5Config`, so this is really about the Qwen3-VL processor. ## Launch commands ### Low latency NVFP4, TP1: ```bash vllm serve Inferact/Qwen3.8-27B-NVFP4 \ --tensor-parallel-size 1 \ --max-model-len 262144 \ --kv-cache-dtype fp8 \ --reasoning-parser qwen3 \ --enable-auto-tool-choice --tool-call-parser qwen3_coder ``` FP8, TP4 (one GB300 tray) for the largest KV cache: ```bash vllm serve Qwen/Qwen3.8-27B-FP8 \ --tensor-parallel-size 4 \ --max-model-len 262144 \ --kv-cache-dtype fp8 \ --reasoning-parser qwen3 ``` Add `--speculative-config '{"method":"mtp","num_speculative_tokens":3}'` for MTP. ### MXFP4 (W4A4) — Blackwell ```bash vllm serve RedHatAI/Qwen3.8-27B-MXFP4 \ --tensor-parallel-size 1 \ --reasoning-parser qwen3 \ --enable-auto-tool-choice --tool-call-parser qwen3_coder ``` True W4A4 on Blackwell. The in-checkpoint MTP head, long-context YaRN and tool calling all carry over from the base recipe. ### 2x RTX 5090 (consumer Blackwell, sm120) Verified on vLLM `0.26.1rc1.dev608+g99a10304d`, TP2 across two cards. **NVFP4 uses the real kernel here.** vLLM selects `FlashInferCutlassNvFp4LinearKernel for NVFP4 GEMM` on sm120 — a cutlass path, not an emulation fallback — for both the `Inferact` build above and `unsloth/Qwen3.8-27B-NVFP4`. **The block-scaled FP8 checkpoint needs no workaround.** vLLM auto-disables DeepGemm for `model_type=qwen3_5_text` on Blackwell and falls back to CUTLASS, so it loads unaided. Verified by running *without* `VLLM_USE_DEEP_GEMM=0`: identical 377,456-token KV pool. ```bash vllm serve unsloth/Qwen3.8-27B-NVFP4 \ --tensor-parallel-size 2 \ --max-model-len 262144 \ --kv-cache-dtype fp8 \ --reasoning-parser qwen3 \ --enable-auto-tool-choice --tool-call-parser qwen3_coder ``` At 262,144 context, GPU KV cache size as reported at startup: | precision | KV tokens | weights/GPU | MTP acceptance | |---|---|---|---| | FP8 | 377,456 | 14.28 GiB | 0.771 | | NVFP4 (Inferact) | 445,875 | 12.02 GiB | 0.897 | | NVFP4 (unsloth) | 920,517 | 10.64 GiB | 0.788 | The two NVFP4 builds are not interchangeable on 32 GB cards: `unsloth` is mixed-precision (FP8 channel-wise alongside the 4-bit groups) and leaves room for roughly twice the KV cache, while the uniform-W4A4 `Inferact` build drafts better. Both serve correctly. The in-checkpoint MTP head works in every precision above. Acceptance is measured from `vllm:spec_decode_num_{accepted,draft}_tokens_total`, since throughput alone cannot distinguish a working drafter from one that loaded and was ignored. ### 1x RTX 5090 — NVFP4 needs `--enforce-eager` One card has **31.4 GiB usable**, not 32, and NVFP4 fits only with CUDA graphs off: ```bash vllm serve Inferact/Qwen3.8-27B-NVFP4 \ --tensor-parallel-size 1 \ --max-model-len 32768 \ --kv-cache-dtype fp8 \ --enforce-eager \ --reasoning-parser qwen3 ``` Without `--enforce-eager`, startup dies in CUDA graph capture with `torch.OutOfMemoryError: Tried to allocate 784.00 MiB`. **Raising or lowering `--gpu-memory-utilization` does not help** — 0.80 and 0.93 both leave the same 47.06 MiB free, because that budget covers weights and KV while graph capture allocates outside it. KV pool at 32K context: 91,022 tokens with `--enforce-eager` alone; 135,926 adding `--language-model-only`; 152,917 also capping `--max-num-seqs 8`; 76,458 with bf16 KV instead of fp8 — so fp8 KV is a choice here, not a requirement. The MTP head still fits (90,112 tokens, 0.754 acceptance). Those two flags are levers for a bigger pool, not fixes for the OOM: only `--enforce-eager` decides whether the server starts. ### Intel XPU B60 / B70 Validated on 4x Intel Arc Pro B60 / B70 (B60 24 GB, B70 32 GB per card) with the official vLLM XPU image `vllm/vllm-openai-xpu:latest`. `--enforce-eager` is required on this backend. Select Arc Pro B60/B70 in the hardware picker above for the exact command. ### Huawei Ascend 950PR-w8a8 (vLLM Ascend) Verified on a single 950PR, TP1, vLLM Ascend 0.23.0 from the `quay.io/ascend/vllm-ascend:qwen3.8-a5` image. The ModelSlim INT8 build is the checkpoint to use here. `--quantization ascend` is what selects the ModelSlim scheme — without it the loader treats the weights as float. The MTP head inside the checkpoint works and is spelled `qwen3_5_mtp` on this backend. Both the weight and the image come from ModelScope, not HuggingFace: set `VLLM_USE_MODELSCOPE=True` or pass a local path. ```bash export VLLM_USE_MODELSCOPE=True export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export HCCL_BUFFSIZE=512 export OMP_PROC_BIND=false export OMP_NUM_THREADS=1 vllm serve Eco-Tech/Qwen3.8-27B-w8a8 \ --tensor-parallel-size 1 \ --quantization ascend \ --max-num-batched-tokens 16384 \ --enable-prefix-caching \ --reasoning-parser qwen3 \ --tool-call-parser qwen3_coder \ --mm-encoder-tp-mode data \ --speculative-config '{"method":"qwen3_5_mtp","num_speculative_tokens":3,"enforce_eager":true}' \ --compilation-config '{"cudagraph_mode":"FULL_DECODE_ONLY"}' \ --additional-config '{"enable_cpu_binding":true}' ``` `--reasoning-parser qwen3` is not optional in practice: the chat template opens every assistant turn with ``, so without it the entire reasoning block lands in `message.content` and a 2048-token budget can be spent before the answer starts. Measured single-stream decode: 64 tok/s (2048-token completion, one request, MTP on). Read MTP acceptance from `vllm:spec_decode_num_{accepted,draft}_tokens_total` before tuning `num_speculative_tokens` — throughput alone cannot tell a working drafter from one that loaded and was ignored. ### Huawei Ascend 950PR-Native block-scaled FP8 (vLLM Ascend) Verified on a single 950PR, TP1, vLLM Ascend 0.23.0 from the `quay.io/ascend/vllm-ascend:nightly-main-a5` image. The official `Qwen/Qwen3.8-27B-FP8` checkpoint serves directly on a 950PR — no offline re-quantization and **no `--quantization ascend`**. vLLM Ascend reads `quant_method: fp8` and `weight_block_size` straight from the checkpoint, expands the `weight_scale_inv` tiles at load time and re-quantizes to MXFP8 on 950. Requires a vLLM Ascend build that includes [vllm-ascend#14852](https://github.com/vllm-project/vllm-ascend/pull/14852) (merged 2026-08-26); no tagged release carries it yet. The W8A8 path above is unaffected. ```bash vllm serve Qwen/Qwen3.8-27B-FP8 \ --tensor-parallel-size 1 \ --max-model-len 128000 \ --max-num-seqs 16 \ --gpu-memory-utilization 0.85 \ --trust-remote-code \ --enable-prefix-caching \ --reasoning-parser qwen3 \ --tool-call-parser qwen3_coder \ --mm-encoder-tp-mode data \ --speculative-config '{"method": "qwen3_5_mtp", "num_speculative_tokens": 3, "enforce_eager": true}' ``` ### DFlash2 (opt-in) DFlash2 is not a standalone model. Serve `Qwen/Qwen3.8-27B` and attach [incoai/Qwen3.8-27B-DFlash2](https://huggingface.co/incoai/Qwen3.8-27B-DFlash2) as the drafter. Requires vLLM >=0.28.0, which shipped [vllm#52816](https://github.com/vllm-project/vllm/pull/52816). ```bash vllm serve Qwen/Qwen3.8-27B \ --speculative-config '{"method":"dflash","model":"incoai/Qwen3.8-27B-DFlash2","num_speculative_tokens":7}' ``` ### DSpark (opt-in) DSpark is an external confidence-scheduled drafter. Serve `Qwen/Qwen3.8-27B` and attach [RedHatAI/Qwen3.8-27B-speculator.dspark](https://huggingface.co/RedHatAI/Qwen3.8-27B-speculator.dspark) with eight speculative tokens: ```bash vllm serve Qwen/Qwen3.8-27B \ --speculative-config '{"method":"dspark","model":"RedHatAI/Qwen3.8-27B-speculator.dspark","num_speculative_tokens":8}' ``` ## Client usage `generation_config.json` ships `temperature: 1.0`, `top_p: 0.95`, `top_k: 20`. ```python from openai import OpenAI client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1", timeout=3600) resp = client.chat.completions.create( model="Qwen/Qwen3.8-27B", messages=[{"role": "user", "content": "Give me three primes above 100."}], temperature=1.0, top_p=0.95, max_tokens=2048, ) print(resp.choices[0].message.content) ``` ### Thinking modes The model supports no-think and adaptive thinking through `chat_template_kwargs`, per request or server-wide via `--default-chat-template-kwargs`: - `{"enable_thinking": false}` — no thinking, the model answers directly. - `{"reasoning_effort": "low"}` — adaptive thinking. `xhigh` (default), `medium`, `low`. ## Processing Ultra-Long Texts The model is native at 262k. To extend it to 1M, turn on the **Long Context** feature above: it applies the model card's YaRN RoPE configuration via `--hf-overrides` and raises `--max-model-len` to 1,000,000. Static YaRN holds the scaling factor constant regardless of input length, so leave it off unless you actually serve long prompts; for a ~524K workload, halve the card's `factor` to `2.0`. Note the override is nested under `text_config` here, where the 2.4T takes it flat. See the [model card](https://huggingface.co/Qwen/Qwen3.8-27B#processing-ultra-long-texts) for the full parameter reference. ## Troubleshooting **Two different "MXFP4" paths.** The *native* `Mxfp4Config` path (gpt-oss-style weight-only checkpoints) has no linear-layer method and falls back to `UnquantizedLinearMethod`, so it does not run. This recipe's **RedHatAI MXFP4** variant instead uses the *compressed-tensors* `mxfp4-pack-quantized` format, which maps to `CompressedTensorsW4A4Mxfp4` — a real W4A4 method on Blackwell (SM100+) and W4A16-via-Marlin elsewhere. That path loads and serves correctly on Nvidia; pick the **MXFP4** variant (or NVFP4) when you want the 4-bit build. ## References - Model card: https://huggingface.co/Qwen/Qwen3.8-27B - FP8 checkpoint: https://huggingface.co/Qwen/Qwen3.8-27B-FP8 - NVFP4 build (NVIDIA): https://huggingface.co/nvidia/Qwen3.8-27B-NVFP4 - NVFP4 build (Inferact): https://huggingface.co/Inferact/Qwen3.8-27B-NVFP4 - MXFP4 build (RedHatAI): https://huggingface.co/RedHatAI/Qwen3.8-27B-MXFP4 - Ascend INT8 checkpoint (ModelScope): https://www.modelscope.cn/models/Eco-Tech/Qwen3.8-27B-w8a8 - vLLM documentation: https://docs.vllm.ai/ - vLLM Ascend tutorial: https://docs.vllm.ai/projects/ascend/en/latest/tutorials/models/Qwen3.8-27B.html - DFlash2 draft: https://huggingface.co/incoai/Qwen3.8-27B-DFlash2 - DSpark draft: https://huggingface.co/RedHatAI/Qwen3.8-27B-speculator.dspark - DFlash2 blog: https://inco.ai/blog/dflash2/ - vLLM DFlash2 PR: https://github.com/vllm-project/vllm/pull/52816 - Native FP8 support on Ascend: https://github.com/vllm-project/vllm-ascend/pull/14852 --- # black-forest-labs/flux-3-image-klein on vLLM Source: https://recipes.vllm.ai/black-forest-labs/flux-3-image-klein Provider: Black Forest Labs Released: 2026-10-13T10:58:40.000Z Updated: 2026-10-15 Architecture: dense Parameters: 5.6B Min vLLM: 0.32.0 Tasks: omni Verified hardware: GB200 NVL4 Summary: 5.6B distilled image transformer for text-to-image and image editing, served via vLLM-Omni with a Qwen3-VL-4B text encoder ## Variants - default: precision=bf16, vram_minimum_gb=24 Root BF16 distilled package (5.6B DiT + Qwen3-VL-4B text encoder + VAE), fixed 16-step schedule, no guidance - fp8r: precision=fp8, vram_minimum_gb=17 FP8R DiT variant from the package's variants/fp8r folder; text encoder and VAE stay BF16 - nvfp4: precision=nvfp4, vram_minimum_gb=13 NVFP4 DiT variant from the package's variants/nvfp4 folder; text encoder and VAE stay BF16 ## Guide ## Overview [FLUX 3 Image Klein](https://huggingface.co/black-forest-labs/flux-3-image-klein) is a 5.6B distilled image transformer from Black Forest Labs for text-to-image generation and image editing (single- and multi-reference). It uses Qwen3-VL-4B-Instruct as the text encoder and runs a fixed 16-step schedule with no classifier-free guidance. vLLM-Omni serves it through an OpenAI-compatible API: | Mode | Endpoint | Request body | |------|----------|--------------| | T2I | `/v1/images/generations` | JSON (`prompt`, `size`, `seed`) | | I2I | `/v1/chat/completions` | Chat message with the reference image as a base64 data URI | The checkpoint is a single package: DiT (`model.safetensors`), image autoencoder (`ae.safetensors`), and `text_encoder/` at the root, with quantized DiT packages under `variants/fp8r` and `variants/nvfp4`. Select a variant with `--stage-overrides '{"0":{"additional_config":{"variant":"fp8r"}}}'`; omit it for the BF16 root package. ## Prerequisites - Accept the [FLUX Kommunity License](https://huggingface.co/black-forest-labs/flux-3-image-klein/blob/main/LICENSE.md) on Hugging Face, then `hf auth login`. The license permits non-commercial use with a commercial-output exception for Qualifying Users. - Recommended: use the day-0 Docker image `vllm/vllm-omni:flux-3-image-klein`, which ships vLLM-Omni and all dependencies. Bare-metal installs use vLLM-Omni from source (see Install above). ## Launching the server With the day-0 image: ```bash docker run --runtime nvidia --gpus all \ -v ${HF_HOME:-$HOME/.cache/huggingface}:/root/.cache/huggingface \ --env "HF_TOKEN=$HF_TOKEN" \ -p 8099:8099 \ --ipc=host \ vllm/vllm-omni:flux-3-image-klein \ vllm serve black-forest-labs/flux-3-image-klein --omni \ --served-model-name flux-3-image-klein \ --port 8099 ``` Bare metal: ```bash vllm serve black-forest-labs/flux-3-image-klein --omni \ --served-model-name flux-3-image-klein \ --port 8099 # FP8R / NVFP4 DiT variant vllm serve black-forest-labs/flux-3-image-klein --omni \ --served-model-name flux-3-image-klein \ --stage-overrides '{"0":{"additional_config":{"variant":"fp8r"}}}' \ --port 8099 ``` Useful extras (see `examples/online_serving/flux3/run_server.sh` in the vLLM-Omni repository): - `--tensor-parallel-size N` / `--cfg-parallel-size N` for multi-GPU scaling. - `--fp8-attention` (FlashInfer/cuDNN FP8 DiT attention, SM100 only, needs cuDNN >= 9.17.1). - `--lora-path /path/to/adapter --lora-scale 1.0` to preload a PEFT LoRA adapter; each request still selects the adapter by path. - `uv pip install -e '.[prims-ts]'` enables PrimsTS kernels for FP8R/NVFP4 packages on SM100/SM103/SM107; BF16 always uses vLLM kernels. ## Offline inference ```bash # T2I python examples/offline_inference/flux3/end2end.py \ --model black-forest-labs/flux-3-image-klein \ --prompt "a serene alpine meadow at golden hour, wildflowers in the foreground" # I2I (conditioning image; repeat --image for multi-reference, up to 10) python examples/offline_inference/flux3/end2end.py \ --model black-forest-labs/flux-3-image-klein \ --image /absolute/path/to/reference.png \ --prompt "restyle as a watercolor painting with soft pastel tones" \ --seed 7 ``` ## Prompt upsampling (optional) [black-forest-labs/flux-3-image-klein-pup](https://huggingface.co/black-forest-labs/flux-3-image-klein-pup) is a Qwen3.5-9B vision-language fine-tune that expands a short request into the detailed caption FLUX 3 Image Klein works best with — the reference inference code runs it in front of the image model by default. It is a standard `Qwen3_5ForConditionalGeneration` checkpoint, so it is served with the stock vLLM image on a separate GPU: ```bash hf download black-forest-labs/flux-3-image-klein-pup --local-dir /path/to/pup docker run --gpus '"device=0"' --ipc=host -p 8000:8000 \ -v /path/to/pup:/model:ro \ vllm/vllm-openai:v0.29.0 \ --model /model \ --served-model-name flux-3-image-klein-pup \ --max-model-len 32768 \ --gpu-memory-utilization 0.90 \ --reasoning-parser qwen3 ``` The system prompts (`system_prompts.json` in the pup repo) select text-to-image vs. editing mode, and `t2i_bbox` / `i2i_bbox` variants return the caption plus a JSON table of `[y_min, x_min, y_max, x_max]` boxes on the 0-1000 grid. A ready-made client is `openai_client_pup.py` next to `openai_client.py` in `examples/online_serving/flux3/` of the vLLM-Omni repository — pass `--image` repeatedly (up to 10) for multi-reference editing; feed its printed caption to the image model as the prompt. ## Behavior and constraints - **Distilled package**: do not pass `guidance_scale` or a nonempty `negative_prompt` — the model rejects them. Step count and flow shift default to the package config (16 steps, shift 6.93); `--steps` and `--flow-shift` override them. - **Multi-reference editing**: I2I accepts up to 10 reference images, sent in order as multiple `image_url` entries in the chat message. Prompts can cite them by position ("the first image", "the second image", ...). - **I2I has no strength parameter**: the reference image is fed to the DiT as conditioning tokens and generation always starts from pure noise. The seed controls how far the output drifts from the reference — some seeds stay very close to the input, so change the seed when you want a bolder restyle. - Startup logs show two benign `Error parsing config ... Should have a 'model_type' key` lines: the package's root `config.json` is a custom FLUX format, and vLLM-Omni falls back to the Flux3 pipeline automatically. - Bounding boxes on a 0-1000 grid can be embedded in the prompt to place, move, recolor, or remove elements (see the model card). ## Verified setup Recipe validated end-to-end on 2026-10-15 with the day-0 image build (arm64, CUDA 13.0) on a GB200: T2I 1024x1024 in ~2.1s and I2I in ~2.8s after a ~105s engine startup; ~19 GiB GPU memory with the BF16 package. ## References - [FLUX 3 Image Klein model card](https://huggingface.co/black-forest-labs/flux-3-image-klein) - [FLUX 3 Image Klein Base](https://huggingface.co/black-forest-labs/flux-3-image-klein-base) (undistilled, for fine-tuning / LoRA training) - [FLUX 3 Image Klein PUP](https://huggingface.co/black-forest-labs/flux-3-image-klein-pup) (prompt upsampler) - [vLLM-Omni](https://github.com/vllm-project/vllm-omni) --- # inclusionAI/Ling-3.1-flash on vLLM Source: https://recipes.vllm.ai/inclusionAI/Ling-3.1-flash Provider: inclusionAI Released: 2026-10-13T11:16:41.000Z Updated: 2026-10-15 Architecture: moe Parameters: 560B Context length: 262,144 Min vLLM: nightly Tasks: text Verified hardware: MI355X Summary: Ling-3.1-flash MoE model with block-FP8 weights, tool calling, and reasoning support ## Variants - default: precision=fp8, vram_minimum_gb=689 Block-FP8 (E4M3) weights, 128x128 block scaling, with BF16 compute. ## Recommended base flags ```bash vllm serve 'inclusionAI/Ling-3.1-flash' \ --trust-remote-code \ --enable-expert-parallel \ --gpu-memory-utilization \ 0.9 \ --enable-chunked-prefill \ --compilation-config \ {"cudagraph_mode":"FULL_AND_PIECEWISE"} \ --enable-prefix-caching ``` ## Guide ## Overview `inclusionAI/Ling-3.1-flash` uses the `BailingMoeV3ForCausalLM` architecture with a hybrid MLA/KDA attention stack, 72 layers, 512 routed experts (8 active per token), and one shared expert. The checkpoint uses serialized block-FP8 E4M3 weights with 128x128 block scaling and dynamic activation quantization. Non-quantized modules use BF16. ## Prerequisites - **vLLM:** nightly build (`vllm/vllm-openai-rocm:nightly` on AMD) - **Deployment:** a single node with 8 GPUs, using TP8 + EP8 - **Precision:** serialized block FP8 weights with BF16 compute - **Context length:** 262,144 tokens - **Memory:** approximately 689 GB aggregate VRAM as a loading estimate; allow additional capacity for KV cache and the intended workload ## Launching the Server ```bash vllm serve inclusionAI/Ling-3.1-flash \ --trust-remote-code \ --tensor-parallel-size 8 \ --enable-expert-parallel \ --gpu-memory-utilization 0.9 \ --enable-chunked-prefill \ --compilation-config '{"cudagraph_mode":"FULL_AND_PIECEWISE"}' \ --enable-prefix-caching \ --enable-auto-tool-choice \ --tool-call-parser ling3 \ --reasoning-parser ling3 ``` ### AMD MI355X Use the ROCm nightly image `vllm/vllm-openai-rocm:nightly`, enable AITER and launch using the command: ```bash VLLM_ROCM_USE_AITER=1 vllm serve inclusionAI/Ling-3.1-flash \ --trust-remote-code \ --tensor-parallel-size 8 \ --enable-expert-parallel \ --gpu-memory-utilization 0.9 \ --enable-chunked-prefill \ --compilation-config '{"cudagraph_mode":"FULL_AND_PIECEWISE"}' \ --enable-prefix-caching \ --enable-auto-tool-choice \ --tool-call-parser ling3 \ --reasoning-parser ling3 ``` Each GPU has about 168 GiB of KV cache available. ## Speculative Decoding (MTP) The checkpoint ships a native MTP head. Enable it with three speculative tokens: ```bash --speculative-config '{"method":"mtp","num_speculative_tokens":3}' ``` On MI355X this gave a mean acceptance length of 3.0 (per-position acceptance 0.96 / 0.67 / 0.38) on short reasoning prompts. ## Thinking Mode Thinking is selected per request through the chat template rather than by a server flag: ```python from openai import OpenAI client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY") response = client.chat.completions.create( model="inclusionAI/Ling-3.1-flash", messages=[{"role": "user", "content": "Solve the problem step by step."}], temperature=0.0, max_tokens=16384, extra_body={"chat_template_kwargs": {"enable_thinking": True}}, ) msg = response.choices[0].message print(msg.reasoning) print(msg.content) ``` `--reasoning-parser ling3` separates thinking traces from the final answer. Keep the prompt and requested output within the configured context limit. ## References - [Ling-3.1-flash on Hugging Face](https://huggingface.co/inclusionAI/Ling-3.1-flash) - [vLLM nightly installation](https://docs.vllm.ai/en/latest/getting_started/installation/gpu/#install-the-latest-code) --- # nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-NVFP4 on vLLM Source: https://recipes.vllm.ai/nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-NVFP4 Provider: NVIDIA Released: 2026-10-12T15:32:30.000Z Updated: 2026-10-15 Architecture: moe Parameters: 120B Context length: 1,048,576 Min vLLM: 0.31.0 Tasks: multimodal Verified hardware: H100, H200, GB200 NVL4, DGX Spark (GB10), RTX Pro 6000 Summary: 120B total / 12B active vision-language model with NVFP4, FP8, and BF16 recipes for H100, H200, GB200, DGX Spark, and RTX Pro 6000 ## Variants - default: precision=nvfp4, vram_minimum_gb=91 NVFP4 W4A16 checkpoint; select a hardware profile for its cookbook configuration - fp8: precision=fp8, vram_minimum_gb=155, model_id=RedHatAI/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-FP8 FP8 Dynamic W8A8; model-card baseline uses 2x H100 with MTP or DSpark - bf16: precision=bf16, vram_minimum_gb=299, model_id=nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-BF16 BF16 multimodal checkpoint on 4x H100 or 2x GB200 - nvfp4_gb200_4x: precision=nvfp4, vram_minimum_gb=91, model_id=nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-NVFP4 Four-GPU text-only expert-parallel configuration - bf16_gb200_4x: precision=bf16, vram_minimum_gb=299, model_id=nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-BF16 Four-GPU text-only expert-parallel configuration with float32 Mamba cache ## Recommended base flags ```bash vllm serve 'nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-NVFP4' \ --trust-remote-code ``` ## Guide ## Overview Nemotron 3.5 Super VL is a hybrid Mamba-MoE vision-language model with image and video input. The profiles below follow the NVIDIA cookbook's configurations on `vllm/vllm-openai:v0.31.0`. The page opens on the one-H200 NVFP4 configuration. Every generated command uses the full Hugging Face model ID as its served name. ## Prerequisites - Pull the pinned vLLM image and make the selected checkpoint available from Hugging Face. LiLiCorr on DGX Spark also needs its separate draft checkpoint. - The NVFP4 checkpoint's safetensors index reports 75.01 GB; the recipe's 91 GB minimum includes the schema's 20% allowance. The BF16 index reports 248.91 GB, giving a 299 GB minimum by the same method. - These are weight allowances, not a guarantee that every context and batch setting fits. Use the cookbook's per-hardware memory settings. ## Hardware configurations | Hardware | Variant | TP | Context | Cookbook workload | | --- | --- | ---: | ---: | --- | | 1x H200 | NVFP4 | 1 | 114,688 | Multimodal, MTP 3 | | 2x H100 | NVFP4 | 2 | 114,688 | Multimodal, MTP 3 | | 4x H100 | BF16 | 4 | 114,688 | Multimodal, MTP 3 | | 1x GB200 | NVFP4 | 1 | 114,688 | Text only, MTP 5 | | 2x GB200 | BF16 | 2 | 114,688 | Multimodal, MTP 5 | | 4x GB200 | NVFP4 or BF16 4x | 4 | 28,672 | Text only, MTP 5 | | 1x DGX Spark | NVFP4 | 1 | 114,688 | Text-only MTP 5 or LiLiCorr 7 | | 1x RTX Pro 6000 | NVFP4 | 1 | 16,384 | MTP 5 | Select the matching Hardware and Variant pills to generate these commands. On GB200, choose BF16 with TP 2 for image or video input: the 1x NVFP4 and 4x variants use `--chat-template-content-format string` for text workloads. DGX Spark defaults to its text-only MTP command; the LiLiCorr mode uses the separate DFlash draft checkpoint and changes its concurrency and memory budget. The BF16 notebook examples allow local media paths under `/`; turn on the optional Local Media feature only when serving trusted clients that need it. The cookbook also validates NVFP4 on 2x A100 with an 8,192-token allocation. A100 is not in the site's hardware taxonomy, so use its command directly: ```bash vllm serve nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-NVFP4 \ --served-model-name nemotron-3.5-super \ --trust-remote-code \ --tensor-parallel-size 2 \ --pipeline-parallel-size 1 \ --data-parallel-size 1 \ --max-model-len 8192 \ --max-num-seqs 512 \ --speculative_config.method mtp \ --speculative_config.num_speculative_tokens 5 \ --enable-auto-tool-choice \ --tool-call-parser qwen3_coder \ --reasoning-parser nemotron_v3 ``` Use `nemotron-3.5-super` as the client model name for this A100 command. The cookbook also provides 1M-token text-only commands. They use `--language-model-only` and `VLLM_ALLOW_LONG_MAX_MODEL_LEN=1`; apply those complete commands from the notebook when the full window is needed. The selectable configurations here retain shorter hardware-specific allocations. ## Client usage Send requests to `http://127.0.0.1:8000/v1` with model `nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-NVFP4` or the BF16 model ID, depending on the selected variant. The cookbook includes image and video requests, tool calls, and reasoning output examples. ## References - [NVIDIA vLLM cookbook](https://github.com/NVIDIA-NeMo/Nemotron/blob/main/usage-cookbook/Nemotron-3.5-Super-VL/vllm_cookbook.ipynb) - [NVFP4 checkpoint](https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-NVFP4) - [BF16 checkpoint](https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-BF16) - [LiLiCorr draft checkpoint](https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-NVFP4-LiLiCorr) - [RedHatAI FP8 model card](https://huggingface.co/RedHatAI/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-FP8) - [RedHatAI DSpark model card](https://huggingface.co/RedHatAI/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B.dspark) --- # stepfun-ai/Step-5-Preview on vLLM Source: https://recipes.vllm.ai/stepfun-ai/Step-5-Preview Provider: StepFun Released: 2026-09-20T03:15:06.000Z Updated: 2026-10-15 Architecture: moe Parameters: 604B Context length: 1,048,576 Tasks: multimodal Verified hardware: H200, B300, GB300 NVL4 Summary: Multimodal MoE with hybrid sparse/sliding-window attention, a 1M-token context window, and MTP. ## Variants - default: precision=fp8, vram_minimum_gb=773 - nvfp4: precision=nvfp4, vram_minimum_gb=473, model_id=stepfun-ai/Step-5-Preview-NVFP4 ## Recommended base flags ```bash vllm serve 'stepfun-ai/Step-5-Preview' \ --data-parallel-backend \ mp \ --distributed-executor-backend \ mp \ --disable-custom-all-reduce \ --max-model-len \ 1048576 \ --max-num-batched-tokens \ 16384 \ --max-num-seqs \ 64 \ --gpu-memory-utilization \ 0.85 \ --enable-prefix-caching ``` ## Guide ## Overview [Step-5 Preview](https://huggingface.co/stepfun-ai/Step-5-Preview) is a multimodal MoE model with hybrid sparse/sliding-window attention, a 1M-token context window, and multi-token prediction (MTP). The [NVFP4 checkpoint](https://huggingface.co/stepfun-ai/Step-5-Preview-NVFP4) is available through the NVFP4 variant below. ## Docker ```bash docker pull vllm/vllm-openai:step-5-preview docker run --rm -it \ --gpus all \ --ipc=host \ --network=host \ --volume "$HOME/.cache/huggingface:/root/.cache/huggingface" \ --entrypoint /bin/bash \ vllm/vllm-openai:step-5-preview ``` Run the following commands inside the container. Model access is required; use `hf auth login` if the checkpoint requires authentication. ## FP8: single node on 4 B300 GPUs The examples use four B300 GPUs. Choose TP4, TEP4, or DEP4: | Mode | Tensor parallel | Data parallel | Expert parallel | | --- | --- | --- | --- | | TP4 | 4 | 1 | Disabled | | TEP4 | 4 | 1 | 4 | | DEP4 | 1 | 4 | 4 | ```bash export CUDA_VISIBLE_DEVICES=0,1,2,3 ``` On Blackwell, TEP4 and DEP4 use MegaMoE. TP4 uses the default MoE backend because MegaMoE requires expert parallelism. For FP8 on H200, use eight GPUs: TP8, TEP8, or DEP8, respectively. Set `CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7` and change the parallel size from 4 to 8 (DEP keeps TP=1). Omit the MegaMoE flag on Hopper. These are separate layouts: TP4 and TEP4 use DP=1; DEP4 uses TP=1. `--max-num-seqs` applies per DP rank. The 1M-token context limit includes input and output; reduce `--max-model-len` or concurrency if KV cache memory is insufficient. ## NVFP4: single node on 4 B300 GPUs Use [stepfun-ai/Step-5-Preview-NVFP4](https://huggingface.co/stepfun-ai/Step-5-Preview-NVFP4) with the same Step-5 image. The checkpoint includes both NVFP4 and BF16 layers; quantization is detected from its config. ```bash docker pull vllm/vllm-openai:step-5-preview docker run --rm -it \ --gpus 4 \ --ipc=host \ --network=host \ --volume "$HOME/.cache/huggingface:/root/.cache/huggingface" \ --entrypoint /bin/bash \ vllm/vllm-openai:step-5-preview ``` Inside the container, reuse one of the TP4, TEP4, or DEP4 commands above, replacing the checkpoint with `stepfun-ai/Step-5-Preview-NVFP4` and setting `--max-model-len 320000`. Keep `--reasoning-parser step5` and `--tool-call-parser step5` unchanged. | Mode | Parallel arguments | | --- | --- | | TP4 | `--tensor-parallel-size 4` | | TEP4 | `--tensor-parallel-size 4 --enable-expert-parallel --moe-backend deep_gemm_mega_moe` | | DEP4 | `--tensor-parallel-size 1 --data-parallel-size 4 --enable-expert-parallel --moe-backend deep_gemm_mega_moe` | TP4 uses automatic MoE backend selection for the mixed-precision layers. TEP4 and DEP4 use MegaMoE for routed experts; shared experts run separately. Send requests with `"model": "stepfun-ai/Step-5-Preview-NVFP4"`. Available context and concurrency depend on the remaining KV cache memory. ## Send a request ```bash curl http://127.0.0.1:8000/v1/chat/completions \ --header 'Content-Type: application/json' \ --data '{ "model": "stepfun-ai/Step-5-Preview", "messages": [{"role": "user", "content": "Hello!"}], "max_tokens": 4096, "stream": true }' ``` ## References - [Step-5 Preview NVFP4 model](https://huggingface.co/stepfun-ai/Step-5-Preview-NVFP4) - [Step-5 Preview model](https://huggingface.co/stepfun-ai/Step-5-Preview) - [Step-5 Docker image](https://hub.docker.com/r/vllm/vllm-openai/tags?name=step-5-preview) --- # vllm-sr/d3-edge on vLLM Source: https://recipes.vllm.ai/vllm-sr/d3-edge Provider: vLLM Semantic Router Released: 2026-10-10T23:49:23.000Z Updated: 2026-10-15 Architecture: dense Parameters: 0.59B Context length: 262,144 Min vLLM: nightly Tasks: embedding, multimodal Verified hardware: MI300X Summary: Decision 3.0's 0.6B decision model: answers typed questions (choice, yes / no, score) about text, JSON, images and videos with a probability for every option, without generating text. ## Variants - default: precision=bf16, vram_minimum_gb=2 BF16 weights (1.2 GB) and an FP32 answer-code readout ## Guide ## Overview [vllm-sr/d3-edge](https://huggingface.co/vllm-sr/d3-edge) is the 0.6B model of Decision 3.0, the decision models of [vLLM Semantic Router](https://github.com/vllm-project/semantic-router). Give it a state (text or JSON, optionally with images and videos) and typed questions: pick one of several options (`choice`), yes or no (`noul`), or a level on a scale (`score`). It returns a probability for every option, without generating text. It is a backbone of the Qwen3.5 architecture (0.59B parameters, including the 0.10B vision encoder), trained from `Qwen/Qwen3.5-0.8B`, with an answer-code readout. vLLM detects the checkpoint's `decision_config.json` and serves it as a classification model (`D3ForSequenceClassification`) on its System One endpoint, `/v1/systemone`: one forward pass per question, whose readout scores that question's options at the last token. The full attention layers read the prompt in both directions, so chunked prefill and prefix caching are off. The Decision 3.0 models, all served the same way: - [`vllm-sr/d3`](https://huggingface.co/vllm-sr/d3) (27B) - [`vllm-sr/d3-flash`](https://huggingface.co/vllm-sr/d3-flash) (9B) - [`vllm-sr/d3-mini`](https://huggingface.co/vllm-sr/d3-mini) (4B) - [`vllm-sr/d3-nano`](https://huggingface.co/vllm-sr/d3-nano) (2B) - [`vllm-sr/d3-lite`](https://huggingface.co/vllm-sr/d3-lite) (0.8B) - [`vllm-sr/d3-edge`](https://huggingface.co/vllm-sr/d3-edge) (0.6B) — this recipe ## Prerequisites - **Hardware**: one GPU. The BF16 weights take 1.2 GB; vLLM reserves 92 % of the GPU's memory by default. - **vLLM**: a nightly build. Decision 3.0 support ([vllm-project/vllm#61565](https://github.com/vllm-project/vllm/pull/61565)) is on vLLM main since 2026-10-14 and not in a release yet (the latest, v0.31.0, does not include it). Every nightly built since 2026-10-15 includes it. - **ROCm**: Python 3.12, glibc 2.39 or newer (for example Ubuntu 24.04), and the system libraries `libopenmpi3t64` and `libdw1t64`, which the ROCm wheels load at import. ```bash uv venv --python 3.12 --seed source .venv/bin/activate # NVIDIA GPUs (CUDA) uv pip install -U vllm --torch-backend=auto --extra-index-url https://wheels.vllm.ai/nightly # AMD GPUs (ROCm 7.2; glibc 2.39 or newer, as on Ubuntu 24.04) sudo apt install -y libopenmpi3t64 libdw1t64 export VLLM_ROCM_VARIANT=$(curl -s https://wheels.vllm.ai/rocm/nightly | grep -oP 'rocm\d+' | head -1 | sed 's/%2B/+/g') uv pip install --pre vllm --extra-index-url https://wheels.vllm.ai/rocm/nightly/${VLLM_ROCM_VARIANT} --index-strategy unsafe-best-match ``` ## Launch ```bash vllm serve vllm-sr/d3-edge ``` No flags are needed; the first start downloads the model. ## Client usage The command builder's cURL and Bench buttons send chat completions, which a decision model does not serve. Ask questions on `/v1/systemone` instead: ```bash curl -s localhost:8000/v1/systemone -H 'content-type: application/json' -d '{ "model": "vllm-sr/d3-edge", "state": "Write a Python function that merges two sorted lists.", "questions": { "kind": {"type": "choice", "instructions": "What kind of work is this?", "criteria": {"code": "Writing or fixing code", "math": "Mathematics", "chat": "Anything else"}}, "reasoning": {"type": "noul", "instructions": "Does answering this need multi-step reasoning?"} } }' ``` Each answer carries the pick, the probability of every option, and `confidence`, the chosen option's probability; a `noul` answer's value is the probability of yes. The probabilities are the checkpoint runtime's (`model.system_one()`). Images and videos go in `images` and `videos` as http(s) or base64 data URLs; vLLM places them before the text and reads them with the runtime's budgets (images up to 1.6 megapixels; videos at 2 fps, at most 32 frames): ```bash curl -s localhost:8000/v1/systemone -H 'content-type: application/json' --data-binary @- <` … `<|speaker:15|>`) and inline event tags such as `[sighs]` or `[laughs]`. vLLM-Omni serves it as two stages on one GPU: the AR stage (backbone + codebook decoder) streams frames to the codec stage, which returns audio through the OpenAI-compatible `/v1/audio/speech` API. ## Prerequisites - **Hardware**: a single CUDA GPU with at least 48 GB. Reference run: 1x H100 80 GB (stage-0 weights 36.9 GiB). - **vLLM-Omni** with Drama 3 support, on vLLM >= 0.31. ## Installation ```bash uv venv && source .venv/bin/activate uv pip install vllm==0.31.0 --torch-backend=auto uv pip install git+https://github.com/vllm-project/vllm-omni.git ``` ## Launch the server The deploy config (`vllm_omni/deploy/fish_omni.yaml`) is picked up automatically from `model_type=fish_omni`: ```bash vllm serve fishaudio/drama-3 --omni --port 8000 ``` The first request after start compiles kernels and takes about 10 s; send a few warmup requests before measuring latency. ## Client usage ### Voice cloning (reference audio + transcript) ```bash curl -X POST http://localhost:8000/v1/audio/speech \ -H "Content-Type: application/json" \ -d '{ "model": "fishaudio/drama-3", "input": "The quarterly report is due on Thursday, so please finish it by noon.", "voice": "default", "ref_audio": "https://huggingface.co/datasets/zhaochenyang20/seed-tts-eval-mini/resolve/main/en/prompt-wavs/common_voice_en_10119832.wav", "ref_text": "We asked over twenty different people, and they all said it was his.", "response_format": "wav" }' --output cloned.wav ``` ### Streaming Add `"stream": true, "response_format": "pcm"` to receive Server-Sent Events whose `speech.audio.delta` events carry base64 16-bit PCM at 44.1 kHz. ### Event and emotion tags Non-verbal events and delivery are controlled only through the input text. Write a short natural-language descriptor in square brackets where the event should happen: ```text [sighs] Fine. If we miss it, it's on you. [laughs] Well, maybe that's fair. ``` - There is no fixed tag list. Tags are ordinary text (no special tokens), so any short descriptor is accepted; wordings close to the ones Fish Audio documents work best: `[sighs]`, `[laughs]`, `[laughing]`, `[whisper]`, `[angry]`, `[excited]`, `[pause]`. - Speaker tags `<|speaker:0|>` … `<|speaker:15|>` are the only special control tokens: `"<|speaker:0|> Hi. <|speaker:1|> Hello."`. Text without a leading speaker tag is spoken by speaker 0. - The reference clip also shapes delivery implicitly (an expressive or pause-heavy reference carries over), but it is not a targeted control. - A tag or filler word at the very start of a line (`[sighs] …`, `Yeah. …`) more often renders the first word or event and then only silence until `max_new_tokens`; mid-sentence tags are more reliable. ## Performance 1x H100 80 GB, both stages on one GPU, streaming `/v1/audio/speech`, one voice-clone reference, temperature 1.0 / top_p 0.9 / top_k 30, warm server: | Concurrency | First audio (p50) | RTF | |---:|---:|---:| | 1 | 126 ms | 0.28 | | 4 | 246 ms | 0.40 | | 8 | 360 ms | 0.51 | Voice cloning, 5 public Common Voice reference voices x 4 English sentences: Whisper-large-v3 WER 0.000. ## Known limitations - Output is **44.1 kHz mono**; voice cloning requires **both** `ref_audio` and `ref_text`. - Some takes finish speaking and then keep emitting silence until `max_new_tokens`, most often on prompts that start with fillers or event tags; cap `max_new_tokens` to the expected length of the text. - Editing as continuation (`previous_takes`) and long-text chunking are not exposed through the speech API yet. ## References - [Model card](https://huggingface.co/fishaudio/drama-3) - vLLM-Omni support: vllm-project/vllm-omni [#8900](https://github.com/vllm-project/vllm-omni/pull/8900) --- # Qwen/Qwen3.6-27B on vLLM Source: https://recipes.vllm.ai/Qwen/Qwen3.6-27B Provider: Qwen Released: 2026-04-21T07:50:43.000Z Updated: 2026-10-12 Architecture: dense Parameters: 27B Context length: 262,144 Min vLLM: 0.17.0 Tasks: multimodal, text Verified hardware: DGX Spark (GB10), RTX Pro 6000, DGX Station (GB300), RTX 5090, Ascend 950DT, Ascend 910C, Arc Pro B60, Arc Pro B70, MI300X Summary: Qwen3.6 dense multimodal model (27B) with gated delta networks hybrid attention, MTP, and 262K context ## Variants - default: precision=bf16, vram_minimum_gb=65 Full precision BF16 — fits on 1x H200, 2x H100, 4x Intel Arc Pro B60, 2x Intel Arc Pro B70, or an Ascend A3 node (TP2+DP8) - fp8: precision=fp8, vram_minimum_gb=33, model_id=Qwen/Qwen3.6-27B-FP8 Qwen official FP8 checkpoint — single 40 GB GPU - nvfp4: precision=nvfp4, vram_minimum_gb=27, model_id=nvidia/Qwen3.6-27B-NVFP4 NVIDIA ModelOpt NVFP4 checkpoint — MLP linears quantized to NVFP4, attention linears and KV cache in FP8. Single Blackwell GPU, including DGX Spark. - redhat_fp8: precision=fp8, vram_minimum_gb=33, model_id=RedHatAI/Qwen3.6-27B-FP8 RedHatAI FP8 dynamic quantization - ascend_w8a8_mxfp8: precision=mxfp8, vram_minimum_gb=34, model_id=Eco-Tech/Qwen3.6-27B-w8a8-mxfp8 - ascend_w8a8: precision=int8, vram_minimum_gb=33, model_id=Eco-Tech/Qwen3.6-27B-w8a8 ModelSlim INT8 W8A8 checkpoint for one Atlas 800 A3 node (16x Ascend 910C), using TP2+DP8. Requires --quantization ascend. ## Recommended base flags ```bash vllm serve 'Qwen/Qwen3.6-27B' \ --trust-remote-code ``` ## Guide ## Overview [Qwen3.6-27B](https://huggingface.co/Qwen/Qwen3.6-27B) is the flagship dense model of the Qwen3.6 family. It uses the same gated delta networks hybrid attention as its MoE siblings, supports vision+text input, and natively serves 262K context. MTP (multi-token prediction) is supported out of the box for low-latency decoding. ## Prerequisites - **vLLM version:** >= 0.17.0 - **NVFP4 vLLM version:** >= 0.28.0 (currently nightly). 0.24.0 loads the checkpoint, but FlashInfer only selects its XQA decode kernel on SM120 GPUs from 0.28.0. - **Hardware (BF16):** 1x H200, 2x H100, 4x Intel Arc Pro B60, 2x Intel Arc Pro B70 - **Hardware (FP8):** single 40 GB GPU (H100/H200/L40S) - **Hardware (Int4):** single 24 GB GPU - **Hardware (NVFP4):** single NVIDIA Blackwell GPU (B200/B300/GB10 DGX Spark) - **Hardware (AMD):** 1x MI300X (BF16 fits one 192 GB GPU) ### Install vLLM ```bash uv venv source .venv/bin/activate uv pip install -U vllm --torch-backend=auto ``` ### Docker ```bash docker pull vllm/vllm-openai-xpu:latest # Intel XPU (B60 / B70) ``` ## Launching the Server ### Single-GPU FP8 ```bash vllm serve Qwen/Qwen3.6-27B-FP8 \ --max-model-len 262144 \ --reasoning-parser qwen3 ``` ### BF16 on 2xH100 (TP2) ```bash vllm serve Qwen/Qwen3.6-27B \ --tensor-parallel-size 2 \ --max-model-len 262144 \ --reasoning-parser qwen3 ``` ### MTP speculative decoding ```bash vllm serve Qwen/Qwen3.6-27B-FP8 \ --speculative-config '{"method": "mtp", "num_speculative_tokens": 1}' \ --reasoning-parser qwen3 ``` ### Text-only (skip vision encoder) ```bash vllm serve Qwen/Qwen3.6-27B-FP8 \ --language-model-only \ --reasoning-parser qwen3 \ --enable-prefix-caching ``` ### NVFP4 on Blackwell The [`nvidia/Qwen3.6-27B-NVFP4`](https://huggingface.co/nvidia/Qwen3.6-27B-NVFP4) checkpoint is NVIDIA's ModelOpt re-quantization: MLP linears drop to NVFP4 (W4A16) while the attention linears and KV cache stay FP8, so the ~22 GB weights fit a single Blackwell GPU. vLLM auto-detects the ModelOpt quantization from the checkpoint, so no `--quantization` flag is needed, but it does need vLLM 0.28.0+ (currently nightly) for FlashInfer's XQA decode path on SM120. NVIDIA reports near-lossless accuracy versus the FP8 baseline. Select the NVFP4 variant in the command builder above and switch the hardware pill for the validated single-node launch on DGX Spark (GB10), DGX Station (GB300), RTX PRO 6000 or RTX 5090. Memory utilization, context length, batch size, attention backend and MTP depth all differ per box and are encoded per hardware, so the generated command is the tested one. #### **2× DGX Spark (TP2)** Two Sparks joined over ConnectX is the one layout the builder can't render, so it is spelled out here. Download the checkpoint on all nodes. Start the worker node first and then the head node — the head coordinates the others, so they need to be ready to connect when it comes up. ```bash # COMMON variables example values — same on all nodes export HEAD_IP="" # The ConnectX IP of Node1 (Head Node) export ETH_IF="enp1s0f1np1" # CX7 ethernet interface which is UP export IB_IF="rocep1s0f1,roceP2p1s0f1" # BOTH matching RoCE interfaces export MASTER_PORT=29501 # Free port on the Head Node export CONTAINER_NAME=vllm_node export IMAGE="vllm/vllm-openai:nightly" ``` ```bash # Head Node docker run --privileged --ulimit nofile=1048576:1048576 --ipc=host \ --gpus all --rm --network host --name "$CONTAINER_NAME" --entrypoint="" \ -e MN_IF_NAME="$ETH_IF" \ -e UCX_NET_DEVICES="$ETH_IF" \ -e NCCL_SOCKET_IFNAME="$ETH_IF" \ -e NCCL_IB_HCA="$IB_IF" \ -e NCCL_IB_DISABLE=0 \ -e OMPI_MCA_btl_tcp_if_include="$ETH_IF" \ -e GLOO_SOCKET_IFNAME="$ETH_IF" \ -e TP_SOCKET_IFNAME="$ETH_IF" \ -e NCCL_IGNORE_CPU_AFFINITY=1 \ -e VLLM_FLOAT32_MATMUL_PRECISION=high \ -e HEAD_IP="$HEAD_IP" \ -e MASTER_PORT="$MASTER_PORT" \ -v "$HOME/.cache/huggingface:/root/.cache/huggingface" \ "$IMAGE" vllm serve nvidia/Qwen3.6-27B-NVFP4 \ --host 0.0.0.0 \ --port 8000 \ --trust-remote-code \ --kv-cache-dtype fp8 \ --attention-backend flashinfer \ --gpu-memory-utilization 0.5 \ --max-model-len 262144 \ --max-num-seqs 8 \ --max-num-batched-tokens 8192 \ --enable-chunked-prefill \ --async-scheduling \ --enable-prefix-caching \ --speculative-config '{"method":"mtp","num_speculative_tokens":5,"draft_tensor_parallel_size":2}' \ --load-format fastsafetensors \ --reasoning-parser qwen3 \ --tool-call-parser qwen3_coder \ --enable-auto-tool-choice \ --tensor-parallel-size 2 \ --nnodes 2 \ --node-rank 0 \ --master-addr "$HEAD_IP" \ --master-port "$MASTER_PORT" # Worker node docker run --privileged --ulimit nofile=1048576:1048576 --ipc=host \ --gpus all --rm --network host --name "$CONTAINER_NAME" --entrypoint="" \ -e MN_IF_NAME="$ETH_IF" \ -e UCX_NET_DEVICES="$ETH_IF" \ -e NCCL_SOCKET_IFNAME="$ETH_IF" \ -e NCCL_IB_HCA="$IB_IF" \ -e NCCL_IB_DISABLE=0 \ -e OMPI_MCA_btl_tcp_if_include="$ETH_IF" \ -e GLOO_SOCKET_IFNAME="$ETH_IF" \ -e TP_SOCKET_IFNAME="$ETH_IF" \ -e NCCL_IGNORE_CPU_AFFINITY=1 \ -e VLLM_FLOAT32_MATMUL_PRECISION=high \ -e HEAD_IP="$HEAD_IP" \ -e MASTER_PORT="$MASTER_PORT" \ -v "$HOME/.cache/huggingface:/root/.cache/huggingface" \ "$IMAGE" vllm serve nvidia/Qwen3.6-27B-NVFP4 \ --host 0.0.0.0 \ --port 8000 \ --trust-remote-code \ --kv-cache-dtype fp8 \ --attention-backend flashinfer \ --gpu-memory-utilization 0.5 \ --max-model-len 262144 \ --max-num-seqs 8 \ --max-num-batched-tokens 8192 \ --enable-chunked-prefill \ --async-scheduling \ --enable-prefix-caching \ --speculative-config '{"method":"mtp","num_speculative_tokens":5,"draft_tensor_parallel_size":2}' \ --load-format fastsafetensors \ --reasoning-parser qwen3 \ --tool-call-parser qwen3_coder \ --enable-auto-tool-choice \ --tensor-parallel-size 2 \ --nnodes 2 \ --node-rank 1 \ --master-addr "$HEAD_IP" \ --master-port "$MASTER_PORT" \ --headless ``` ### AMD (MI300X, single GPU, BF16) BF16 fits one MI300X. Two settings matter on ROCm and neither is on by default: `VLLM_ROCM_USE_AITER=1` and an explicit attention backend. The auto-selected `ROCM_ATTN` cannot use its native kernel for this model and its decode slows sharply as the prompt grows; `ROCM_AITER_FA` stays flat. ```bash VLLM_ROCM_USE_AITER=1 vllm serve Qwen/Qwen3.6-27B \ --attention-backend ROCM_AITER_FA \ --enable-auto-tool-choice --tool-call-parser qwen3_coder \ --reasoning-parser qwen3 ``` MTP works with the same flags (add the **Speculative decoding** feature above). Single-stream decode (tokens/s at ~200 / ~2k / ~12k / ~32k prompt tokens, 512 output tokens, prefix caching on, which is the default), measured 2026-10-12 on one MI300X with `vllm/vllm-openai-rocm:v0.31.0`: | | MI300X | |---|---| | image defaults (`ROCM_ATTN`) | 66 / 56 / 27 / 13 | | AITER + `ROCM_AITER_UNIFIED_ATTN` | 66 / 66 / 60 / 49 | | AITER + `ROCM_AITER_FA` | 67 / 67 / 65 / 62 | | + MTP (`num_speculative_tokens: 3`) | 139 / 176 / 168 / 157 | The synthetic prompts are repetitive, which flatters MTP; on real coding-agent requests (10-25k-token contexts) decode averaged 27 tok/s with image defaults, 68 with `ROCM_AITER_FA` and 161 with MTP added. GSM8K (first 250 test questions, greedy, thinking off) is unchanged: 0.952 image defaults, 0.952 `ROCM_AITER_FA`, 0.964 `ROCM_AITER_UNIFIED_ATTN`, 0.964 `ROCM_AITER_FA` + MTP. ### Docker (Intel XPU B60 / B70) Validated on 4× Intel Arc Pro B60 or 2x Intel Arc Pro B70 (B60 24 GB, B70 32 GB per card) with the official vLLM XPU image `vllm/vllm-openai-xpu:latest`. ```bash docker run --device /dev/dri \ -v /dev/dri/by-path:/dev/dri/by-path --shm-size=16g \ --privileged --ipc=host -p 8000:8000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ vllm/vllm-openai-xpu:latest \ Qwen/Qwen3.6-27B \ --reasoning-parser qwen3 \ --tensor-parallel-size 4 \ --max-model-len 8192 \ --enforce-eager ``` ## Processing Ultra-Long Texts Qwen3.6-27B natively supports `262,144` tokens. To serve longer inputs, turn on the **Long Context** feature above: it applies the model card's YaRN RoPE configuration via `--hf-overrides` and raises `--max-model-len` to ~1,010,000 tokens. Static YaRN holds the scaling factor constant regardless of input length, so leave it off unless you actually serve long prompts; for a ~524K workload, halve the card's `factor` to `2.0`. See the [model card](https://huggingface.co/Qwen/Qwen3.6-27B#processing-ultra-long-texts) for the full parameter reference. ## Client Usage ```python from openai import OpenAI client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1") resp = client.chat.completions.create( model="Qwen/Qwen3.6-27B", messages=[{"role": "user", "content": "Write a haiku about gated delta networks."}], max_tokens=256, ) print(resp.choices[0].message.content) ``` ## Troubleshooting - **CUDA graph / Mamba cache size error:** reduce `--max-cudagraph-capture-size` (default 512). See [vLLM PR #34571](https://github.com/vllm-project/vllm/pull/34571). - **Disable reasoning:** add `--default-chat-template-kwargs '{"enable_thinking": false}'`. - **Prefix Caching (Mamba):** currently experimental in "align" mode. ## References - [Model card](https://huggingface.co/Qwen/Qwen3.6-27B) - [FP8 checkpoint](https://huggingface.co/Qwen/Qwen3.6-27B-FP8) - [GPTQ-Int4 checkpoint](https://huggingface.co/Qwen/Qwen3.6-27B-GPTQ-Int4) - [NVIDIA NVFP4 checkpoint](https://huggingface.co/nvidia/Qwen3.6-27B-NVFP4) - [Ascend W8A8 checkpoint](https://www.modelscope.cn/models/Eco-Tech/Qwen3.6-27B-w8a8-mxfp8) - [Ascend A3 W8A8 checkpoint](https://www.modelscope.cn/models/Eco-Tech/Qwen3.6-27B-w8a8) - [vLLM Ascend tutorial](https://docs.vllm.ai/projects/ascend/en/latest/tutorials/models/Qwen3.5-27B-Qwen3.6-27B.html) - [Qwen3.6-35B-A3B recipe](../Qwen3.6-35B-A3B) --- # XiaomiMiMo/MiMo-V2-Flash on vLLM Source: https://recipes.vllm.ai/XiaomiMiMo/MiMo-V2-Flash Provider: MiMo (Xiaomi) Released: 2025-12-16T08:47:02.000Z Updated: 2026-10-08 Architecture: moe Parameters: 309B Context length: 262,144 Min vLLM: 0.20.0 Tasks: text Verified hardware: H200, MI300X, MI325X, MI355X Summary: Xiaomi's MoE reasoning model (309B total / 15B active) with hybrid attention and MTP for fast agentic workflows ## Variants - default: precision=fp8, vram_minimum_gb=371 Native FP8 weights; 4x H200 recommended with TP4 ## Recommended base flags ```bash vllm serve 'XiaomiMiMo/MiMo-V2-Flash' \ --trust-remote-code \ --generation-config \ vllm ``` ## Guide ## Overview [MiMo-V2-Flash](https://huggingface.co/XiaomiMiMo/MiMo-V2-Flash) is a MoE language model with 309B total parameters and 15B active. Designed for high-speed reasoning and agentic workflows, it features hybrid attention and Multi-Token Prediction (MTP) to reduce inference cost. ## Prerequisites - Hardware: 4x H200 (TP4) or equivalent aggregate VRAM (~320 GB with FP8) - vLLM >= 0.20.0 ### Install vLLM (NVIDIA) ```bash uv venv source .venv/bin/activate uv pip install vllm --torch-backend auto ``` ### Install vLLM (AMD ROCm MI300X/MI325X/MI355X) ```bash uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/ ``` ## Launch commands Basic TP4: ```bash vllm serve XiaomiMiMo/MiMo-V2-Flash \ --host 0.0.0.0 --port 9001 --seed 1024 \ --served-model-name mimo_v2_flash \ --tensor-parallel-size 4 \ --trust-remote-code \ --gpu-memory-utilization 0.9 \ --generation-config vllm ``` With tool calling + reasoning: ```bash vllm serve XiaomiMiMo/MiMo-V2-Flash \ --tensor-parallel-size 4 --trust-remote-code --gpu-memory-utilization 0.9 \ --tool-call-parser mimo \ --reasoning-parser qwen3 \ --generation-config vllm \ --served-model-name mimo_v2_flash ``` DP + TP + EP: ```bash vllm serve XiaomiMiMo/MiMo-V2-Flash \ --data-parallel-size 2 \ --tensor-parallel-size 4 \ --enable-expert-parallel \ --trust-remote-code \ --gpu-memory-utilization 0.9 \ --generation-config vllm \ --served-model-name mimo_v2_flash ``` AMD: ```bash export VLLM_ROCM_USE_AITER=0 vllm serve XiaomiMiMo/MiMo-V2-Flash --tensor-parallel-size 4 \ --trust-remote-code --gpu-memory-utilization 0.9 --generation-config vllm ``` Tunable flags: - `--max-model-len=65536` works well; max is 128K. - `--max-num-batched-tokens=32768` for prompt-heavy; 16K/8K for lower latency. - `--gpu-memory-utilization=0.95` to maximize KV cache. ## Client Usage ```bash curl -X POST http://localhost:9001/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "mimo_v2_flash", "messages": [{"role": "user", "content": "Hello MiMo!"}], "chat_template_kwargs": {"enable_thinking": true} }' ``` Set `"enable_thinking": false` (or omit the kwargs) to disable thinking mode. ## Benchmarking ```bash vllm bench serve \ --model XiaomiMiMo/MiMo-V2-Flash \ --dataset-name random --random-input-len 8000 --random-output-len 1000 \ --request-rate 3 --num-prompts 1800 --ignore-eos ``` ## Accuracy (GSM8K) Reported 5-shot `exact_match`: flexible 0.9128, strict 0.9075. ## References - [MiMo-V2-Flash on Hugging Face](https://huggingface.co/XiaomiMiMo/MiMo-V2-Flash) --- # nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16 on vLLM Source: https://recipes.vllm.ai/nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16 Provider: NVIDIA Released: 2026-03-10T18:32:14.000Z Updated: 2026-10-08 Architecture: moe Parameters: 120B Context length: 262,144 Min vLLM: 0.17.1 Tasks: text Verified hardware: H100, H200, B200, DGX Station (GB300), DGX Spark (GB10), RTX Pro 6000 Summary: NVIDIA Nemotron-3-Super Mamba-hybrid latent-MoE (~120B total / ~12B active) with BF16, FP8, and NVFP4 variants ## Variants - default: precision=bf16, vram_minimum_gb=298 BF16 weights (FP8 KV cache recommended) - fp8: precision=fp8, vram_minimum_gb=149, model_id=nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-FP8 FP8 weights + FP8 KV cache - nvfp4: precision=nvfp4, vram_minimum_gb=75, model_id=nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4 NVFP4 weights for Blackwell - base_bf16: precision=bf16, vram_minimum_gb=298, model_id=nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-Base-BF16 Pre-RL base checkpoint (BF16) ## Recommended base flags ```bash vllm serve 'nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16' \ --trust-remote-code ``` ## Guide ## Overview NVIDIA Nemotron-3-Super-120B-A12B is a hybrid-Mamba latent-MoE model (~120B total, ~12B active per token) trained for general reasoning, tool use, and agentic workflows. It supports a 1M-token context window and Multi-Token Prediction (MTP). Variants ship in BF16, FP8, and NVFP4 (Blackwell). A pre-RL Base BF16 checkpoint is also available for downstream fine-tuning. ## Prerequisites - Hardware: 4-8x H100/H200/B200/RTX Pro 6000, DGX Station, or DGX Spark - vLLM >= 0.17.1 - Docker with NVIDIA Container Toolkit (recommended) ## Launch commands Reference command from the [vLLM blog](https://vllm.ai/blog/nemotron-3-super) (BF16, 4x H100, FP8 KV cache): ```bash vllm serve nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16 \ --kv-cache-dtype fp8 \ --tensor-parallel-size 4 \ --trust-remote-code \ --served-model-name nemotron \ --enable-auto-tool-choice \ --tool-call-parser qwen3_coder \ --reasoning-parser nemotron_v3 ``` FP8 weights: ```bash vllm serve nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-FP8 \ --kv-cache-dtype fp8 \ --tensor-parallel-size 4 \ --trust-remote-code \ --enable-auto-tool-choice \ --tool-call-parser qwen3_coder \ --reasoning-parser nemotron_v3 ``` NVFP4 (Blackwell only): ```bash vllm serve nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4 \ --tensor-parallel-size 2 \ --trust-remote-code \ --enable-auto-tool-choice \ --tool-call-parser qwen3_coder \ --reasoning-parser nemotron_v3 ``` ### DGX Spark (GB10) The NVFP4 variant runs on a single DGX Spark (GB10) at TP=1 inside the `vllm/vllm-openai:v0.28.0-ubuntu2404` container. The command below is the optimal config tuned end-to-end on DGX Spark: the full 262144-token context fits with an FP8 KV cache and a float32 Mamba SSM cache (`VLLM_ALLOW_LONG_MAX_MODEL_LEN=1` allows the long context, `VLLM_FLOAT32_MATMUL_PRECISION=high` keeps the Mamba scan numerically stable), `--gpu-memory-utilization 0.8` leaves headroom in the 128 GB unified memory, batching is sized for a single GB10 (`--max-num-seqs 8`, `--max-num-batched-tokens 16384`), `--load-format fastsafetensors` speeds up weight loading, and MTP speculative decoding plus prefix caching are enabled for lower latency: ```bash # Use container: vllm/vllm-openai:v0.28.0-ubuntu2404 VLLM_FLOAT32_MATMUL_PRECISION="high" \ VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 \ vllm serve nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4 \ --host 0.0.0.0 \ --port 8000 \ --max-model-len 262144 \ --gpu-memory-utilization 0.8 \ --max-num-seqs 8 \ --load-format fastsafetensors \ --attention-backend flashinfer \ --max-num-batched-tokens 16384 \ --kv-cache-dtype fp8 \ --mamba-cache-mode align \ --mamba-ssm-cache-dtype float32 \ --speculative-config '{"method":"mtp","num_speculative_tokens":3}' \ --enable-prefix-caching \ --reasoning-parser nemotron_v3 \ --enable-auto-tool-choice \ --tool-call-parser qwen3_coder ``` ### RTX Pro 6000 NVFP4 The NVFP4 variant runs on a single RTX Pro 6000 at TP=1. Requires vLLM v0.24.0 or later. A few notes: - **Set `--gpu-memory-utilization 0.92`** to avoid an OOM at startup. If you encounter an OOM, try adjusting the value. - **Set `VLLM_FLOAT32_MATMUL_PRECISION="high"`** to use high precision for the matmul operations. - **Set `VLLM_ALLOW_LONG_MAX_MODEL_LEN=1`** to allow the model to use the full context length. ```bash # Use container: vllm/vllm-openai:v0.28.0-ubuntu2404 VLLM_FLOAT32_MATMUL_PRECISION="high" \ VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 \ vllm serve nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4 \ --max-model-len 262144 \ --gpu-memory-utilization 0.92 \ --max-num-seqs 8 \ --max-num-batched-tokens 16384 \ --kv-cache-dtype fp8 \ --mamba-cache-mode align \ --mamba-ssm-cache-dtype float32 \ --speculative-config '{"method":"mtp","num_speculative_tokens":3}' \ --enable-prefix-caching \ --reasoning-parser nemotron_v3 \ --enable-auto-tool-choice \ --tool-call-parser qwen3_coder ``` ## Benchmarking ```bash vllm bench serve \ --model nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-FP8 \ --trust-remote-code \ --dataset-name random \ --random-input-len 1024 --random-output-len 1024 \ --num-warmups 20 \ --ignore-eos \ --max-concurrency 1024 \ --num-prompts 2048 ``` ## References - [vLLM blog: Nemotron-3-Super](https://vllm.ai/blog/nemotron-3-super) - [vLLM blog: vLLM on DGX Spark](https://vllm.ai/blog/2026-06-01-vllm-dgx-spark) - [Nemotron-3-Super-120B-A12B-BF16](https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16) - [Nemotron-3-Super-120B-A12B-FP8](https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-FP8) - [Nemotron-3-Super-120B-A12B-NVFP4](https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4) - [Nemotron-3-Super-120B-A12B-Base-BF16](https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-Base-BF16) --- # nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16 on vLLM Source: https://recipes.vllm.ai/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16 Provider: NVIDIA Released: 2026-08-01T06:55:21.000Z Updated: 2026-10-08 Architecture: moe Parameters: 30B Context length: 262,144 Min vLLM: 0.27.1 Tasks: text Verified hardware: H100, DGX Spark (GB10), RTX Pro 6000, DGX Station (GB300) Summary: NVIDIA Nemotron 3.5 Lightning hybrid Mamba-MoE (30B total / 3B active) with NVFP4 and BF16 checkpoints, 1M context, and MTP / DSpark / DFlash speculative decoding ## Variants - default: precision=nvfp4, vram_minimum_gb=18, model_id=nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 ModelOpt NVFP4 (W4A16) checkpoint with FP8 KV cache — the default deployment target, and the only one the DSpark/DFlash drafts are published for - bf16: precision=bf16, vram_minimum_gb=72, model_id=nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16 Full-precision BF16 reference checkpoint — MTP only (no DSpark/DFlash draft is published for BF16) ## Recommended base flags ```bash vllm serve 'nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16' \ --mamba-backend \ flashinfer \ --mamba-cache-mode \ align \ --enable-prefix-caching \ --max-num-batched-tokens \ 16384 ``` ## Guide ## Overview Nemotron 3.5 Lightning is a hybrid Mamba-MoE model — 30B total parameters, 3B active, up to 1M context, distilled from Nemotron 3 Ultra and trained for popular agent harnesses. It ships as an NVFP4 (W4A16) ModelOpt checkpoint, the default deployment target, and a BF16 reference checkpoint. ## Prerequisites - **vLLM:** 0.27.1 or newer — `vllm/vllm-openai:v0.28.0` - **Hardware:** 1x H100, 1x DGX Spark (GB10), or 1x DGX Station (GB300) - **Speculative decoding:** **MTP** (built into the checkpoint), **DSpark** (hybrid autoregressive + diffusion) and **DFlash** (diffusion) are all supported. DSpark gives the best latency on H100 and DGX Spark; for maximum throughput, serve without speculative decoding. ## Client Usage Once the vLLM server is running, consume it via the OpenAI-compatible API. Reasoning is returned separately from the answer, on `message.reasoning`: ```python from openai import OpenAI # Set this to match the --served-model-name used when starting the server SERVED_MODEL_NAME = "nemotron-3.5-lightning" BASE_URL = "http://127.0.0.1:8000/v1" client = OpenAI(base_url=BASE_URL, api_key="null") response = client.chat.completions.create( model=SERVED_MODEL_NAME, messages=[ {"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Briefly explain: what is vLLM and why is it useful for large model inference?"}, ], temperature=1.0, top_p=0.95, max_tokens=2048, ) choice = response.choices[0] print("Reasoning:", choice.message.reasoning) print("Content:", choice.message.content) ``` Reasoning tokens count toward `max_tokens`. If `content` comes back empty or `None`, the reasoning trace consumed the entire budget before the model produced an answer — raise `max_tokens`. ## Notes - **Speculative decoding**: DSpark is preferred on DGX Spark. On DGX Station, prefer MTP — choose it in the Speculative Decoding panel above. - **DGX Spark (GB10)**: the selectable config is the one measured on 2026-08-31 with aiperf 0.12.0 on vLLM `0.28.1rc1.dev130+g44fe2a392` — DSpark drafting 7 tokens at `--gpu-memory-utilization 0.7`, `--mamba-cache-mode align` and fastsafetensors loading, served from `vllm/vllm-openai:nightly`. On SPEED-Bench at 8K input / 256 output, concurrency 1, it reached 7,252 prefill tok/s and 93.3 decode tok/s at a 3.98 s mean end-to-end latency. --- # Qwen/Qwen3.5-122B-A10B on vLLM Source: https://recipes.vllm.ai/Qwen/Qwen3.5-122B-A10B Provider: Qwen Released: 2026-02-24T09:43:37.000Z Updated: 2026-10-07 Architecture: moe Parameters: 122B Context length: 262,144 Min vLLM: 0.17.0 Tasks: multimodal, text Verified hardware: MI325X, MI355X Summary: Mid-size Qwen3.5 multimodal MoE (122B total / 10B active) with gated delta networks, 256 experts, and 262K context ## Variants - default: precision=bf16, vram_minimum_gb=293 Full precision BF16 — requires 4x H200 or equivalent - fp8: precision=fp8, vram_minimum_gb=147, model_id=Qwen/Qwen3.5-122B-A10B-FP8 Qwen official FP8 checkpoint — fits on 2x H200 - nvfp4: precision=nvfp4, vram_minimum_gb=74, model_id=RedHatAI/Qwen3.5-122B-A10B-NVFP4 RedHatAI compressed-tensors NVFP4 quantization for Blackwell GPUs - gptq_int4: precision=int4, vram_minimum_gb=74, model_id=Qwen/Qwen3.5-122B-A10B-GPTQ-Int4 GPTQ Int4 checkpoint — single-GPU serving on 80GB hardware - redhat_fp8: precision=fp8, vram_minimum_gb=147, model_id=RedHatAI/Qwen3.5-122B-A10B-FP8-dynamic RedHatAI FP8 (E4M3) with dynamic per-token activation quantization ## Recommended base flags ```bash vllm serve 'Qwen/Qwen3.5-122B-A10B' \ --trust-remote-code ``` ## Guide ## Overview [Qwen3.5-122B-A10B](https://huggingface.co/Qwen/Qwen3.5-122B-A10B) is a mid-tier member of the Qwen3.5 family, sharing the gated delta networks MoE architecture with 122B total parameters and 10B activated per token (256 experts). It is multimodal (vision + text) and natively supports 262K context. ## Prerequisites - **vLLM version:** >= 0.17.0 - **Hardware (BF16):** 4x H200 or 8x H100 - **Hardware (FP8):** 2x H200 or 4x H100 - **Hardware (Int4):** single 80 GB GPU ### Install vLLM ```bash uv venv source .venv/bin/activate uv pip install -U vllm --torch-backend=auto ``` ## Launching the Server ### BF16 on 4xH200 (TP4) ```bash vllm serve Qwen/Qwen3.5-122B-A10B \ --tensor-parallel-size 4 \ --max-model-len 262144 \ --reasoning-parser qwen3 ``` ### FP8 on 2xH200 (TP2) ```bash vllm serve Qwen/Qwen3.5-122B-A10B-FP8 \ --tensor-parallel-size 2 \ --max-model-len 262144 \ --reasoning-parser qwen3 ``` ### Throughput-focused (text-only, EP) ```bash vllm serve Qwen/Qwen3.5-122B-A10B-FP8 \ -dp 4 --enable-expert-parallel \ --language-model-only \ --reasoning-parser qwen3 \ --enable-prefix-caching ``` ### MTP speculative decoding ```bash vllm serve Qwen/Qwen3.5-122B-A10B-FP8 \ --tensor-parallel-size 2 \ --speculative-config '{"method": "mtp", "num_speculative_tokens": 1}' \ --reasoning-parser qwen3 ``` ### AMD (MI325X / MI355X, single GPU, FP8) The FP8 checkpoint fits one MI300X-class GPU. Three settings matter on ROCm and none is on by default: `VLLM_ROCM_USE_AITER=1` switches the MoE to the AITER FP8 kernels, `--attention-backend ROCM_AITER_FA` keeps decode flat across context (the auto-selected `ROCM_ATTN` decays with prompt length on this hybrid GDN model), and `--kv-cache-dtype fp8` keeps MTP flat across context too. ```bash VLLM_ROCM_USE_AITER=1 vllm serve Qwen/Qwen3.5-122B-A10B-FP8 \ --attention-backend ROCM_AITER_FA \ --kv-cache-dtype fp8 --max-num-seqs 64 \ --reasoning-parser qwen3 ``` `--max-num-seqs 64` because the default exceeds the Mamba block budget on 192 GB. MTP (`--speculative-config '{"method":"mtp","num_speculative_tokens":1}'`) works with the same flags. Single-stream decode (tokens/s at ~200 / ~2k / ~12k / ~30k prompt tokens, 128 output tokens, prefix caching off), measured 2026-10-07 on `vllm/vllm-openai-rocm:v0.31.0`: | | MI325X | MI355X | |---|---|---| | AITER + `ROCM_AITER_FA` | 128 / 127 / 127 / 127 | 143 / 143 / 143 / 143 | | + MTP | 184 / 169 / 172 / 164 | 204 / 193 / 195 / 190 | This recipe used `ROCM_AITER_UNIFIED_ATTN` before v0.31.0. Avoid it on MI350X/MI355X with v0.31.0: there it runs AITER's new gfx950 Gluon kernel, which decodes at 110 tokens/s and, with MTP, falls from 200 to 72 tokens/s between ~200 and ~30k prompt tokens ([vllm-project/vllm#60408](https://github.com/vllm-project/vllm/issues/60408)). MI325X is unaffected (126 tokens/s; 182 to 162 with MTP). ## Client Usage ```python from openai import OpenAI client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1") resp = client.chat.completions.create( model="Qwen/Qwen3.5-122B-A10B", messages=[{"role": "user", "content": "Summarize the gated delta networks paper."}], max_tokens=512, ) print(resp.choices[0].message.content) ``` ## Troubleshooting - **CUDA graph / Mamba cache size error:** reduce `--max-cudagraph-capture-size` (default 512). See [vLLM PR #34571](https://github.com/vllm-project/vllm/pull/34571). - **Disable reasoning:** add `--default-chat-template-kwargs '{"enable_thinking": false}'`. - **Prefix Caching (Mamba):** currently experimental in "align" mode. ## References - [Model card](https://huggingface.co/Qwen/Qwen3.5-122B-A10B) - [FP8 checkpoint](https://huggingface.co/Qwen/Qwen3.5-122B-A10B-FP8) - [GPTQ-Int4 checkpoint](https://huggingface.co/Qwen/Qwen3.5-122B-A10B-GPTQ-Int4) - [Qwen3.5-397B-A17B recipe](../Qwen3.5-397B-A17B) --- # Qwen/Qwen3.5-397B-A17B on vLLM Source: https://recipes.vllm.ai/Qwen/Qwen3.5-397B-A17B Provider: Qwen Released: 2026-02-16T04:55:12.000Z Updated: 2026-10-07 Architecture: moe Parameters: 397B Context length: 262,144 Min vLLM: 0.17.0 Tasks: multimodal, text Verified hardware: H200, GB200 NVL4, MI300X, MI325X, MI350X, MI355X, Ascend 950DT Summary: Multimodal MoE model with gated delta networks architecture, 397B total / 17B active parameters, up to 262K context ## Variants - default: precision=bf16, vram_minimum_gb=953 Full precision BF16 — requires 8x H200 or equivalent - nvfp4: precision=nvfp4, vram_minimum_gb=238, model_id=nvidia/Qwen3.5-397B-A17B-NVFP4 - fp8: precision=fp8, vram_minimum_gb=477, model_id=RedHatAI/Qwen3.5-397B-A17B-FP8-dynamic RedHatAI FP8 (E4M3) with dynamic per-token activation quantization - gptq_int4: precision=int4, vram_minimum_gb=239, model_id=Qwen/Qwen3.5-397B-A17B-GPTQ-Int4 GPTQ Int4 checkpoint — halves VRAM vs FP8 - amd_mxfp4: precision=mxfp4, vram_minimum_gb=287, model_id=amd/Qwen3.5-397B-A17B-MXFP4 AMD Quark MXFP4 checkpoint for MI350X/MI355X (gfx950) — fits one 288 GB GPU - ascend_w8a8_mtp: precision=int8, vram_minimum_gb=477, model_id=Eco-Tech/Qwen3.5-397B-A17B-w8a8-mtp - ascend_w4a4_mxfp4: precision=mxfp4, vram_minimum_gb=239, model_id=Eco-Tech/Qwen3.5-397B-A17B-w4a4-mxfp4 ## Recommended base flags ```bash vllm serve 'Qwen/Qwen3.5-397B-A17B' \ --trust-remote-code ``` Environment: ```bash VLLM_DEEP_GEMM_WARMUP='skip' VLLM_USE_DEEP_GEMM='0' ``` ## Guide ## Overview [Qwen3.5](https://huggingface.co/Qwen/Qwen3.5-397B-A17B) is a multimodal mixture-of-experts model featuring a gated delta networks architecture with 397B total parameters and 17B active parameters. This guide covers how to efficiently deploy and serve the model across different hardware configurations and workload profiles using vLLM. ## Prerequisites ### Pip Install #### NVIDIA ```bash uv venv source .venv/bin/activate uv pip install -U vllm --torch-backend=auto ``` #### AMD > Note: The vLLM wheel for ROCm requires Python 3.12, ROCm 7.0, and glibc >= 2.35. If your environment does not meet these requirements, please use the Docker-based setup. Supported GPUs: MI300X, MI325X, MI355X. ```bash uv venv --python 3.12 source .venv/bin/activate uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm ``` ### Docker #### NVIDIA ```bash docker run --gpus all \ -p 8000:8000 \ --ipc=host \ -v ~/.cache/huggingface:/root/.cache/huggingface \ vllm/vllm-openai Qwen/Qwen3.5-397B-A17B \ --tensor-parallel-size 8 \ --reasoning-parser qwen3 \ --enable-prefix-caching ``` For Blackwell GPUs, use `vllm/vllm-openai:cu130-nightly`. #### AMD ```bash docker run --device=/dev/kfd --device=/dev/dri \ --security-opt seccomp=unconfined \ --group-add video \ --ipc=host \ -p 8000:8000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ vllm/vllm-openai-rocm:latest \ Qwen/Qwen3.5-397B-A17B-FP8 \ --tensor-parallel-size 8 \ --reasoning-parser qwen3 \ --enable-prefix-caching ``` ## Deployment Configurations The configurations below have been verified on 8x H200 GPUs and 8x MI300X/MI355X GPUs. We recommend using the official FP8 checkpoint [Qwen/Qwen3.5-397B-A17B-FP8](https://huggingface.co/Qwen/Qwen3.5-397B-A17B-FP8) for optimal serving efficiency. ### Throughput-Focused (Text-Only) For maximum text throughput under high concurrency, use `--language-model-only` to skip loading the vision encoder and free up memory for KV cache, and enable Expert Parallelism. ```bash vllm serve Qwen/Qwen3.5-397B-A17B-FP8 \ -dp 8 \ --enable-expert-parallel \ --language-model-only \ --reasoning-parser qwen3 \ --enable-prefix-caching ``` ### Throughput-Focused (Multimodal) For multimodal workloads, use `--mm-encoder-tp-mode data` for data-parallel vision encoding and `--mm-processor-cache-type shm` for shared-memory caching of preprocessed multimodal inputs. ```bash vllm serve Qwen/Qwen3.5-397B-A17B-FP8 \ -dp 8 \ --enable-expert-parallel \ --mm-encoder-tp-mode data \ --mm-processor-cache-type shm \ --reasoning-parser qwen3 \ --enable-prefix-caching ``` To enable tool calling, add `--enable-auto-tool-choice --tool-call-parser qwen3_coder` to the serve command. ### Latency-Focused For latency-sensitive workloads at low concurrency, enable MTP-1 speculative decoding and disable prefix caching. MTP-1 reduces time-per-output-token (TPOT) with a high acceptance rate, at the cost of lower throughput under load. > Note (AMD): MTP runs on ROCm with vLLM 0.31.0 (verified with the single-GPU MXFP4 setup below). On ROCm, > pair it with `--kv-cache-dtype fp8`: with the default KV cache its speedup fades as the prompt grows. ```bash vllm serve Qwen/Qwen3.5-397B-A17B-FP8 \ --tensor-parallel-size 8 \ --speculative-config '{"method": "mtp", "num_speculative_tokens": 1}' \ --reasoning-parser qwen3 ``` ### GB200 Deployment We recommend using the NVFP4 checkpoint [nvidia/Qwen3.5-397B-A17B-NVFP4](https://huggingface.co/nvidia/Qwen3.5-397B-A17B-NVFP4) for optimal serving efficiency on GB200 nodes. ```bash vllm serve nvidia/Qwen3.5-397B-A17B-NVFP4 \ -dp 4 \ --enable-expert-parallel \ --language-model-only \ --reasoning-parser qwen3 \ --enable-prefix-caching ``` ### MI355X Deployment ```bash vllm serve Qwen/Qwen3.5-397B-A17B-FP8 \ -tp 2 \ --enable-expert-parallel \ --language-model-only \ --reasoning-parser qwen3 \ --enable-prefix-caching ``` ### Single MI350X / MI355X (MXFP4) AMD's MXFP4 checkpoint [amd/Qwen3.5-397B-A17B-MXFP4](https://huggingface.co/amd/Qwen3.5-397B-A17B-MXFP4) (239 GB) fits one 288 GB MI350X or MI355X: pick the **MXFP4 (W4A4)** variant above. It sets three things vLLM does not pick on its own: `VLLM_ROCM_USE_AITER=1` turns on the AITER kernels; `--attention-backend ROCM_AITER_FA` replaces the auto-selected `ROCM_ATTN`, whose decode slows down as the context grows on this hybrid model; and `--max-num-seqs 64`, because the default (1024) is more than the ~296 Mamba cache blocks a single GPU leaves, and the server would refuse to start. ```bash VLLM_ROCM_USE_AITER=1 vllm serve amd/Qwen3.5-397B-A17B-MXFP4 \ --tensor-parallel-size 1 \ --attention-backend ROCM_AITER_FA \ --max-num-seqs 64 \ --trust-remote-code \ --reasoning-parser qwen3 ``` Single-stream decode in tokens/s at ~200 / ~2k / ~12k / ~30k prompt tokens (128 output tokens), one GPU, `vllm/vllm-openai-rocm:v0.31.0`: | | ~200 | ~2k | ~12k | ~30k | |---|---|---|---|---| | MXFP4, this variant (MI355X) | 119 | 119 | 119 | 117 | | + MTP (3 tokens) + `--kv-cache-dtype fp8` (MI355X) | 219 | 201 | 209 | 188 | | + MTP (3 tokens), default KV cache (MI350X) | 198 | 164 | 125 | 80 | | auto-selected `ROCM_ATTN` instead of `ROCM_AITER_FA` (MI355X) | 116 | 89 | 39 | 19 | | GPTQ-Int4, AITER + `ROCM_AITER_FA` (MI350X) | 75 | 74 | 74 | 74 | With MTP, add `--kv-cache-dtype fp8`: on ROCm, MTP's gain otherwise fades as the prompt grows. The MXFP4 checkpoint quantizes activations as well as weights (W4A4): on the same wikitext-2 tokens vLLM scored PPL 3.43 with it and 3.39 with GPTQ-Int4 (W4A16), so GPTQ-Int4 is the higher-fidelity option at about two thirds of the speed. ## Processing Ultra-Long Texts Qwen3.5-397B-A17B natively supports `262,144` tokens. To serve longer inputs, turn on the **Long Context** feature above: it applies the model card's YaRN RoPE configuration via `--hf-overrides` and raises `--max-model-len` to ~1,010,000 tokens. Static YaRN holds the scaling factor constant regardless of input length, so leave it off unless you actually serve long prompts; for a ~524K workload, halve the card's `factor` to `2.0`. See the [model card](https://huggingface.co/Qwen/Qwen3.5-397B-A17B#processing-ultra-long-texts) for the full parameter reference. ## Client Usage ```python import time from openai import OpenAI client = OpenAI( api_key="EMPTY", base_url="http://localhost:8000/v1", timeout=3600 ) messages = [ { "role": "user", "content": [ { "type": "image_url", "image_url": { "url": "https://ofasys-multimodal-wlcb-3-toshanghai.oss-accelerate.aliyuncs.com/wpf272043/keepme/image/receipt.png" } }, { "type": "text", "text": "Read all the text in the image." } ] } ] start = time.time() response = client.chat.completions.create( model="Qwen/Qwen3.5-397B-A17B", messages=messages, max_tokens=2048 ) print(f"Response costs: {time.time() - start:.2f}s") print(f"Generated text: {response.choices[0].message.content}") ``` ## Troubleshooting **CUDA graph / Mamba cache size error** You may encounter: ``` assert num_cache_lines >= batch ``` This occurs because the CUDA graph capture size is larger than the Mamba cache size. Reduce `--max-cudagraph-capture-size` (default is 512). See https://github.com/vllm-project/vllm/pull/34571 for details. **Configuration tips** - **Disable Reasoning**: Add `--reasoning-parser qwen3 --default-chat-template-kwargs '{"enable_thinking": false}'` to disable reasoning mode via command-line parameters. - **Prefix Caching**: Prefix caching for Mamba cache "align" mode is currently experimental. - **Multi-token Prediction**: MTP-1 reduces per-token latency but degrades throughput under high concurrency because speculative tokens consume KV cache capacity. Adjust `num_speculative_tokens` (1-5) based on your use case. - **Encoder Data Parallelism**: `--mm-encoder-tp-mode data` deploys the vision encoder in a data-parallel fashion. This consumes additional memory and may require adjustment of `--gpu-memory-utilization`. ## References - Model card: https://huggingface.co/Qwen/Qwen3.5-397B-A17B - FP8 checkpoint: https://huggingface.co/Qwen/Qwen3.5-397B-A17B-FP8 - NVFP4 checkpoint: https://huggingface.co/nvidia/Qwen3.5-397B-A17B-NVFP4 - GPTQ-Int4 checkpoint: https://huggingface.co/Qwen/Qwen3.5-397B-A17B-GPTQ-Int4 - AMD MXFP4 checkpoint: https://huggingface.co/amd/Qwen3.5-397B-A17B-MXFP4 - Ascend W8A8 checkpoint: https://www.modelscope.cn/models/Eco-Tech/Qwen3.5-397B-A17B-w8a8-mtp - Ascend W4A4 checkpoint: https://www.modelscope.cn/models/Eco-Tech/Qwen3.5-397B-A17B-w4a4-mxfp4 - vLLM Ascend tutorial: https://docs.vllm.ai/projects/ascend/en/latest/tutorials/models/Qwen3.5-397B-A17B.html - vLLM documentation: https://docs.vllm.ai/ --- # XiaomiMiMo/MiMo-V2.6-Flash-RL on vLLM Source: https://recipes.vllm.ai/XiaomiMiMo/MiMo-V2.6-Flash-RL Provider: MiMo (Xiaomi) Released: 2026-09-21T15:39:51.000Z Updated: 2026-10-07 Architecture: moe Parameters: 309B Context length: 1,048,576 Min vLLM: 0.31.0 Tasks: text, multimodal Verified hardware: H200, GB300 NVL4, MI300X, MI325X, MI355X Summary: Xiaomi's efficiency-balanced omnimodal MoE (309B total / 15B active) with hybrid attention, FP8-compute/mxfp4-stored weights, 1M context, and a DFlash speculative decoder — the default MOPD checkpoint fixes the RL release's tool-call repetition ## Variants - default: precision=fp8, vram_minimum_gb=208, model_id=XiaomiMiMo/MiMo-V2.6-Flash-MOPD FP8 (e4m3) compute with mxfp4-stored MoE weights; 173 GB checkpoint — 4x H200 with TP4. MOPD2-distilled upgrade that fixes tool-call repetition; recommended. - rl: precision=fp8, vram_minimum_gb=208 Same shape as the MOPD variant (173 GB, FP8-compute/mxfp4-stored) — the original RL release, kept for reproducing earlier results. Suffers from tool-call repetition (the more affected of the two sizes); prefer MOPD for agentic workloads. ## Recommended base flags ```bash vllm serve 'XiaomiMiMo/MiMo-V2.6-Flash-RL' \ --trust-remote-code \ --max-model-len \ auto \ --generation-config \ vllm ``` ## Guide ## Overview This recipe serves the MiMo-V2.6-Flash family from two checkpoints on the Variant row: - **MOPD** (default) — [XiaomiMiMo/MiMo-V2.6-Flash-MOPD](https://huggingface.co/XiaomiMiMo/MiMo-V2.6-Flash-MOPD), the MOPD2-distilled upgrade and the recommended starting point for new deployments. It fixes the RL release's most visible agentic failure mode — tool-call repetition, where the model keeps re-issuing the same or highly similar tool calls while making no progress (the blog measures the Flash line as the more affected of the two) — with benchmark quality unchanged, folded into a regular MOPD2 pass via a short specialized-teacher run ([diagnosis blog](https://mimo.xiaomi.com/blog/mimo-v2-6-tool-call-repetition)). - **RL (original)** — [XiaomiMiMo/MiMo-V2.6-Flash-RL](https://huggingface.co/XiaomiMiMo/MiMo-V2.6-Flash-RL), the 2026-09-22 RL release, kept for reproducing earlier results. It can get stuck re-issuing the same tool calls in agentic workloads; prefer **MOPD** unless you are reproducing a specific earlier run. MOPD2 (Technical Report §5.6) is multi-teacher on-policy distillation: domain-specialized mixRL and SFT teachers supervise the student on three on-policy streams (standard MOPD, teacher-prefix OPD, and SFT-prefix OPD). The architecture is shared by both checkpoints: a sparse MoE model with 309B total parameters and 15B active per token — 48 layers (1 dense + 47 MoE) with 256 routed experts (top-8), hybrid attention (sliding-window 128 and global attention), and a 5-layer DFlash-style MTP drafter that predicts 7 tokens per pass. The model is natively omnimodal (text, image, video, audio via a 681M-param MiMo ViT and audio encoders) and supports up to 1M tokens of context. Weights are stored as mxfp4 and computed as FP8 (block-wise e4m3, 128x128), so the checkpoint is only 173 GB on disk. Loading this mixed storage format requires vLLM with the MiMo V2 mxfp4/bf16-router support ([vllm-project/vllm#57784](https://github.com/vllm-project/vllm/pull/57784)), first shipped in the v0.31.0 stable release. ## Prerequisites - Hardware: 4x H200 (TP4), 4x MI355X/MI325X/MI300X (TP4), or equivalent aggregate VRAM (>= 208 GB) ### Pull the vLLM docker image vLLM 0.31.0 is the first stable release that can load the mxfp4-stored weights ([vllm-project/vllm#57784](https://github.com/vllm-project/vllm/pull/57784)); earlier stable releases cannot: ```bash docker pull vllm/vllm-openai:v0.31.0 ``` MI300X / MI325X (gfx942) additionally need the MiMo MXFP4 routing gate ([vllm-project/vllm#58262](https://github.com/vllm-project/vllm/pull/58262)), which merged after the v0.31.0 branch cut — on those GPUs use a nightly wheel built after 2026-09-30 (`uv pip install -U vllm --extra-index-url https://wheels.vllm.ai/nightly/cu130`) or the ROCm nightly image. This release also registers the `mimo` tool-call and reasoning parsers with the Rust frontend ([vllm-project/vllm#57933](https://github.com/vllm-project/vllm/pull/57933)). The Hugging Face model cards point at `vllm/vllm-openai:mimov25-cu129`, the V2.5-era image; that tag predates the mxfp4-stored-weight support linked above, so use the v0.31.0 image recommended above. ## Launch command Single-node TP4 (H200). Selecting the **RL (original)** variant in the builder swaps only the checkpoint id (`XiaomiMiMo/MiMo-V2.6-Flash-RL`); everything else is identical: ```bash vllm serve XiaomiMiMo/MiMo-V2.6-Flash-MOPD \ --tensor-parallel-size 4 \ --trust-remote-code \ --gpu-memory-utilization 0.95 \ --max-model-len auto \ --reasoning-parser mimo \ --tool-call-parser mimo \ --enable-auto-tool-choice \ --generation-config vllm ``` ### AMD (MI355X, gfx950) ```bash VLLM_ROCM_USE_AITER=1 \ VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4 \ vllm serve XiaomiMiMo/MiMo-V2.6-Flash-MOPD \ --tensor-parallel-size 4 \ --trust-remote-code \ --kv-cache-dtype fp8 \ --max-model-len auto \ --reasoning-parser mimo \ --tool-call-parser mimo \ --enable-auto-tool-choice \ --generation-config vllm ``` MI355X also runs TP2 just fine. ### AMD (MI300X / MI325X, gfx942) gfx942 has no native MXFP4 MoE path: without an explicit backend vLLM's oracle falls back to OCP emulation — correct, but about 6.5x lower output throughput. Select AITER's Triton W4A16 kernel instead with `--moe-backend aiter_triton_mxfp4_bf16`; supported since [vllm-project/vllm#58262](https://github.com/vllm-project/vllm/pull/58262) (merged 2026-09-30, after the v0.31.0 branch cut — not in any stable release yet, so this flag needs a nightly wheel; it also needs aiter >= 0.1.20, which the ROCm nightly image ships). Verified end-to-end on MI300X in that PR — see it for the gsm8k and `vllm bench serve` numbers against the emulation baseline: ```bash VLLM_ROCM_USE_AITER=1 \ VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4 \ vllm serve XiaomiMiMo/MiMo-V2.6-Flash-MOPD \ --tensor-parallel-size 2 \ --trust-remote-code \ --max-model-len auto \ --reasoning-parser mimo \ --tool-call-parser mimo \ --enable-auto-tool-choice \ --moe-backend aiter_triton_mxfp4_bf16 \ --generation-config vllm ``` With DFlash speculative decoding (drafter ships inside the checkpoint). vLLM does not resolve a `/dflash` Hub subfolder as the draft model — `model` must be a concrete local path to the `dflash/` directory inside the downloaded snapshot. Resolve it per checkpoint (run `hf download ` first if it is not cached yet): ```bash # MOPD variant DFLASH_DIR=$(ls -d ~/.cache/huggingface/hub/models--XiaomiMiMo--MiMo-V2.6-Flash-MOPD/snapshots/*/dflash) # RL (original) variant # DFLASH_DIR=$(ls -d ~/.cache/huggingface/hub/models--XiaomiMiMo--MiMo-V2.6-Flash-RL/snapshots/*/dflash) echo "$DFLASH_DIR" ``` ```bash vllm serve XiaomiMiMo/MiMo-V2.6-Flash-MOPD \ --tensor-parallel-size 4 \ --trust-remote-code \ --gpu-memory-utilization 0.95 \ --max-model-len auto \ --speculative-config "{\"method\":\"dflash\",\"model\":\"$DFLASH_DIR\",\"num_speculative_tokens\":7}" \ --compilation-config '{"cudagraph_mode":"FULL_DECODE_ONLY"}' \ --reasoning-parser mimo \ --tool-call-parser mimo \ --enable-auto-tool-choice \ --generation-config vllm ``` The drafter inherits the target's tensor-parallel size by default. To pin it explicitly, add `"draft_tensor_parallel_size": 4` to the speculative config (useful if you want the draft on fewer GPUs than the target, or to silence mis-detection on asymmetric topologies). ## Client Usage ```bash curl -X POST http://localhost:8000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "XiaomiMiMo/MiMo-V2.6-Flash-MOPD", "messages": [{"role": "user", "content": "Hello MiMo!"}], "temperature": 1.0, "top_p": 0.95, "chat_template_kwargs": {"enable_thinking": true} }' ``` Recommended sampling: `temperature=1.0`, `top_p=0.95`. Set `"enable_thinking": false` (or omit the kwargs) to disable thinking mode. ## References - [MiMo-V2.6-Flash-MOPD on Hugging Face](https://huggingface.co/XiaomiMiMo/MiMo-V2.6-Flash-MOPD) - [MiMo-V2.6-Flash-RL on Hugging Face](https://huggingface.co/XiaomiMiMo/MiMo-V2.6-Flash-RL) - [MiMo-V2.6 technical report](https://huggingface.co/XiaomiMiMo/MiMo-V2.6-Pro-RL/blob/main/MiMo_V2_6_technical_report.pdf) (§5.6 covers MOPD2) - [Diagnosing and Mitigating Tool-Call Repetition in MiMo-V2.6](https://mimo.xiaomi.com/blog/mimo-v2-6-tool-call-repetition) - [MiMo-V2.6 blog post](https://mimo.xiaomi.com/mimo-v2-6) - [vLLM PR #57784 — mxfp4 MoE / bf16 router / DFlash for MiMo V2](https://github.com/vllm-project/vllm/pull/57784) - [vLLM PR #57933 — MiMo tool-call/reasoning parsers for the Rust frontend](https://github.com/vllm-project/vllm/pull/57933) - [vLLM PR #58262 — MiMo-V2.6 MXFP4 on gfx942 (`--moe-backend aiter_triton_mxfp4_bf16`)](https://github.com/vllm-project/vllm/pull/58262) - [SGLang MiMo cookbook](https://docs.sglang.io/cookbook/autoregressive/Xiaomi/MiMo-V2.5) - [vLLM ROCm GPU install docs](https://docs.vllm.ai/en/latest/getting_started/installation/gpu/?device=rocm) --- # XiaomiMiMo/MiMo-V2.6-Pro-RL on vLLM Source: https://recipes.vllm.ai/XiaomiMiMo/MiMo-V2.6-Pro-RL Provider: MiMo (Xiaomi) Released: 2026-09-21T15:39:33.000Z Updated: 2026-10-07 Architecture: moe Parameters: 1T Context length: 1,048,576 Min vLLM: 0.31.0 Tasks: text, multimodal Verified hardware: H200, GB300 NVL4, MI355X Summary: Xiaomi's flagship omnimodal MoE (1.02T total / 42B active) with hybrid attention, FP8-compute/mxfp4-stored weights, 1M context, and a DFlash speculative decoder — the default MOPD checkpoint fixes the RL release's tool-call repetition ## Variants - default: precision=fp8, vram_minimum_gb=680, model_id=XiaomiMiMo/MiMo-V2.6-Pro-MOPD FP8 (e4m3) compute with mxfp4-stored MoE weights; 566 GB checkpoint — 8x H200 with TP8, or 4x MI355X with TP4. MOPD2-distilled upgrade that fixes tool-call repetition; recommended. - rl: precision=fp8, vram_minimum_gb=680 Same shape as the MOPD variant (566 GB, FP8-compute/mxfp4-stored) — the original RL release, kept for reproducing earlier results. Suffers from tool-call repetition; prefer MOPD for agentic workloads. ## Recommended base flags ```bash vllm serve 'XiaomiMiMo/MiMo-V2.6-Pro-RL' \ --trust-remote-code \ --max-model-len \ auto \ --generation-config \ vllm ``` ## Guide ## Overview This recipe serves the MiMo-V2.6-Pro family from two checkpoints on the Variant row: - **MOPD** (default) — [XiaomiMiMo/MiMo-V2.6-Pro-MOPD](https://huggingface.co/XiaomiMiMo/MiMo-V2.6-Pro-MOPD), the MOPD2-distilled upgrade and the recommended starting point for new deployments. It fixes the RL release's most visible agentic failure mode — tool-call repetition, where the model keeps re-issuing the same or highly similar tool calls while making no progress — with benchmark quality unchanged, folded into a regular MOPD2 pass via a short specialized-teacher run ([diagnosis blog](https://mimo.xiaomi.com/blog/mimo-v2-6-tool-call-repetition)). - **RL (original)** — [XiaomiMiMo/MiMo-V2.6-Pro-RL](https://huggingface.co/XiaomiMiMo/MiMo-V2.6-Pro-RL), the 2026-09-22 RL release, kept for reproducing earlier results. It can get stuck re-issuing the same tool calls in agentic workloads; prefer **MOPD** unless you are reproducing a specific earlier run. MOPD2 (Technical Report §5.6) is multi-teacher on-policy distillation: domain-specialized mixRL and SFT teachers supervise the student on three on-policy streams (standard MOPD, teacher-prefix OPD, and SFT-prefix OPD). The architecture is shared by both checkpoints: a sparse MoE model with 1.02T total parameters and 42B active per token — 70 layers (1 dense + 69 MoE) with 384 routed experts (top-8), hybrid attention (sliding-window 128 and global attention at a 6:1 ratio), and a 5-layer DFlash-style MTP drafter that predicts 7 tokens per pass. The model is natively omnimodal (text, image, video, audio via a 681M-param MiMo ViT and audio encoders) and supports up to 1M tokens of context. Weights are stored as mxfp4 and computed as FP8 (block-wise e4m3, 128x128), so the checkpoint is 566 GB on disk. Loading this mixed storage format requires vLLM with the MiMo V2 mxfp4/bf16-router support ([vllm-project/vllm#57784](https://github.com/vllm-project/vllm/pull/57784)), first shipped in the v0.31.0 stable release. ## Prerequisites - Hardware: 8x H200 (TP8), 4x MI355X (TP4), or equivalent aggregate VRAM (>= 680 GB) ### Pull the vLLM docker image vLLM 0.31.0 is the first stable release that can load the mxfp4-stored weights ([vllm-project/vllm#57784](https://github.com/vllm-project/vllm/pull/57784)); earlier stable releases cannot: ```bash docker pull vllm/vllm-openai:v0.31.0 ``` A vLLM nightly wheel built after 2026-09-20 also works (`uv pip install -U vllm --extra-index-url https://wheels.vllm.ai/nightly/cu130`). The Hugging Face model cards point at `vllm/vllm-openai:mimov25-cu129`, the V2.5-era image; that tag predates the mxfp4-stored-weight support linked above, so use the v0.31.0 image recommended above. This release also registers the `mimo` tool-call and reasoning parsers with the Rust frontend ([vllm-project/vllm#57933](https://github.com/vllm-project/vllm/pull/57933)). ## Launch command Single-node TP8 (H200). Selecting the **RL (original)** variant in the builder swaps only the checkpoint id (`XiaomiMiMo/MiMo-V2.6-Pro-RL`); everything else is identical: ```bash vllm serve XiaomiMiMo/MiMo-V2.6-Pro-MOPD \ --tensor-parallel-size 8 \ --trust-remote-code \ --gpu-memory-utilization 0.95 \ --max-model-len auto \ --reasoning-parser mimo \ --tool-call-parser mimo \ --enable-auto-tool-choice \ --generation-config vllm ``` ### AMD (MI355X, TP4) ```bash VLLM_ROCM_USE_AITER=1 \ VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4 \ vllm serve XiaomiMiMo/MiMo-V2.6-Pro-MOPD \ --tensor-parallel-size 4 \ --trust-remote-code \ --gpu-memory-utilization 0.93 \ --kv-cache-dtype fp8 \ --max-model-len auto \ --reasoning-parser mimo \ --tool-call-parser mimo \ --enable-auto-tool-choice \ --generation-config vllm ``` MXFP4 MoE weights run natively on CDNA 4, so no load-time requantization is needed. `VLLM_ROCM_USE_AITER=1` is off by default and is required to reach the AITER MXFP4 MoE and FP8 block-scaled GEMM kernels. Attention resolves to `TRITON_ATTN_DIFFKV`, the only backend covering this model's asymmetric head dims (QK 192 / V 128) together with attention sinks and sliding-window attention. On MI355X, **Tensor Parallel** defaults to **TP4**: 4x288 GiB clears the 680 GiB requirement, so TP4 halves the GPUs per server and doubles the servers per node. For TP8, pass `--tensor-parallel-size 8`. The checkpoint pre-shards its fused QKV into `num_key_value_heads` (8) chunks, each carrying its own FP8 scales. TP8 consumes them directly; TP4 merges two chunks per rank, which vLLM handles by dequantizing, reordering and re-quantizing at load time. TP4 costs no accuracy. With DFlash speculative decoding (drafter ships inside the checkpoint). vLLM does not resolve a `/dflash` Hub subfolder as the draft model — `model` must be a concrete local path to the `dflash/` directory inside the downloaded snapshot. Resolve it per checkpoint (run `hf download ` first if it is not cached yet): ```bash # MOPD variant DFLASH_DIR=$(ls -d ~/.cache/huggingface/hub/models--XiaomiMiMo--MiMo-V2.6-Pro-MOPD/snapshots/*/dflash) # RL (original) variant # DFLASH_DIR=$(ls -d ~/.cache/huggingface/hub/models--XiaomiMiMo--MiMo-V2.6-Pro-RL/snapshots/*/dflash) echo "$DFLASH_DIR" ``` ```bash vllm serve XiaomiMiMo/MiMo-V2.6-Pro-MOPD \ --tensor-parallel-size 8 \ --trust-remote-code \ --gpu-memory-utilization 0.95 \ --max-model-len auto \ --speculative-config "{\"method\":\"dflash\",\"model\":\"$DFLASH_DIR\",\"num_speculative_tokens\":7}" \ --compilation-config '{"cudagraph_mode":"FULL_DECODE_ONLY"}' \ --reasoning-parser mimo \ --tool-call-parser mimo \ --enable-auto-tool-choice \ --generation-config vllm ``` The drafter inherits the target's tensor-parallel size by default. To pin it explicitly, add `"draft_tensor_parallel_size": 8` to the speculative config (useful if you want the draft on fewer GPUs than the target, or to silence mis-detection on asymmetric topologies). ## Client Usage ```bash curl -X POST http://localhost:8000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "XiaomiMiMo/MiMo-V2.6-Pro-MOPD", "messages": [{"role": "user", "content": "Hello MiMo!"}], "temperature": 1.0, "top_p": 0.95, "chat_template_kwargs": {"enable_thinking": true} }' ``` Recommended sampling: `temperature=1.0`, `top_p=0.95`. Set `"enable_thinking": false` (or omit the kwargs) to disable thinking mode. ## References - [MiMo-V2.6-Pro-MOPD on Hugging Face](https://huggingface.co/XiaomiMiMo/MiMo-V2.6-Pro-MOPD) - [MiMo-V2.6-Pro-RL on Hugging Face](https://huggingface.co/XiaomiMiMo/MiMo-V2.6-Pro-RL) - [MiMo-V2.6 technical report](https://huggingface.co/XiaomiMiMo/MiMo-V2.6-Pro-RL/blob/main/MiMo_V2_6_technical_report.pdf) (§5.6 covers MOPD2) - [Diagnosing and Mitigating Tool-Call Repetition in MiMo-V2.6](https://mimo.xiaomi.com/blog/mimo-v2-6-tool-call-repetition) - [MiMo-V2.6 blog post](https://mimo.xiaomi.com/mimo-v2-6) - [vLLM PR #57784 — mxfp4 MoE / bf16 router / DFlash for MiMo V2](https://github.com/vllm-project/vllm/pull/57784) - [vLLM PR #57933 — MiMo tool-call/reasoning parsers for the Rust frontend](https://github.com/vllm-project/vllm/pull/57933) - [SGLang MiMo cookbook](https://docs.sglang.io/cookbook/autoregressive/Xiaomi/MiMo-V2.5) - [vLLM ROCm GPU install docs](https://docs.vllm.ai/en/latest/getting_started/installation/gpu/?device=rocm) --- # IFM/Jais-3-0.9B on vLLM Source: https://recipes.vllm.ai/IFM/Jais-3-0.9B Provider: IFM Released: 2026-09-29T10:38:24.000Z Updated: 2026-10-05 Architecture: dense Parameters: 1.08B Context length: 131,072 Tasks: text Verified hardware: H200 Summary: Compact dense model for mathematics, code, instruction following, and STEM tasks ## Variants - default: precision=bf16, vram_minimum_gb=3 Released BF16 checkpoint; the command starts at an 8K runtime context ## Recommended base flags ```bash vllm serve 'IFM/Jais-3-0.9B' \ --trust-remote-code \ --chat-template-content-format \ string \ --dtype \ bfloat16 \ --max-model-len \ 131072 \ --gpu-memory-utilization \ 0.85 ``` ## Guide ## Overview Jais-3-0.9B is a compact dense model in the IFM Jais-3 family, with 1.08B stored parameters including embeddings and a 131,072-token context window. Jais-3 is an open-weight IFM model family built for transparent foundation-model research, staged checkpoint analysis, and practical deployment. The series spans compact dense models such as Jais-3-0.9B for local experimentation, dense mid-size and large models for high-quality research workloads, and mixture-of-expert (MoE) models for higher-capacity serving. ## Launch Commands Jais-3-0.9B's `config.json` already enables YaRN, so the full 131,072-token context needs no RoPE override. ```bash vllm serve IFM/Jais-3-0.9B \ --trust-remote-code \ --chat-template-content-format string \ --dtype bfloat16 \ --max-model-len 131072 \ --gpu-memory-utilization 0.85 \ --reasoning-parser k2_horizon \ --enable-auto-tool-choice \ --tool-call-parser k2_horizon ``` Lower `--max-model-len` (e.g. 8192) to reduce KV-cache memory. ## Client usage ```python from openai import OpenAI client = OpenAI( api_key="EMPTY", base_url="http://localhost:8000/v1", timeout=3600 ) resp = client.chat.completions.create( model="IFM/Jais-3-0.9B", messages=[{"role": "user", "content": "لدينا عددان مجموعهما 50 وحاصل ضربهما 600. ما هما العددان؟ اشرح خطوات الحل."}], temperature=1.0, top_p=0.95, max_tokens=2048, ) print(resp.choices[0].message.content) ``` ### Thinking modes The model supports selectable thinking through `chat_template_kwargs`, per request or server-wide via `--default-chat-template-kwargs`: - `{"reasoning_effort": "high"}` — full thinking (default). - `{"reasoning_effort": "low"}` — no thinking. ## References - [IFM/Jais-3-0.9B on Hugging Face](https://huggingface.co/IFM/Jais-3-0.9B) - [Jais 3 Collection](https://huggingface.co/collections/IFM/jais-3) --- # IFM/Jais-3-3.7B on vLLM Source: https://recipes.vllm.ai/IFM/Jais-3-3.7B Provider: IFM Released: 2026-09-29T10:39:15.000Z Updated: 2026-10-05 Architecture: dense Parameters: 5.06B Context length: 524,288 Tasks: text Verified hardware: H200 Summary: Dense model for long-context research, serving, and downstream adaptation ## Variants - default: precision=bf16, vram_minimum_gb=13 BF16 serving runtime with a 131K context baseline ## Recommended base flags ```bash vllm serve 'IFM/Jais-3-3.7B' \ --trust-remote-code \ --chat-template-content-format \ string \ --dtype \ bfloat16 ``` ## Guide ## Overview Jais-3-3.7B is a small dense model in the IFM Jais-3 family, with 5.06B stored parameters including embeddings and support for up to 524,288 tokens of context. Jais-3 is an open-weight IFM model family built for transparent foundation-model research, staged checkpoint analysis, and practical deployment. The series spans compact dense models for local experimentation, dense mid-size and large models for high-quality research workloads, and mixture-of-expert (MoE) models for higher-capacity serving. ## Launch Commands ```bash vllm serve IFM/Jais-3-3.7B \ --trust-remote-code \ --chat-template-content-format string \ --dtype bfloat16 \ --reasoning-parser k2_horizon \ --enable-auto-tool-choice \ --tool-call-parser k2_horizon ``` ## Client usage ```python from openai import OpenAI client = OpenAI( api_key="EMPTY", base_url="http://localhost:8000/v1", timeout=3600 ) resp = client.chat.completions.create( model="IFM/Jais-3-3.7B", messages=[{"role": "user", "content": "لدينا عددان مجموعهما 50 وحاصل ضربهما 600. ما هما العددان؟ اشرح خطوات الحل."}], temperature=1.0, top_p=0.95, max_tokens=2048, ) print(resp.choices[0].message.content) ``` ### Thinking modes The model supports selectable thinking through `chat_template_kwargs`, per request or server-wide via `--default-chat-template-kwargs`: - `{"reasoning_effort": "high"}` — full thinking (default). - `{"reasoning_effort": "low"}` — no thinking. ## References - [IFM/Jais-3-3.7B on Hugging Face](https://huggingface.co/IFM/Jais-3-3.7B) - [Jais 3 Collection](https://huggingface.co/collections/IFM/jais-3) --- # IFM/Jais-3-32B on vLLM Source: https://recipes.vllm.ai/IFM/Jais-3-32B Provider: IFM Released: 2026-09-29T10:45:14.000Z Updated: 2026-10-05 Architecture: dense Parameters: 34.78B Context length: 524,288 Tasks: text Verified hardware: H200 Summary: Dense long-context model for reasoning research and high-capacity serving ## Variants - default: precision=bf16, vram_minimum_gb=84 BF16 serving runtime with native vLLM execution ## Recommended base flags ```bash vllm serve 'IFM/Jais-3-32B' \ --trust-remote-code \ --chat-template-content-format \ string \ --dtype \ bfloat16 ``` ## Guide ## Overview Jais-3-32B is a large dense model in the IFM Jais-3 family, with 34.78B stored parameters including embeddings and support for up to 524,288 tokens of context. Jais-3 is an open-weight IFM model family built for transparent foundation-model research, staged checkpoint analysis, and practical deployment. The series spans compact dense models for local experimentation, dense mid-size and large models such as Jais-3-32B for high-quality research workloads, and mixture-of-expert (MoE) models for higher-capacity serving. ## Launch Commands ```bash vllm serve IFM/Jais-3-32B \ --tensor-parallel-size 2 \ --trust-remote-code \ --chat-template-content-format string \ --dtype bfloat16 \ --reasoning-parser k2_horizon \ --enable-auto-tool-choice \ --tool-call-parser k2_horizon ``` ## Client usage ```python from openai import OpenAI client = OpenAI( api_key="EMPTY", base_url="http://localhost:8000/v1", timeout=3600 ) resp = client.chat.completions.create( model="IFM/Jais-3-32B", messages=[{"role": "user", "content": "لدينا عددان مجموعهما 50 وحاصل ضربهما 600. ما هما العددان؟ اشرح خطوات الحل."}], temperature=1.0, top_p=0.95, max_tokens=2048, ) print(resp.choices[0].message.content) ``` ### Thinking modes The model supports selectable thinking through `chat_template_kwargs`, per request or server-wide via `--default-chat-template-kwargs`: - `{"reasoning_effort": "high"}` — full thinking (default). - `{"reasoning_effort": "low"}` — no thinking. ## References - [IFM/Jais-3-32B on Hugging Face](https://huggingface.co/IFM/Jais-3-32B) - [Jais 3 Collection](https://huggingface.co/collections/IFM/jais-3) --- # IFM/Jais-3-375B-A23B on vLLM Source: https://recipes.vllm.ai/IFM/Jais-3-375B-A23B Provider: IFM Released: 2026-09-29T11:26:05.000Z Updated: 2026-10-05 Architecture: moe Parameters: 379.17B Context length: 524,288 Tasks: text Verified hardware: H200, MI355X Summary: Frontier-scale sparse MoE model for long-context research and high-capacity serving ## Variants - default: precision=bf16, vram_minimum_gb=911 BF16 serving runtime with an eight-GPU tensor-parallel baseline ## Recommended base flags ```bash vllm serve 'IFM/Jais-3-375B-A23B' \ --trust-remote-code \ --chat-template-content-format \ string \ --dtype \ bfloat16 ``` ## Guide ## Overview Jais-3-375B-A23B is a frontier-scale mixture-of-expert (MoE) model in the IFM Jais-3 family, with 379.17B stored parameters including embeddings, 26.67B active parameters per token, and support for up to 524,288 tokens of context. Jais-3 is an open-weight IFM model family built for transparent foundation-model research, staged checkpoint analysis, and practical deployment. The series spans compact dense models for local experimentation, dense mid-size and large models for high-quality research workloads, and mixture-of-expert (MoE) models such as Jais-3-375B-A23B for frontier-scale research and higher-capacity serving. ## Launch Commands ```bash vllm serve IFM/Jais-3-375B-A23B \ --tensor-parallel-size 8 \ --trust-remote-code \ --chat-template-content-format string \ --dtype bfloat16 \ --enable-expert-parallel \ --reasoning-parser k2_horizon \ --enable-auto-tool-choice \ --tool-call-parser k2_horizon ``` ### AMD MI355X Verified on an 8-GPU MI355X (gfx950) node. AITER supplies the fused MoE and the unified attention backend: ```bash export VLLM_ROCM_USE_AITER=1 export NCCL_MIN_NCHANNELS=112 vllm serve IFM/Jais-3-375B-A23B \ --tensor-parallel-size 8 \ --trust-remote-code \ --chat-template-content-format string \ --dtype bfloat16 \ --max-model-len 131072 \ --gpu-memory-utilization 0.9 \ --attention-backend ROCM_AITER_UNIFIED_ATTN \ --generation-config auto \ --reasoning-parser k2_horizon \ --enable-auto-tool-choice \ --tool-call-parser k2_horizon ``` ## Client usage ```python from openai import OpenAI client = OpenAI( api_key="EMPTY", base_url="http://localhost:8000/v1", timeout=3600 ) resp = client.chat.completions.create( model="IFM/Jais-3-375B-A23B", messages=[{"role": "user", "content": "لدينا عددان مجموعهما 50 وحاصل ضربهما 600. ما هما العددان؟ اشرح خطوات الحل."}], temperature=1.0, top_p=0.95, max_tokens=2048, ) print(resp.choices[0].message.content) ``` ### Thinking modes The model supports selectable thinking through `chat_template_kwargs`, per request or server-wide via `--default-chat-template-kwargs`: - `{"reasoning_effort": "high"}` — full thinking (default). - `{"reasoning_effort": "low"}` — no thinking. ## References - [IFM/Jais-3-375B-A23B on Hugging Face](https://huggingface.co/IFM/Jais-3-375B-A23B) - [Jais 3 Collection](https://huggingface.co/collections/IFM/jais-3) --- # IFM/Jais-3-7B on vLLM Source: https://recipes.vllm.ai/IFM/Jais-3-7B Provider: IFM Released: 2026-09-29T10:41:20.000Z Updated: 2026-10-05 Architecture: dense Parameters: 9B Context length: 524,288 Tasks: text Verified hardware: H200 Summary: Dense model for long-context research, fine-tuning, and cost-conscious deployment ## Variants - default: precision=bf16, vram_minimum_gb=22 BF16 serving runtime with a 131K context baseline ## Recommended base flags ```bash vllm serve 'IFM/Jais-3-7B' \ --trust-remote-code \ --chat-template-content-format \ string \ --dtype \ bfloat16 ``` ## Guide ## Overview Jais-3-7B is a compact dense model in the IFM Jais-3 family, with 9B stored parameters including embeddings and support for up to 524,288 tokens of context. Jais-3 is an open-weight IFM model family built for transparent foundation-model research, staged checkpoint analysis, and practical deployment. The series spans compact dense models such as Jais-3-7B for local experimentation, dense mid-size and large models for high-quality research workloads, and mixture-of-expert (MoE) models for higher-capacity serving. ## Launch Commands ```bash vllm serve IFM/Jais-3-7B \ --trust-remote-code \ --chat-template-content-format string \ --dtype bfloat16 \ --reasoning-parser k2_horizon \ --enable-auto-tool-choice \ --tool-call-parser k2_horizon ``` ## Client usage ```python from openai import OpenAI client = OpenAI( api_key="EMPTY", base_url="http://localhost:8000/v1", timeout=3600 ) resp = client.chat.completions.create( model="IFM/Jais-3-7B", messages=[{"role": "user", "content": "لدينا عددان مجموعهما 50 وحاصل ضربهما 600. ما هما العددان؟ اشرح خطوات الحل."}], temperature=1.0, top_p=0.95, max_tokens=2048, ) print(resp.choices[0].message.content) ``` ### Thinking modes The model supports selectable thinking through `chat_template_kwargs`, per request or server-wide via `--default-chat-template-kwargs`: - `{"reasoning_effort": "high"}` — full thinking (default). - `{"reasoning_effort": "low"}` — no thinking. ## References - [IFM/Jais-3-7B on Hugging Face](https://huggingface.co/IFM/Jais-3-7B) - [Jais 3 Collection](https://huggingface.co/collections/IFM/jais-3) --- # IFM/Jais-3-MoVA-36B-A4B on vLLM Source: https://recipes.vllm.ai/IFM/Jais-3-MoVA-36B-A4B Provider: IFM Released: 2026-09-29T11:01:01.000Z Updated: 2026-10-05 Architecture: moe Parameters: 37.44B Context length: 524,288 Tasks: text Verified hardware: H200 Summary: Sparse MoVA and MoE long-context model for research and production-style serving experiments ## Variants - default: precision=bf16, vram_minimum_gb=90 BF16 serving runtime with native vLLM execution ## Recommended base flags ```bash vllm serve 'IFM/Jais-3-MoVA-36B-A4B' \ --trust-remote-code \ --chat-template-content-format \ string \ --dtype \ bfloat16 \ --enable-expert-parallel ``` ## Guide ## Overview Jais-3-MoVA-36B-A4B is a mixture-of-expert (MoE) model in the IFM Jais-3 family. It combines MoVA attention and MoE layers, with 37.44B stored parameters including embeddings, 5.95B active parameters per token, and support for up to 524,288 tokens of context. Jais-3 is an open-weight IFM model family built for transparent foundation-model research, staged checkpoint analysis, and practical deployment. The series spans compact dense models for local experimentation, dense mid-size and large models for high-quality research workloads, and mixture-of-expert (MoE) models such as Jais-3-MoVA-36B-A4B for higher-capacity serving. ## Launch Commands ```bash vllm serve IFM/Jais-3-MoVA-36B-A4B \ --tensor-parallel-size 2 \ --trust-remote-code \ --chat-template-content-format string \ --dtype bfloat16 \ --enable-expert-parallel \ --reasoning-parser k2_horizon \ --enable-auto-tool-choice \ --tool-call-parser k2_horizon ``` ## Client usage ```python from openai import OpenAI client = OpenAI( api_key="EMPTY", base_url="http://localhost:8000/v1", timeout=3600 ) resp = client.chat.completions.create( model="IFM/Jais-3-MoVA-36B-A4B", messages=[{"role": "user", "content": "لدينا عددان مجموعهما 50 وحاصل ضربهما 600. ما هما العددان؟ اشرح خطوات الحل."}], temperature=1.0, top_p=0.95, max_tokens=2048, ) print(resp.choices[0].message.content) ``` ### Thinking modes The model supports selectable thinking through `chat_template_kwargs`, per request or server-wide via `--default-chat-template-kwargs`: - `{"reasoning_effort": "high"}` — full thinking (default). - `{"reasoning_effort": "low"}` — no thinking. ## References - [IFM/Jais-3-MoVA-36B-A4B on Hugging Face](https://huggingface.co/IFM/Jais-3-MoVA-36B-A4B) - [Jais 3 Collection](https://huggingface.co/collections/IFM/jais-3) --- # zai-org/GLM-5.1 on vLLM Source: https://recipes.vllm.ai/zai-org/GLM-5.1 Provider: GLM (Z-AI) Released: 2026-04-03T09:28:47.000Z Updated: 2026-10-05 Architecture: moe Parameters: 744B Context length: 202,752 Min vLLM: 0.24.0 Tasks: text Verified hardware: H200, MI325X, MI350X, MI355X Summary: GLM-5.1 refreshed version of GLM-5 — frontier-scale MoE language model (~744B total parameters) with MTP speculative decoding and thinking mode ## Variants - default: precision=bf16, vram_minimum_gb=1786 Full precision BF16 — requires multi-node deployment - fp8: precision=fp8, vram_minimum_gb=893, model_id=zai-org/GLM-5.1-FP8 Native FP8 checkpoint — 8xH200/H20 (141GB × 8) single-node serving - nvfp4: precision=nvfp4, vram_minimum_gb=446, model_id=nvidia/GLM-5.1-NVFP4 NVIDIA NVFP4 quantized weights for Blackwell GPUs - mxfp4: precision=mxfp4, vram_minimum_gb=447, model_id=amd/GLM-5.1-MXFP4 AMD Quark MXFP4 checkpoint — MoE weights quantized for MI355X (gfx950) ## Recommended base flags ```bash vllm serve 'zai-org/GLM-5.1' \ --trust-remote-code \ --chat-template-content-format=string ``` ## Guide ## Overview GLM-5.1 is a refreshed version of GLM-5, the 744B parameter frontier MoE model from Z-AI. It keeps the asynchronous RL training recipe and delivers best-in-class open-source performance on reasoning, coding, and agentic benchmarks. Both BF16 and native FP8 checkpoints are published. Thinking mode is enabled by default; disable it by passing `"chat_template_kwargs": {"enable_thinking": false}` in request extras. ## Prerequisites - **vLLM version:** >= 0.24.0 for the `glm47` reasoning parser. Use the latest main branch if you need tool calling + MTP simultaneously. - **Hardware (FP8):** 8xH200 or 8xH20 (141GB × 8) - **DeepGEMM (FP8, NVIDIA):** install via `install_deepgemm.sh` from vLLM repo - **Hardware (FP8, AMD):** 8x MI325X (256GB × 8) or 8x MI350X (288GB × 8), `vllm/vllm-openai-rocm:nightly-92044241a02f05de51420d654daf669de20b4691`; uses AITER, not DeepGEMM - **Hardware (MXFP4):** 8x MI355X (gfx950), vLLM >= 0.24.0, `vllm/vllm-openai-rocm:latest` ### Using Docker ```bash docker run --gpus all \ -p 8000:8000 \ --ipc=host \ -v ~/.cache/huggingface:/root/.cache/huggingface \ vllm/vllm-openai:latest zai-org/GLM-5.1-FP8 \ --tensor-parallel-size 8 \ --tool-call-parser glm47 \ --reasoning-parser glm47 \ --enable-auto-tool-choice \ --chat-template-content-format=string \ --served-model-name glm-5.1-fp8 ``` Use `vllm/vllm-openai:latest-cu129` for CUDA 12.x. ### Install vLLM from Source ```bash uv venv source .venv/bin/activate uv pip install "vllm==0.24.0" --torch-backend=auto uv pip install "transformers>=5.4.0" ``` ## Launching the Server ### FP8 on 8xH200 with MTP ```bash vllm serve zai-org/GLM-5.1-FP8 \ --tensor-parallel-size 8 \ --speculative-config.method mtp \ --speculative-config.num_speculative_tokens 3 \ --tool-call-parser glm47 \ --reasoning-parser glm47 \ --enable-auto-tool-choice \ --chat-template-content-format=string \ --served-model-name glm-5.1-fp8 ``` ### FP8 on AMD MI325X and MI350X Select **MI325X** or **MI350X** and the **FP8** variant in the command builder above. The generated command adds `VLLM_ROCM_USE_AITER=1`, `VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS=1`, `--linear-backend aiter` and `--moe-backend aiter`, and raises `VLLM_ENGINE_READY_TIMEOUT_S` because loading the weights can take longer than the Rust frontend's 600 s default. The Docker command uses the nightly build `vllm/vllm-openai-rocm:nightly-92044241a02f05de51420d654daf669de20b4691`. On 8x MI325X that command, with MTP enabled, serves the full 202,752-token context with a 1,292,544-token KV cache (6.38x maximum concurrency) and an MTP mean acceptance length of 2.72 with 3 draft tokens on a ~380-token greedy generation. On 8x MI350X the same command serves the full context with a 1,587,312-token KV cache (7.83x maximum concurrency) and an MTP mean acceptance length of 3.13 with 3 draft tokens on a 512-token greedy generation. ## Client Usage ```python from openai import OpenAI client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1") # Thinking ON (default) resp_on = client.chat.completions.create( model="glm-5.1-fp8", messages=[{"role": "user", "content": "Summarize GLM-5.1 in one sentence."}], temperature=1, max_tokens=4096, ) print(resp_on.choices[0].message.reasoning) # Thinking OFF resp_off = client.chat.completions.create( model="glm-5.1-fp8", messages=[{"role": "user", "content": "Summarize GLM-5.1 in one sentence."}], temperature=1, max_tokens=4096, extra_body={"chat_template_kwargs": {"enable_thinking": False}}, ) ``` ### cURL (Thinking ON) ```bash curl http://localhost:8000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "glm-5.1-fp8", "messages": [ {"role": "user", "content": "Summarize GLM-5.1 in one sentence."} ], "temperature": 1, "max_tokens": 4096 }' ``` ## Benchmarking ```bash vllm bench serve \ --model zai-org/GLM-5.1-FP8 \ --dataset-name random \ --random-input 8000 \ --random-output 1024 \ --request-rate 10 \ --num-prompts 32 \ --ignore-eos ``` ## Troubleshooting - **Accuracy drift:** Prefer the 0.19.0 stable release for best accuracy. - **Tool calling + MTP:** If both are needed, use the latest vLLM main branch. - **FP8 installation (NVIDIA):** DeepGEMM required for FP8 performance. - **MXFP4 (MI355X):** MTP speculative decoding isn't yet supported on this checkpoint/kernel path. Mooncake and LMCache KV-store offload are still untested on MI355X — worth a check before leaning on them in production. ## References - [Model card](https://huggingface.co/zai-org/GLM-5.1) - [FP8 checkpoint](https://huggingface.co/zai-org/GLM-5.1-FP8) - [MXFP4 checkpoint](https://huggingface.co/amd/GLM-5.1-MXFP4) - [DeepGEMM install script](https://github.com/vllm-project/vllm/blob/v0.16.0rc0/tools/install_deepgemm.sh) --- # zai-org/GLM-5 on vLLM Source: https://recipes.vllm.ai/zai-org/GLM-5 Provider: GLM (Z-AI) Released: 2026-02-11T04:55:46.000Z Updated: 2026-10-05 Architecture: moe Parameters: 744B Context length: 202,752 Min vLLM: 0.24.0 Tasks: text Verified hardware: H200, MI325X, MI350X Summary: GLM-5 frontier-scale MoE language model (~744B total parameters, 28.5T training tokens) with asynchronous RL infrastructure for reasoning, coding, and agentic tasks ## Variants - default: precision=bf16, vram_minimum_gb=1786 Full precision BF16 — requires multi-node deployment - fp8: precision=fp8, vram_minimum_gb=893, model_id=zai-org/GLM-5-FP8 Native FP8 checkpoint — 8xH200/H20 (141GB x 8) single-node serving - nvfp4: precision=nvfp4, vram_minimum_gb=446, model_id=nvidia/GLM-5-NVFP4 NVIDIA NVFP4 quantized weights for Blackwell GPUs ## Recommended base flags ```bash vllm serve 'zai-org/GLM-5' \ --trust-remote-code \ --chat-template-content-format=string ``` ## Guide ## Overview GLM-5 is a significantly scaled-up language model with 744B parameters, trained on 28.5T tokens using novel asynchronous RL infrastructure. It delivers best-in-class open-source performance on reasoning, coding, and agentic tasks, rivaling frontier closed-source models. GLM-5 is available in both BF16 and native FP8 precisions. Thinking mode is enabled by default; disable it by passing `"chat_template_kwargs": {"enable_thinking": false}` in request extras. ## Prerequisites - **vLLM version:** >= 0.24.0 for the `glm47` reasoning parser - **Hardware (FP8):** 8xH200 or 8xH20 (141GB × 8) - **DeepGEMM (FP8, NVIDIA):** install via `install_deepgemm.sh` from the vLLM repo - **Hardware (FP8, AMD):** 8x MI325X (256GB × 8) or 8x MI350X (288GB × 8), `vllm/vllm-openai-rocm:nightly-0cbac6cd1305f710e12193596b27488397bcb205`; uses AITER, not DeepGEMM ### Using Docker ```bash docker run --gpus all \ -p 8000:8000 \ --ipc=host \ -v ~/.cache/huggingface:/root/.cache/huggingface \ vllm/vllm-openai:latest zai-org/GLM-5-FP8 \ --tensor-parallel-size 8 \ --tool-call-parser glm47 \ --reasoning-parser glm47 \ --enable-auto-tool-choice \ --chat-template-content-format=string ``` Use `vllm/vllm-openai:latest-cu129` for CUDA 12.x. ### Install vLLM from Source ```bash uv venv source .venv/bin/activate uv pip install "vllm==0.24.0" --torch-backend=auto uv pip install "transformers>=5.4.0" ``` On NVIDIA, install DeepGEMM using `install_deepgemm.sh` from the vLLM tools directory. ## Launching the Server ### FP8 on 8xH200 with MTP ```bash vllm serve zai-org/GLM-5-FP8 \ --tensor-parallel-size 8 \ --speculative-config.method mtp \ --speculative-config.num_speculative_tokens 3 \ --tool-call-parser glm47 \ --reasoning-parser glm47 \ --enable-auto-tool-choice \ --chat-template-content-format=string \ --served-model-name glm-5-fp8 ``` ### FP8 on AMD MI325X and MI350X Select **MI325X** or **MI350X** and the **FP8** variant in the command builder above. The generated command adds `VLLM_ROCM_USE_AITER=1`, `VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS=1`, `--linear-backend aiter` and `--moe-backend aiter`, and raises `VLLM_ENGINE_READY_TIMEOUT_S` because loading the weights can take longer than the Rust frontend's 600 s default. The Docker command uses the nightly build `vllm/vllm-openai-rocm:nightly-0cbac6cd1305f710e12193596b27488397bcb205`. On 8x MI325X that command, with MTP enabled, serves the full 202,752-token context with a 1,292,544-token KV cache (6.38x maximum concurrency) and an MTP mean acceptance length of 2.74 with 3 draft tokens on a ~380-token greedy generation. On 8x MI350X the same command serves the full context with a 1,587,312-token KV cache (7.83x maximum concurrency) and an MTP mean acceptance length of 2.95 with 3 draft tokens on a 512-token greedy generation. ## Client Usage ```python from openai import OpenAI client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1") # Thinking ON (default) resp_on = client.chat.completions.create( model="glm-5-fp8", messages=[{"role": "user", "content": "Summarize GLM-5 in one sentence."}], temperature=1, max_tokens=4096, ) print("thinking=on, think content:\n", resp_on.choices[0].message.reasoning) # Thinking OFF resp_off = client.chat.completions.create( model="glm-5-fp8", messages=[{"role": "user", "content": "Summarize GLM-5 in one sentence."}], temperature=1, max_tokens=4096, extra_body={"chat_template_kwargs": {"enable_thinking": False}}, ) print("thinking=off:\n", resp_off.choices[0].message.reasoning) ``` ## Benchmarking ```bash vllm bench serve \ --model zai-org/GLM-5-FP8 \ --dataset-name random \ --random-input 8000 \ --random-output 1024 \ --request-rate 10 \ --num-prompts 32 \ --ignore-eos ``` The MTP acceptance rate can be relatively low in pure benchmarks; measured throughput may underestimate real-world speed. ## Troubleshooting - **Accuracy drift:** Prefer the 0.19.0 stable release over nightly for best accuracy. - **Tool calling + MTP:** If you need both, use the latest vLLM main branch. - **FP8 installation (NVIDIA):** DeepGEMM is required for FP8 performance. ## References - [Model card](https://huggingface.co/zai-org/GLM-5) - [FP8 checkpoint](https://huggingface.co/zai-org/GLM-5-FP8) - [DeepGEMM install script](https://github.com/vllm-project/vllm/blob/v0.16.0rc0/tools/install_deepgemm.sh) --- # zai-org/GLM-5.2 on vLLM Source: https://recipes.vllm.ai/zai-org/GLM-5.2 Provider: GLM (Z-AI) Released: 2026-06-16T07:39:20.000Z Updated: 2026-10-03 Architecture: moe Parameters: 743B Context length: 1,048,576 Min vLLM: 0.24.0 Tasks: text Verified hardware: B300, B200, MI300X, MI325X, MI350X, MI355X Summary: GLM-5.2 — frontier-scale MoE language model (~743B total parameters, 39B active) with up to 5-token MTP speculative decoding and thinking mode ## Variants - default: precision=fp8, vram_minimum_gb=893, model_id=zai-org/GLM-5.2-FP8 Native FP8 checkpoint — 8xH200/H20 (141GB × 8) single-node serving - mxfp4: precision=mxfp4, vram_minimum_gb=446, model_id=amd/GLM-5.2-MXFP4 AMD Quark MXFP4 checkpoint — MoE weights quantized for MI355X (gfx950) - nvfp4: precision=nvfp4, vram_minimum_gb=558, model_id=nvidia/GLM-5.2-NVFP4 NVIDIA modelopt NVFP4 checkpoint — only MoE expert linears are quantized to NVFP4; shared experts, attention, embeddings, and the early dense layers stay FP8/BF16, with FP8 KV cache. Blackwell GPUs (B200/B300) only. - redhat_nvfp4: precision=nvfp4, vram_minimum_gb=558, model_id=RedHatAI/GLM-5.2-NVFP4-FP8 RedHatAI NVFP4+FP8 mixed-precision checkpoint (MoE expert linears NVFP4, attention/embeddings FP8) for Blackwell GPUs - bf16: precision=bf16, vram_minimum_gb=1786 Full precision BF16 — requires multi-node deployment - redhat_fp8: precision=fp8, vram_minimum_gb=892, model_id=RedHatAI/GLM-5.2-FP8 RedHatAI FP8 quantization - redhat_nvfp4_plain: precision=nvfp4, vram_minimum_gb=477, model_id=RedHatAI/GLM-5.2-NVFP4 RedHatAI NVFP4 quantization ## Recommended base flags ```bash vllm serve 'zai-org/GLM-5.2' \ --kv-cache-dtype \ fp8_e4m3 ``` ## Guide ## Overview GLM-5.2 is the newest model in the GLM-5 series — a ~743B-parameter MoE (39B active) from Z-AI. The headline change over GLM-5 / 5.1 is that **Multi-Token Prediction (MTP) is extended from 3 to 5 draft tokens**, lifting end-to-end throughput on reasoning, coding, and agentic workloads. It ships as BF16, native-FP8, and AMD Quark MXFP4 checkpoints, and keeps the GLM thinking-mode behavior. This recipe targets the **FP8** checkpoint, the practical default: it fits on a single 8xH200 / 8xH20 node and — with FP8 KV cache — reaches the full 1M-token context on 8xB200. On AMD MI355X hardware, use the **MXFP4** variant when HBM footprint is the priority. ## Prerequisites - **vLLM >= 0.24.0** for the `glm47` reasoning parser. If you need tool calling and MTP at the same time, use the latest `main` branch. - **GPU:** 8xH200 or 8xH20 (141 GB each) for single-node FP8; 8xB200 (180 GB each) for the full 1M context. - **MXFP4:** use the AMD Quark checkpoint on MI355X with ROCm and AITER enabled. TP=8 is recommended for the full MXFP4 model on an 8x MI355X node; TP=4 is also supported when memory headroom allows. ## Installation ### Docker ```bash docker run --gpus all \ -p 8000:8000 \ --ipc=host \ -v ~/.cache/huggingface:/root/.cache/huggingface \ vllm/vllm-openai:v0.28.0 zai-org/GLM-5.2-FP8 \ --tensor-parallel-size 8 \ --tool-call-parser glm47 \ --reasoning-parser glm47 \ --enable-auto-tool-choice \ --served-model-name glm-5.2-fp8 \ --max-model-len 131072 ``` On CUDA 12.x, swap the image for `vllm/vllm-openai:v0.28.0-cu129`. ### From source ```bash uv venv source .venv/bin/activate uv pip install "vllm==0.28.0" --torch-backend=auto uv pip install "transformers>=5.9.0" ``` ## Launching the server ### FP8 on 8xH200 (standard) ```bash vllm serve zai-org/GLM-5.2-FP8 \ --kv-cache-dtype fp8 \ --tensor-parallel-size 8 \ --speculative-config.method mtp \ --speculative-config.num_speculative_tokens 5 \ --tool-call-parser glm47 \ --reasoning-parser glm47 \ --enable-auto-tool-choice \ --served-model-name glm-5.2-fp8 ``` ### FP8 on 8xH20 (dspark with best practices) ```bash vllm serve zai-org/GLM-5.2-FP8 \ --tensor-parallel-size 8 \ --tool-call-parser glm47 \ --reasoning-parser glm47 \ --enable-auto-tool-choice \ --max-model-len 200000 \ --max-num-seqs 16 \ --max-num-batched-tokens 32768 \ --speculative-config '{"model": "RedHatAI/GLM-5.2-speculator.dspark","num_speculative_tokens": 7,"method": "dspark"}' \ --served-model-name glm-5.2-fp8 ``` tps up to 70+ for single request ### FP8 on AMD MI300X/MI325X/MI350X/MI355X (full 1M context) GLM-5.2 has a native 1M-token window. Whether the full window fits on ROCm is a KV-cache VRAM question, so the levers are `--max-model-len` and `--max-num-seqs`. Start with the values below, then scale them with your node's HBM and workload: raise the context window when startup reports KV-cache headroom, or lower the sequence cap if long prompts OOM under concurrency. ```bash VLLM_ROCM_USE_AITER=1 \ VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS=1 \ vllm serve zai-org/GLM-5.2-FP8 \ --kv-cache-dtype fp8_e4m3 \ --tensor-parallel-size 8 \ --speculative-config.method mtp \ --speculative-config.num_speculative_tokens 5 \ --tool-call-parser glm47 \ --enable-auto-tool-choice \ --reasoning-parser glm47 \ --gpu-memory-utilization 0.80 \ --max-model-len 524288 \ --max-num-seqs 32 \ --linear-backend aiter \ --moe-backend aiter ``` - **`--max-model-len`** — caps the served context window; raise it toward the native 1M window when your HBM budget and workload leave KV-cache headroom. - **`--max-num-seqs 32`** — the main knob for fitting long context under concurrency; start at 32 and tune it to your HBM (up on headroom, down on OOM). - **`--gpu-memory-utilization 0.80`** — leaves ROCm runtime headroom for MTP graph capture and inference; raise it only after a representative concurrent smoke test. - **`--speculative-config.num_speculative_tokens 5`** — enables GLM-5.2's 5-token MTP path; reduce it if your workload shows low acceptance or higher latency. - **`VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS=1`** — enables the shared-expert fused MoE path. `VLLM_ROCM_USE_AITER_LINEAR` and `VLLM_ROCM_USE_AITER_MOE` default to enabled when `VLLM_ROCM_USE_AITER=1`. On 8x MI325X this command serves the full `--max-model-len 524288` at `--gpu-memory-utilization 0.80`, reporting a 1,564,016-token KV cache and 2.98x maximum concurrency. MTP is healthy there: a mean acceptance length of 4.00 out of 5 drafted tokens on a short greedy generation. On 8x MI350X it reports a 2,205,424-token KV cache and 4.21x maximum concurrency at the same settings, and MTP's mean acceptance length is 3.62 out of 5 drafted tokens on a 512-token greedy generation. ### MXFP4 on AMD MI355X The AMD Quark MXFP4 checkpoint quantizes the MoE weights while leaving non-MoE modules in higher precision. The recipe UI only enables this variant for supported AMD MI355X hardware. Use ROCm AITER kernels: ```bash export HIP_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 export VLLM_ROCM_USE_AITER=1 export VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS=1 export VLLM_USE_V2_MODEL_RUNNER=1 export VLLM_ROCM_USE_AITER_FP8BMM=0 export VLLM_ROCM_USE_AITER_FP4BMM=0 vllm serve amd/GLM-5.2-MXFP4 \ --trust-remote-code \ --kv-cache-dtype fp8_e4m3 \ --tensor-parallel-size 8 \ --gpu-memory-utilization 0.90 \ --max-model-len 32768 \ --tool-call-parser glm47 \ --enable-auto-tool-choice \ --reasoning-parser glm47 \ --no-enable-prefix-caching \ --served-model-name glm-5.2-mxfp4 ``` For TP=4 validation runs, use `HIP_VISIBLE_DEVICES=0,1,2,3` and `--tensor-parallel-size 4`. ### FP8 on 8xB200 (full 1M context) GLM-5.2 has a native 1M-token window. Whether the full window fits is a KV-cache VRAM question, so the lever is `--max-num-seqs` — it bounds how many sequences share the KV budget at once, leaving room for each to hold a long context. **Start at 32** and scale with your node's VRAM: raise it on larger trays (e.g. 8xB300) or short-prompt traffic, lower it if you OOM at full context. FP8 KV cache (`--kv-cache-dtype fp8_e4m3`, already in the base flags) roughly halves that budget, which is what makes 1M reachable at all. ```bash VLLM_DEEP_GEMM_WARMUP=skip vllm serve zai-org/GLM-5.2-FP8 \ --kv-cache-dtype fp8_e4m3 \ --tensor-parallel-size 8 \ --speculative-config.method mtp \ --speculative-config.num_speculative_tokens 5 \ --max-num-seqs 32 \ --tool-call-parser glm47 \ --reasoning-parser glm47 \ --enable-auto-tool-choice \ --served-model-name glm-5.2-fp8 ``` - **`--max-num-seqs 32`** — the single knob for fitting 1M context; start at 32 and tune it to your VRAM (up on headroom, down on OOM). - **`VLLM_DEEP_GEMM_WARMUP=skip`** — skips DeepGEMM JIT warmup for a faster startup; the first few requests compile kernels on demand instead. - **BF16** needs multi-node plus an extra loader flag — see [Troubleshooting](#troubleshooting). ### NVFP4 on Blackwell (B200/B300) The `nvidia/GLM-5.2-NVFP4` variant is NVIDIA's modelopt re-quantization: only the MoE expert linears drop to NVFP4 while shared experts, attention, embeddings, and the early dense layers stay FP8/BF16, with FP8 KV cache. The ~465 GB checkpoint fits comfortably on Blackwell — vLLM auto-detects the quantization from the checkpoint, so no `--quantization` flag is needed. Select the **NVFP4** variant above (Blackwell-only) or run NVIDIA's command: ```bash vllm serve nvidia/GLM-5.2-NVFP4 \ --tensor-parallel-size 8 \ --enable-expert-parallel \ --reasoning-parser glm47 \ --tool-call-parser glm47 \ --enable-auto-tool-choice \ --kv-cache-dtype fp8_e4m3 \ --served-model-name glm-5.2-nvfp4 ``` ## Reasoning modes Thinking is **on by default**. GLM-5.2 reuses the DeepSeek-V4 `reasoning_effort` mechanism, with two effort levels driven by the `reasoning_effort` field: | Mode | How to request | Behavior | |------|----------------|----------| | **Think Max** (default) | omit `reasoning_effort`, or set `"max"` | Deepest reasoning — hard math, multi-step planning, agentic tasks. Highest token cost. | | **Think High** | `"reasoning_effort": "high"` | Balanced depth and latency. | | **Non-think** | `chat_template_kwargs.enable_thinking: false` | Fast, no chain-of-thought. | The chat template resolves effort to **`max` unless `reasoning_effort` is explicitly `"high"`**, so Max is the default and High is opt-in. Pass it through `chat_template_kwargs` (the DeepSeek-V4 path) or the top-level OpenAI `reasoning_effort` field; it has no effect when thinking is disabled. ## Client usage ```python from openai import OpenAI client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1") msgs = [{"role": "user", "content": "Summarize GLM-5.2 in one sentence."}] # Think Max (default) — just omit reasoning_effort client.chat.completions.create(model="glm-5.2-fp8", messages=msgs, max_tokens=4096) # Think High — explicitly request reasoning_effort: "high" client.chat.completions.create( model="glm-5.2-fp8", messages=msgs, max_tokens=4096, extra_body={"chat_template_kwargs": {"reasoning_effort": "high"}}, ) # Non-think client.chat.completions.create( model="glm-5.2-fp8", messages=msgs, max_tokens=4096, extra_body={"chat_template_kwargs": {"enable_thinking": False}}, ) ``` ### cURL — Think High ```bash curl http://localhost:8000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "glm-5.2-fp8", "messages": [{"role": "user", "content": "Summarize GLM-5.2 in one sentence."}], "temperature": 1, "max_tokens": 4096, "chat_template_kwargs": {"reasoning_effort": "high"} }' ``` ### cURL — non-think ```bash curl http://localhost:8000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "glm-5.2-fp8", "messages": [{"role": "user", "content": "Summarize GLM-5.2 in one sentence."}], "temperature": 1, "max_tokens": 4096, "chat_template_kwargs": {"enable_thinking": false} }' ``` ## Benchmarking Add `--no-enable-prefix-caching` to the server command for a clean measurement. ```bash vllm bench serve \ --model zai-org/GLM-5.2-FP8 \ --dataset-name random \ --random-input 8000 \ --random-output 1024 \ --request-rate 10 \ --num-prompts 32 \ --ignore-eos ``` > **Note:** pure throughput benchmarks tend to under-report real speed, because MTP's > acceptance rate is usually low in synthetic runs. ## Troubleshooting - **FP8 performance:** DeepGEMM is required — install via `install_deepgemm.sh`. - **MTP performance:** We fixed some MTP acceptance rate issue in [This PR](https://github.com/vllm-project/vllm/pull/45895). If you encounter MTP acceptance rate issue, please update your branch or refer to [GLM-5.2 Docker Image](https://hub.docker.com/r/vllm/vllm-openai/tags?name=glm52). ## References - [Model card](https://huggingface.co/zai-org/GLM-5.2) - [FP8 checkpoint](https://huggingface.co/zai-org/GLM-5.2-FP8) - [MXFP4 checkpoint](https://huggingface.co/amd/GLM-5.2-MXFP4) - [NVFP4 checkpoint (NVIDIA)](https://huggingface.co/nvidia/GLM-5.2-NVFP4) --- # deepseek-ai/DeepSeek-R1 on vLLM Source: https://recipes.vllm.ai/deepseek-ai/DeepSeek-R1 Provider: DeepSeek Released: 2025-01-20T03:46:07.000Z Updated: 2026-10-02 Architecture: moe Parameters: 671B Context length: 163,840 Min vLLM: 0.12.0 Tasks: text Verified hardware: H200, B200, GB200 NVL4, MI300X, MI325X, MI350X, MI355X Summary: DeepSeek-R1 is a 671B-parameter MoE reasoning model built on the DeepSeek-V3 architecture, trained with large-scale reinforcement learning for strong chain-of-thought capabilities. ## Variants - default: precision=fp8, vram_minimum_gb=805 Native FP8 weights on 8xH200 (recommended) - r1_0528: precision=fp8, vram_minimum_gb=805, model_id=deepseek-ai/DeepSeek-R1-0528 May 2025 DeepSeek-R1 refresh (DeepSeek-R1-0528) - nvfp4: precision=nvfp4, vram_minimum_gb=403, model_id=nvidia/DeepSeek-R1-0528-NVFP4-v2 NVIDIA NVFP4 quantized weights for Blackwell GPUs - mxfp4: precision=mxfp4, vram_minimum_gb=403, model_id=amd/DeepSeek-R1-0528-MXFP4-MTP-MoEFP4 AMD MXFP4 quantized weights for MI350X / MI355X (gfx950) - redhat_nvfp4: precision=nvfp4, vram_minimum_gb=444, model_id=RedHatAI/DeepSeek-R1-NVFP4-FP8-BLOCK RedHatAI NVFP4+FP8 block mixed-precision checkpoint ## Recommended base flags ```bash vllm serve 'deepseek-ai/DeepSeek-R1' \ --trust-remote-code ``` ## Guide ## Overview DeepSeek-R1 is a 671B-parameter Mixture-of-Experts reasoning model (37B activated per token) that shares its architecture with DeepSeek-V3, so the same launch recipes apply to both. DeepSeek publishes a refreshed checkpoint as `DeepSeek-R1-0528`, NVIDIA publishes an FP4 quantized variant (`nvidia/DeepSeek-R1-FP4`) that runs on Blackwell GPUs with fewer devices, and AMD publishes an MXFP4 quantized variant (`amd/DeepSeek-R1-0528-MXFP4-MTP-MoEFP4`) that runs on Instinct GPUs. ## Prerequisites - **Hardware (FP8)**: 8x H200 (CUDA) or 8x MI300X / MI325X / MI355X (ROCm) - **Hardware (NVFP4)**: 4x B200 GPUs - **Hardware (MXFP4)**: 8x MI350X / MI355X (ROCm) - **vLLM**: Current stable release ### CUDA ```bash uv venv source .venv/bin/activate uv pip install -U vllm --torch-backend auto ``` ### ROCm (MI300X, MI325X, MI355X) Requires Python 3.12, ROCm 7.2.1, and glibc >= 2.35. ```bash uv venv --python 3.12 source .venv/bin/activate uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/ ``` ## Client Usage ### 8xH200 / 8xMI300X (FP8) Tensor Parallel + Expert Parallel (TP8+EP) — CUDA: ```bash vllm serve deepseek-ai/DeepSeek-R1-0528 \ --trust-remote-code \ --tensor-parallel-size 8 \ --enable-expert-parallel ``` Tensor Parallel + Expert Parallel (TP8+EP) — ROCm: ```bash export VLLM_ROCM_USE_AITER=1 vllm serve deepseek-ai/DeepSeek-R1 \ --trust-remote-code \ --tensor-parallel-size 8 \ --enable-expert-parallel ``` Data Parallel + Expert Parallel (DP8+EP) — CUDA: ```bash vllm serve deepseek-ai/DeepSeek-R1-0528 \ --trust-remote-code \ --data-parallel-size 8 \ --enable-expert-parallel ``` ### 8xMI355X (FP8) Tensor Parallel (TP8) — ROCm. Validated on MI355X only; INT4 quick reduce and fused shared experts are not enabled on other AMD GPUs or other strategies: ```bash export VLLM_ROCM_USE_AITER=1 export VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4 export VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS=1 vllm serve deepseek-ai/DeepSeek-R1 \ --trust-remote-code \ --tensor-parallel-size 8 ``` ### 8xMI350X / 8xMI355X (DPA+TP) Each GPU runs its own attention replica with a private KV cache, while the MoE experts stay tensor-sharded across all eight GPUs. Aggregate KV-cache capacity is roughly 8x that of a single TP8 replica, useful for workloads benefiting from larger KV capacities. Engine (all eight ranks in one process, listening on 8100): ```bash export VLLM_ROCM_USE_AITER=1 export VLLM_ENGINE_READY_TIMEOUT_S=1800 vllm serve deepseek-ai/DeepSeek-R1 \ --trust-remote-code \ --port 8100 \ --data-parallel-size 8 \ --tensor-parallel-size 1 ``` Router (client-facing on 8000, spreads requests across the eight DP ranks): ```bash uv pip install vllm-router vllm-router \ --host 0.0.0.0 \ --port 8000 \ --policy consistent_hash \ --intra-node-data-parallel-size 8 \ --worker-urls http://localhost:8100 \ --worker-startup-timeout-secs 1800 ``` Notes: - `--intra-node-data-parallel-size` must match the engine's `--data-parallel-size`. A mismatch silently under-uses ranks. - `consistent_hash` is the policy that makes this worthwhile for chat: it pins a conversation to the rank that already holds its prefix, so the larger aggregate cache actually gets hit. `round_robin` scatters turns of the same conversation across ranks and throws the prefix away. - Start the engine first, then the router; `--worker-startup-timeout-secs 1800` covers the engine's weight load either way. - Send client traffic to the router on port 8000, never to 8100 directly. ### 4xB200 (FP4) On vLLM v0.28.0 and later, the FlashInfer MoE kernels are selected automatically for both FP4 and FP8 on Blackwell — no environment variables are needed. To pin the backend explicitly, pass `--moe-backend flashinfer_trtllm`. Tensor Parallel + Expert Parallel (TP4+EP): ```bash CUDA_VISIBLE_DEVICES=0,1,2,3 vllm serve nvidia/DeepSeek-R1-FP4 \ --trust-remote-code \ --tensor-parallel-size 4 \ --enable-expert-parallel ``` Data Parallel + Expert Parallel (DP4+EP): ```bash CUDA_VISIBLE_DEVICES=0,1,2,3 vllm serve nvidia/DeepSeek-R1-FP4 \ --trust-remote-code \ --data-parallel-size 4 \ --enable-expert-parallel ``` ### 8xMI350X / 8xMI355X (MXFP4) Tensor Parallel + FP8 KV cache (TP8): ```bash export VLLM_ROCM_USE_AITER=1 export VLLM_ROCM_USE_SKINNY_GEMM=0 vllm serve amd/DeepSeek-R1-0528-MXFP4-MTP-MoEFP4 \ --trust-remote-code \ --tensor-parallel-size 8 \ --kv-cache-dtype fp8 ``` ## Benchmarking For benchmarking, prefix caching is disabled by default in vLLM — no extra server flag is needed. ```bash # FP8 benchmark vllm bench serve \ --model deepseek-ai/DeepSeek-R1-0528 \ --dataset-name random \ --random-input-len 8000 \ --random-output-len 1000 \ --request-rate 10000 \ --num-prompts 16 \ --ignore-eos ``` ```bash # NVFP4 benchmark vllm bench serve \ --model nvidia/DeepSeek-R1-FP4 \ --dataset-name random \ --random-input-len 8000 \ --random-output-len 1000 \ --request-rate 10000 \ --num-prompts 16 \ --ignore-eos ``` ```bash # MXFP4 benchmark (MI350X / MI355X) vllm bench serve \ --model amd/DeepSeek-R1-0528-MXFP4-MTP-MoEFP4 \ --dataset-name random \ --random-input-len 8000 \ --random-output-len 1000 \ --request-rate 10000 \ --num-prompts 16 \ --ignore-eos ``` Test different workloads by adjusting input/output lengths: - Prompt-heavy: 8000 input / 1000 output - Decode-heavy: 1000 input / 8000 output - Balanced: 1000 input / 1000 output ## Troubleshooting - **Disaggregated Serving with Wide EP (Experimental GB200)**: See [vLLM issue #33583](https://github.com/vllm-project/vllm/issues/33583), [the vLLM blog post](https://blog.vllm.ai/2026/02/03/dsr1-gb200-part1.html), and [this reference fork](https://github.com/minosfuture/vllm/tree/pd_gb200_0114/runs/DS-R1/fp4) for GB200 disaggregated serving recipes. ## References - [DeepSeek-R1 on Hugging Face](https://huggingface.co/deepseek-ai/DeepSeek-R1) - [DeepSeek-R1-0528 on Hugging Face](https://huggingface.co/deepseek-ai/DeepSeek-R1-0528) - [NVIDIA DeepSeek-R1-FP4](https://huggingface.co/nvidia/DeepSeek-R1-FP4) - [AMD DeepSeek-R1-0528-MXFP4-MTP-MoEFP4](https://huggingface.co/amd/DeepSeek-R1-0528-MXFP4-MTP-MoEFP4) - [vLLM Expert Parallelism docs](https://docs.vllm.ai/en/latest/serving/expert_parallel_deployment.html) - [vLLM Data Parallel deployment docs](https://docs.vllm.ai/en/latest/serving/data_parallel_deployment.html) --- # Qwen/Qwen3.5-35B-A3B on vLLM Source: https://recipes.vllm.ai/Qwen/Qwen3.5-35B-A3B Provider: Qwen Released: 2026-02-24T09:39:25.000Z Updated: 2026-09-30 Architecture: moe Parameters: 35B Context length: 262,144 Min vLLM: 0.17.0 Tasks: multimodal, text Verified hardware: Xeon 6, Arc Pro B60, Arc Pro B70 Summary: Compact Qwen3.5 multimodal MoE (35B total / 3B active) with gated delta networks, 256 experts, and 262K context ## Variants - default: precision=bf16, vram_minimum_gb=84 Full precision BF16 — fits on 1x H200, 2x H100, 2x Xeon 6 NUMA nodes, or 4x Intel Arc Pro B60 / B70 - fp8: precision=fp8, vram_minimum_gb=42, model_id=Qwen/Qwen3.5-35B-A3B-FP8 Qwen official FP8 checkpoint — single-GPU serving - gptq_int4: precision=int4, vram_minimum_gb=21, model_id=Qwen/Qwen3.5-35B-A3B-GPTQ-Int4 GPTQ Int4 checkpoint — fits on a single 24GB GPU - redhat_fp8: precision=fp8, vram_minimum_gb=42, model_id=RedHatAI/Qwen3.5-35B-A3B-FP8-dynamic RedHatAI FP8 (E4M3) with dynamic per-token activation quantization ## Recommended base flags ```bash vllm serve 'Qwen/Qwen3.5-35B-A3B' \ --trust-remote-code ``` ## Guide ## Overview [Qwen3.5-35B-A3B](https://huggingface.co/Qwen/Qwen3.5-35B-A3B) is the smallest MoE in the Qwen3.5 family, sharing the gated delta networks architecture with 35B total parameters and 3B activated per token (256 experts). With FP8 weights it fits on a single 80 GB GPU and supports the full 262K context. ## Prerequisites - **vLLM version:** >= 0.17.0 - **Hardware (BF16):** 1x H200, 2x H100, 2x Xeon 6 NUMA nodes, 4x Intel Arc Pro B60 / B70 - **Hardware (FP8):** single H100/H200 - **Hardware (Int4):** single 24 GB GPU ### Pip Install #### NVIDIA ```bash uv venv source .venv/bin/activate uv pip install -U vllm --torch-backend=auto ``` #### CPU For Intel and AMD x86 CPUs, follow the [CPU pre-built wheels](https://docs.vllm.ai/en/latest/getting_started/installation/cpu/#pre-built-wheels) installation instructions. ```bash uv venv source .venv/bin/activate export VLLM_VERSION=$(curl -s https://api.github.com/repos/vllm-project/vllm/releases/latest | jq -r .tag_name | sed 's/^v//') uv pip install https://github.com/vllm-project/vllm/releases/download/v${VLLM_VERSION}/vllm-${VLLM_VERSION}+cpu-cp38-abi3-manylinux_2_35_x86_64.whl --torch-backend cpu ``` ### Docker ```bash docker pull vllm/vllm-openai-xpu:latest # Intel XPU (B60 / B70) ``` ## Launching the Server ### Single-GPU FP8 ```bash vllm serve Qwen/Qwen3.5-35B-A3B-FP8 \ --max-model-len 262144 \ --reasoning-parser qwen3 ``` ### BF16 on 2xH200 (TP2) ```bash vllm serve Qwen/Qwen3.5-35B-A3B \ --tensor-parallel-size 2 \ --max-model-len 262144 \ --reasoning-parser qwen3 ``` ### Intel Xeon 6 Deployment via Docker Launch the x86 CPU vLLM Docker container for `Qwen/Qwen3.5-35B-A3B`: ```bash docker run \ --privileged --ipc=host -p 8000:8000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ -e VLLM_ENGINE_READY_TIMEOUT_S=1800 \ vllm/vllm-openai-cpu:latest-x86_64 Qwen/Qwen3.5-35B-A3B \ --trust-remote-code \ --tensor-parallel-size 1 \ --enable-auto-tool-choice \ --tool-call-parser qwen3_coder \ --reasoning-parser qwen3 \ --mm-encoder-tp-mode data ``` ### MTP speculative decoding ```bash vllm serve Qwen/Qwen3.5-35B-A3B-FP8 \ --speculative-config '{"method": "mtp", "num_speculative_tokens": 1}' \ --reasoning-parser qwen3 ``` ### Docker (Intel XPU B60 / B70) Validated on 4× Intel Arc Pro B60 / B70 (B60 24 GB, B70 32 GB per card) with the official vLLM XPU image `vllm/vllm-openai-xpu:latest`. ```bash docker run --device /dev/dri \ -v /dev/dri/by-path:/dev/dri/by-path --shm-size=16g \ --privileged --ipc=host -p 8000:8000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ vllm/vllm-openai-xpu:latest \ Qwen/Qwen3.5-35B-A3B \ --reasoning-parser qwen3 \ --tensor-parallel-size 4 \ --max-model-len 8192 \ --enforce-eager ``` ## Client Usage ```python from openai import OpenAI client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1") resp = client.chat.completions.create( model="Qwen/Qwen3.5-35B-A3B", messages=[{"role": "user", "content": "Explain gated delta networks in one paragraph."}], max_tokens=512, ) print(resp.choices[0].message.content) ``` ## Troubleshooting - **CPU startup timeout:** a cold weight download followed by CPU compilation can exceed the default 600-second engine readiness timeout. The CPU recipe sets `VLLM_ENGINE_READY_TIMEOUT_S=1800`; set any external health-check deadline higher (for example, 2100 seconds). Increasing only the external deadline does not prevent the frontend from timing out. - **CUDA graph / Mamba cache size error:** reduce `--max-cudagraph-capture-size` (default 512). See [vLLM PR #34571](https://github.com/vllm-project/vllm/pull/34571). - **Disable reasoning:** add `--default-chat-template-kwargs '{"enable_thinking": false}'`. - **Prefix Caching (Mamba):** currently experimental in "align" mode. ## References - [Model card](https://huggingface.co/Qwen/Qwen3.5-35B-A3B) - [FP8 checkpoint](https://huggingface.co/Qwen/Qwen3.5-35B-A3B-FP8) - [GPTQ-Int4 checkpoint](https://huggingface.co/Qwen/Qwen3.5-35B-A3B-GPTQ-Int4) - [Qwen3.5-397B-A17B recipe](../Qwen3.5-397B-A17B) --- # deepseek-ai/DeepSeek-V4-Flash on vLLM Source: https://recipes.vllm.ai/deepseek-ai/DeepSeek-V4-Flash Provider: DeepSeek Released: 2026-04-22T06:04:20.000Z Updated: 2026-09-29 Architecture: moe Parameters: 284B Context length: 1,048,576 Min vLLM: 0.20.0 Tasks: text Verified hardware: H200, B200, GB200 NVL4, B300, GB300 NVL4, DGX Station (GB300), DGX Spark (GB10), RTX Pro 6000 8x, MI300X, MI325X, MI355X, Ascend 950DT 8x, Ascend 950PR 8x, Ascend 910C Summary: DeepSeek V4 MoE model with hybrid CSA+HCA attention, manifold-constrained hyper-connections, and three-tier reasoning (Non-think / Think High / Think Max). ## Variants - default: precision=fp8, vram_minimum_gb=200, model_id=deepseek-ai/DeepSeek-V4-Flash-0731 Official DeepSeek-V4-Flash release (2026-07-31) superseding the preview — substantially stronger agentic performance, with the DSpark draft module attached. - fp8: precision=fp8, vram_minimum_gb=170 Native FP4+FP8 mixed checkpoint (MoE experts FP4, remaining params FP8) - nvfp4: precision=nvfp4, vram_minimum_gb=170, model_id=nvidia/DeepSeek-V4-Flash-NVFP4 NVIDIA modelopt NVFP4 checkpoint (MoE experts NVFP4; attention, shared experts, router head, and MTP stay FP8) for Blackwell GPUs. - redhat_nvfp4: precision=nvfp4, vram_minimum_gb=170, model_id=RedHatAI/DeepSeek-V4-Flash-NVFP4-FP8 RedHatAI NVFP4+FP8 mixed-precision checkpoint for Blackwell GPUs - dspark: precision=fp8, vram_minimum_gb=200, model_id=deepseek-ai/DeepSeek-V4-Flash-DSpark Preview FP4+FP8 weights with the DSpark draft module baked in — same base checkpoint as the default variant plus a fused speculative decoder. - ascend_w8a8: precision=int8, vram_minimum_gb=341, model_id=Eco-Tech/DeepSeek-V4-Flash-w8a8-mtp ModelSlim INT8 W8A8 checkpoint for one Atlas 800 A3 node (16× Ascend 910C), using DP2+TP8. Requires --quantization ascend. ## Recommended base flags ```bash vllm serve 'deepseek-ai/DeepSeek-V4-Flash' \ --trust-remote-code \ --kv-cache-dtype \ fp8 \ --block-size \ 256 ``` ## Guide ## Overview DeepSeek-V4-Flash is a 284B-total / 13B-active MoE model in the V4 preview family. It pairs a **hybrid attention stack** — Compressed Sparse Attention (CSA) + Heavily Compressed Attention (HCA) — with **Manifold-Constrained Hyper-Connections (mHC)** to reach 27% of V3.2's per-token inference FLOPs and 10% of V3.2's KV cache at 1M context. Pre-trained on 32T+ tokens; post-training is a two-stage pipeline (domain- specific expert cultivation + unified consolidation via on-policy distillation). Checkpoint is **FP4+FP8 mixed**: MoE expert weights are stored in FP4 while the remaining (attention / norm / router) params stay in FP8. An **NVFP4 variant** (`nvidia/DeepSeek-V4-Flash-NVFP4`) is also available — NVIDIA modelopt re-quantizes the MoE experts to standard NVFP4 while attention, shared experts, router head, and MTP stay FP8. Pick it from the **Variant** row; it runs on Blackwell GPUs with the FP4 indexer cache. Unlike the native FP8 checkpoint, the NVFP4 experts don't support the `deep_gemm_mega_moe` MoE kernel (FP8-only), so it runs on the default MoE backend. ## Checkpoints Four checkpoints are on the **Variant** row. They differ in *which weights you serve* and *which draft module ships with them* — the speculative method itself is picked on the **Spec Decoding** row. | Variant | Repo | Draft | Notes | | :-- | :-- | :-- | :-- | | **FP8 (0731)** (default) | `deepseek-ai/DeepSeek-V4-Flash-0731` | DSpark | Official release, new weights + DSpark | | **FP8 (Preview)** | `deepseek-ai/DeepSeek-V4-Flash` | MTP | Preview FP4+FP8 mixed weights | | **NVFP4** | `nvidia/DeepSeek-V4-Flash-NVFP4` | MTP | modelopt re-quant, Blackwell | | **DSpark** | `deepseek-ai/DeepSeek-V4-Flash-DSpark` | DSpark | Preview weights + fused DSpark module | **0731 is the official DeepSeek-V4-Flash release** and the default here, superseding the preview with substantially stronger agentic capability — 82.7 on Terminal Bench 2.1 and 54.4 on DeepSWE, versus 61.8 and 7.3 for the preview. It beats DeepSeek-V4-Pro (Preview) on every agentic benchmark DeepSeek published despite its far smaller activated parameter count. The preview weights remain available as the **FP8 (Preview)** variant for reproducing earlier results. **DSpark** is *not* new weights — it is the preview checkpoint with a speculative decoding module attached (see [DeepSpec](https://github.com/deepseek-ai/DeepSpec)). Both fused checkpoints add a `dspark_*` block to `config.json`; the preview and NVFP4 checkpoints have no such block, which is why the DSpark method is offered only on the two variants that carry the draft. Selecting either one auto-enables **Spec Decoding** with `method: dspark`, emitting: ``` --speculative-config '{"method":"dspark","num_speculative_tokens":7,"draft_sample_method":"probabilistic"}' ``` The fused checkpoints are ~167 GB on disk versus ~160 GB for the preview — the draft module is the difference. Both require **vLLM 0.25.0**: DSpark drafting itself landed in 0.25.0, but ROCm support for it (MI325X / MI355X) only shipped in 0.26.0. On NVIDIA-only deployments 0.25.0 is sufficient, except the 0731 checkpoint on RTX PRO 6000 (sm_120) with DSpark, which needs `vllm/vllm-openai:nightly` and a FlashInfer build containing flashinfer-ai/flashinfer#4380. ## Reasoning modes The chat template exposes three reasoning-effort modes: - **Non-think** — fast, intuitive responses. - **Think High** — explicit chain-of-thought for logical analysis and planning. - **Think Max** — maximum reasoning effort; requires `--max-model-len >= 393216` (384K tokens) to avoid truncation. Recommended sampling: `temperature = 1.0`, `top_p = 1.0`. On the **0731** variant the effort levels are named `low` / `high` / `max`, and DeepSeek recommends `top_p = 0.95` for agentic scenarios (`1.0` otherwise) with `temperature = 1.0`. Allow up to **384K output tokens** at the `high` and `max` levels. Note that neither fused checkpoint ships a Jinja chat template — the repos provide an `encoding/` folder with `encode_messages` / `parse_message_from_completion_text` helpers instead. Serving through vLLM with `--tokenizer-mode deepseek_v4` (the **Tool Calling** pill) applies the built-in DeepSeek-V4 encoding, so the OpenAI- compatible endpoint works without the helper scripts. ### OpenAI Client Example For DeepSeek-V4, keep reasoning controls in `chat_template_kwargs`, as it exposes a custom **Think Max** mode via `"reasoning_effort": "max"`. ```python from openai import OpenAI client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY") model = "deepseek-ai/DeepSeek-V4-Flash" messages = [{"role": "user", "content": "What is 17*19? Return only the final integer."}] # Non-think resp = client.chat.completions.create( model=model, messages=messages, ) # Think High resp = client.chat.completions.create( model=model, messages=messages, extra_body={ "chat_template_kwargs": { "thinking": True, "reasoning_effort": "high", }, }, ) # Think Max resp = client.chat.completions.create( model=model, messages=messages, extra_body={ "chat_template_kwargs": { "thinking": True, "reasoning_effort": "max", }, }, ) ``` ## Recommended deployment Non-disaggregated serving on multi-GPU supported hardware: single-node DP + EP with `--data-parallel-size 4`. Fills a GB200 NVL4 tray exactly; uses 4 of 8 GPUs per replica on H200/B200/B300 (leaving headroom for throughput-vs-latency tuning). On DGX Station, use the single-GPU launch below. For disaggregated prefill/decode on GB200, use the PD Cluster tab. ### H100 (8×80 GB) The FP8 (0731) and DSpark checkpoints are selectable on H100. A reported 8×H100 TP8 + EP deployment of 0731 served requests with vLLM 0.26.0, while the same deployment failed during startup on 0.27.0, both with and without DSpark. See [vLLM #52065](https://github.com/vllm-project/vllm/issues/52065). For the separate DSpark checkpoint, a 4×H100 TP4 report on vLLM 0.25.0 reached inference after removing `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True` and setting `--gpu-memory-utilization` below 0.9. A 2×H100 NVL report also ran the checkpoint with DSpark k=5 on a custom vLLM build. These reports establish Hopper compatibility, but do not verify this recipe's generated H100 topology or current image. Check startup and inference before production. See [vLLM #49101](https://github.com/vllm-project/vllm/issues/49101) and [the H100 NVL report](https://github.com/vllm-project/vllm/pull/46995#issuecomment-4835994467). ### DGX Spark (GB10) cluster DGX Spark requires the community Spark vLLM build for this model rather than the stock vLLM release/nightly image. The **DGX Spark (GB10)** hardware pill pins the Docker tab to `eugr/spark-vllm-b12x:latest`, whose build carries the B12X backends and Spark-specific kernels for the GB10 / SM12x family. Use **2 nodes**. A single GB10's 128 GB unified memory is below the FP8/NVFP4 checkpoint footprint, so the recipe grants DGX Spark a Spark-only **Multi-Node TP** layout (`--tensor-parallel-size 2`, one GB10 per node). The generated command also sets the B12X runtime environment and the Spark network environment; export `IFACE_NAME`, `IB_IF` and `HEAD_IP` on each node before launching. To identify the values, first list the ConnectX Ethernet interfaces, RoCE devices and the IP address of the UP ConnectX interface on each Spark: ```bash $ ibdev2netdev roceP2p1s0f0 port 1 ==> enP2p1s0f0np0 (Up) roceP2p1s0f1 port 1 ==> enP2p1s0f1np1 (Down) rocep1s0f0 port 1 ==> enp1s0f0np0 (Up) rocep1s0f1 port 1 ==> enp1s0f1np1 (Down) $ ip addr show enp1s0f0np0 3: enp1s0f0np0: mtu 1500 qdisc mq state UP group default qlen 1000 link/ether 4c:62:56:c1:83:68 brd ff:ff:ff:ff:ff:ff inet 100.100.100.10/24 brd 100.100.100.255 scope global noprefixroute enp1s0f0np0 valid_lft forever preferred_lft forever ``` From above example values, we can get the following common values: ```bash # Common values for all nodes HEAD_IP="100.100.100.10" # IP of the head node's UP ConnectX interface IFACE_NAME="enp1s0f0np0" # The head node's UP ConnectX Ethernet interface IB_IF="rocep1s0f0,roceP2p1s0f0" # Comma-separated RoCE device name list corresponding to the UP ConnectX interface ``` ### DGX Station Single-GPU ```bash vllm serve deepseek-ai/DeepSeek-V4-Flash \ --tensor-parallel-size 1 --pipeline-parallel-size 1 \ --kv-cache-dtype fp8 --trust-remote-code --block-size 256 \ --gpu-memory-utilization 0.92 \ --compilation-config '{"cudagraph_mode":"FULL_AND_PIECEWISE","custom_ops":["all"]}' \ --attention_config.indexer_kv_dtype mxfp4 \ --tokenizer-mode deepseek_v4 --tool-call-parser deepseek_v4 \ --enable-auto-tool-choice --reasoning-parser deepseek_v4 \ --max-cudagraph-capture-size 128 \ --speculative-config '{"method":"mtp","num_speculative_tokens":3}' ``` ### RTX PRO 6000 8× (8×96 GB, sm_120) Single-node TP8 + expert parallelism on the **0731** checkpoint. Serving without speculative decoding was verified on 8× RTX PRO 6000 (PCIe, no NVLink). DSpark on **0731** needs `vllm/vllm-openai:nightly` (or a build that includes vLLM #51538) plus a FlashInfer that instantiates the SM120 sparse-MLA decode kernel for `topk=192`. The Preview, NVFP4, and DSpark variant checkpoints are not verified on this profile. sm_120 cannot use the SM100 FP4 indexer cache or `deep_gemm_mega_moe`. The generated command already sets `--attention_config.indexer_kv_dtype fp8` and `--moe-backend auto`. Do not use `method: mtp` on 0731 — that checkpoint has no MTP head; Spec Decoding defaults to DSpark. Host-side FlashInfer install before launching 0731 with DSpark (`--no-deps` is required so pip does not replace vLLM's torch): ```bash # Trace: flashinfer-ai/flashinfer#4380 # merge: 24d7dfb2639083c5a4d418881099421fc800b7bb (2026-08-08) # wheel: flashinfer-python==0.6.18.dev20260814 # index: https://flashinfer.ai/whl/nightly/ /usr/bin/python3.12 -m pip install -U --pre flashinfer-python \ --index-url https://flashinfer.ai/whl/nightly/ --no-deps /usr/bin/python3.12 -m pip install flashinfer-python /usr/bin/python3.12 -m pip install -U --pre flashinfer-cubin \ --index-url https://flashinfer.ai/whl/nightly/ /usr/bin/python3.12 -m pip install -U --pre flashinfer-jit-cache \ --index-url https://flashinfer.ai/whl/nightly/cu130 vllm serve deepseek-ai/DeepSeek-V4-Flash-0731 \ --trust-remote-code \ --kv-cache-dtype fp8 \ --block-size 256 \ --enable-expert-parallel \ --tensor-parallel-size 8 \ --moe-backend auto \ --attention_config.indexer_kv_dtype fp8 \ --tokenizer-mode deepseek_v4 \ --tool-call-parser deepseek_v4 \ --enable-auto-tool-choice \ --reasoning-parser deepseek_v4 \ --reasoning-config '{"reasoning_parser":"deepseek_v4","reasoning_start_str":"","reasoning_end_str":""}' \ --speculative-config '{"method":"dspark","num_speculative_tokens":7,"draft_sample_method":"probabilistic"}' ``` ### MI300X (8×192GB) Verified single-node serving on MI300X (8×192 GB, CDNA 3) with the same AMD launch as the MI325X/MI355X sections below — the builder lowers `--gpu-memory-utilization` to `0.85` on MI300X (the 192 GB cards leave less headroom at `0.9`), and DSpark speculative decoding is validated on it. ### MI325X (1×256GB) Validated end-to-end with vLLM 0.25.0 and PyTorch HIP 7.2.53211. Use this conservative TP1 eager-mode launch instead of the MI355X throughput settings: ```bash export VLLM_ROCM_USE_AITER=1 vllm serve deepseek-ai/DeepSeek-V4-Flash \ --host 0.0.0.0 \ --port 8002 \ --tensor-parallel-size 1 \ --kv-cache-dtype fp8_e4m3 \ --max-model-len 4096 \ --enable-chunked-prefill \ --max-num-batched-tokens 256 \ --kv-cache-memory-bytes 10000000000 \ --distributed-executor-backend mp \ --trust-remote-code \ --tokenizer-mode deepseek_v4 \ --moe-backend triton_unfused \ --enforce-eager ``` Chunked prefill allows 4K requests while scheduling at most 256 tokens per batch. This configuration loaded the 148.66 GiB checkpoint and passed model-list and chat probes in non-thinking and Think High modes during a 24-hour allocation. Only TP1 at 4K context is verified; TP2+ did not validate successfully. Think Max requires at least 384K context and is not available with this configuration. ### MI355X (4×288GB) If you want to further lower the latency, you can run with `--tensor-parallel-size 8`. ```bash export VLLM_ROCM_USE_AITER=1 vllm serve deepseek-ai/DeepSeek-V4-Flash \ --host localhost \ --port 8001 \ --dtype auto \ --kv-cache-dtype fp8 \ --tensor-parallel-size 4 \ --max-num-seqs 512 \ --max-num-batched-tokens 8192 \ --distributed-executor-backend mp \ --trust-remote-code \ --gpu-memory-utilization 0.9 \ --tokenizer-mode deepseek_v4 \ --reasoning-parser deepseek_v4 \ --tool-call-parser deepseek_v4 \ --enable-auto-tool-choice \ --compilation-config '{"mode": 3, "cudagraph_mode": "FULL_DECODE_ONLY"}' ``` #### MI355X is validated on GSM8K dataset: ##### Launch command ```bash MODEL=deepseek-ai/DeepSeek-V4-Flash lm_eval --model local-completions \ --model_args model=$MODEL,base_url=http://0.0.0.0:8001/v1/completions,num_concurrent=128,max_retries=10,max_gen_toks=2048,timeout=60000 \ --batch_size auto \ --tasks gsm8k \ --num_fewshot 8 \ --output_path . 2>&1 | tee -a eval.log ``` ##### Reported result ```text local-completions ({'model': 'deepseek-ai/DeepSeek-V4-Flash', 'base_url': 'http://0.0.0.0:8001/v1/completions', 'num_concurrent': 128, 'max_retries': 10, 'max_gen_toks': 2048, 'timeout': 60000}), gen_kwargs: ({}), limit: None, num_fewshot: 8, batch_size: auto |Tasks|Version| Filter |n-shot| Metric | |Value | |Stderr| |-----|------:|----------------|-----:|-----------|---|-----:|---|-----:| |gsm8k| 3|flexible-extract| 8|exact_match|↑ |0.9439|± |0.0063| | | |strict-match | 8|exact_match|↑ |0.9431|± |0.0064| ``` ### H200 Single-Node PD (Mooncake) Single-host disaggregated serving: 4 prefill GPUs + 4 decode GPUs on one 8-GPU H200 node, using MooncakeConnector over RDMA for KV cache transfer. **Prefill** (GPUs 0–3, port 8000): ```bash docker run --gpus all \ --privileged --ipc=host -p 8000:8000 \ --network host \ -v ~/.cache/huggingface:/root/.cache/huggingface \ -v /mnt/shared:/mnt/shared \ -e TILELANG_CLEANUP_TEMP_FILES=1 \ -e VLLM_DISABLE_COMPILE_CACHE=1 \ -e VLLM_ENGINE_READY_TIMEOUT_S=3600 \ -e VLLM_LOG_STATS_INTERVAL=1 \ -e VLLM_MOONCAKE_BOOTSTRAP_PORT=8998 \ -e CUDA_VISIBLE_DEVICES=0,1,2,3 \ vllm/vllm-openai:v0.28.0 \ deepseek-ai/DeepSeek-V4-Flash \ --trust-remote-code \ --kv-cache-dtype fp8 \ --block-size 256 \ --port 8000 \ --data-parallel-size 4 \ --enable-expert-parallel \ --tokenizer-mode deepseek_v4 \ --reasoning-parser deepseek_v4 \ --max-model-len auto \ --max-num-batched-tokens 16384 \ --max-num-seqs 8 \ --no-disable-hybrid-kv-cache-manager \ --disable-uvicorn-access-log \ --kv-transfer-config '{"kv_connector":"MooncakeConnector","kv_role":"kv_both","kv_load_failure_policy":"fail","kv_buffer_device":"cuda","kv_connector_extra_config":{"enforce_handshake_compat":false,"mooncake_protocol":"rdma"}}' ``` **Decode** (GPUs 4–7, port 8001): ```bash docker run --gpus all \ --privileged --ipc=host -p 8001:8001 \ --network host \ -v ~/.cache/huggingface:/root/.cache/huggingface \ -v /mnt/shared:/mnt/shared \ -e TILELANG_CLEANUP_TEMP_FILES=1 \ -e VLLM_DISABLE_COMPILE_CACHE=1 \ -e VLLM_ENGINE_READY_TIMEOUT_S=3600 \ -e VLLM_LOG_STATS_INTERVAL=1 \ -e VLLM_MOONCAKE_BOOTSTRAP_PORT=9889 \ -e CUDA_VISIBLE_DEVICES=4,5,6,7 \ vllm/vllm-openai:v0.28.0 \ deepseek-ai/DeepSeek-V4-Flash \ --trust-remote-code \ --kv-cache-dtype fp8 \ --block-size 256 \ --port 8001 \ --data-parallel-size 4 \ --enable-expert-parallel \ --tokenizer-mode deepseek_v4 \ --reasoning-parser deepseek_v4 \ --max-model-len auto \ --max-num-seqs 512 \ --max-num-batched-tokens 512 \ --compilation-config '{"mode":0,"cudagraph_mode":"FULL_DECODE_ONLY","max_cudagraph_capture_size":512,"compile_ranges_endpoints":[512]}' \ --no-disable-hybrid-kv-cache-manager \ --disable-uvicorn-access-log \ --kv-transfer-config '{"kv_connector":"MooncakeConnector","kv_role":"kv_both","kv_load_failure_policy":"fail","kv_buffer_device":"cuda","kv_connector_extra_config":{"enforce_handshake_compat":false,"mooncake_protocol":"rdma"}}' ``` **[Router](https://github.com/vllm-project/router)**: ```bash pip install vllm-router vllm-router --policy round_robin \ --vllm-pd-disaggregation \ --prefill http://localhost:8000 \ --decode http://localhost:8001 \ --host 127.0.0.1 \ --port 30000 \ --intra-node-data-parallel-size 4 \ --kv-connector mooncake ``` ### H200 Single-Node PD (Nixl) Single-host disaggregated serving: 4 prefill GPUs + 4 decode GPUs on one 8-GPU H200 node, using NixlConnector for KV cache transfer. **Prefill** (GPUs 0–3, port 8000): ```bash docker run --gpus all \ --privileged --ipc=host -p 8000:8000 \ --network host \ -v ~/.cache/huggingface:/root/.cache/huggingface \ -v /mnt/shared:/mnt/shared \ -e TILELANG_CLEANUP_TEMP_FILES=1 \ -e VLLM_DISABLE_COMPILE_CACHE=1 \ -e VLLM_ENGINE_READY_TIMEOUT_S=3600 \ -e VLLM_LOG_STATS_INTERVAL=1 \ -e VLLM_NIXL_SIDE_CHANNEL_PORT=5557 \ -e CUDA_VISIBLE_DEVICES=0,1,2,3 \ vllm/vllm-openai:v0.28.0 \ deepseek-ai/DeepSeek-V4-Flash \ --trust-remote-code \ --kv-cache-dtype fp8 \ --block-size 256 \ --port 8000 \ --data-parallel-size 4 \ --enable-expert-parallel \ --tokenizer-mode deepseek_v4 \ --reasoning-parser deepseek_v4 \ --max-model-len auto \ --max-num-batched-tokens 16384 \ --max-num-seqs 8 \ --no-disable-hybrid-kv-cache-manager \ --disable-uvicorn-access-log \ --kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_both"}' ``` **Decode** (GPUs 4–7, port 8001): ```bash docker run --gpus all \ --privileged --ipc=host -p 8001:8001 \ --network host \ -v ~/.cache/huggingface:/root/.cache/huggingface \ -v /mnt/shared:/mnt/shared \ -e TILELANG_CLEANUP_TEMP_FILES=1 \ -e VLLM_DISABLE_COMPILE_CACHE=1 \ -e VLLM_ENGINE_READY_TIMEOUT_S=3600 \ -e VLLM_LOG_STATS_INTERVAL=1 \ -e VLLM_NIXL_SIDE_CHANNEL_PORT=5558 \ -e CUDA_VISIBLE_DEVICES=4,5,6,7 \ vllm/vllm-openai:v0.28.0 \ deepseek-ai/DeepSeek-V4-Flash \ --trust-remote-code \ --kv-cache-dtype fp8 \ --block-size 256 \ --port 8001 \ --data-parallel-size 4 \ --enable-expert-parallel \ --tokenizer-mode deepseek_v4 \ --reasoning-parser deepseek_v4 \ --max-model-len auto \ --max-num-seqs 512 \ --max-num-batched-tokens 512 \ --compilation-config '{"mode":0,"cudagraph_mode":"FULL_DECODE_ONLY","max_cudagraph_capture_size":512,"compile_ranges_endpoints":[512]}' \ --no-disable-hybrid-kv-cache-manager \ --disable-uvicorn-access-log \ --kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_both"}' ``` **[Router](https://github.com/vllm-project/router)**: ```bash pip install vllm-router vllm-router --policy round_robin \ --vllm-pd-disaggregation \ --prefill http://localhost:8000 \ --decode http://localhost:8001 \ --host 127.0.0.1 \ --port 30000 \ --intra-node-data-parallel-size 4 \ --kv-connector nixl ``` --- # thinkingmachines/Inkling-Small on vLLM Source: https://recipes.vllm.ai/thinkingmachines/Inkling-Small Provider: Thinking Machines Lab Released: 2026-07-27T22:33:44.000Z Updated: 2026-09-29 Architecture: moe Parameters: 276B Context length: 1,048,576 Min vLLM: 0.26.0 Tasks: multimodal Verified hardware: MI300X, MI355X Summary: Natively multimodal 276B-parameter MoE from Thinking Machines Lab — 12B active parameters, text/image/audio in, text out, and up to 1M context. ## Variants - default: precision=nvfp4, vram_minimum_gb=180 NVFP4 checkpoint requiring at least 180 GB aggregated VRAM: W4A4 with TP1 on B300/GB300 or TP2 on B200/GB200, and W4A16 with TP4 on H100 or TP2 on H200. - mxfp4: precision=mxfp4, vram_minimum_gb=160, model_id=EmbeddedLLM/Inkling-Small-MXFP4 Quark OCP MXFP4 checkpoint for AMD Instinct. Its measured MI355X TP1 load footprint was 149.03 GiB consumed (~160.0 GB decimal); add KV-cache capacity for serving. Full 1M context is validated at TP1 on MI355X and TP2 on MI300X. MI325X uses the same gfx942 TP2 path but is not yet smoke-tested. - bf16: precision=bf16, vram_minimum_gb=600, model_id=thinkingmachines/Inkling-Small Full BF16 checkpoint requiring at least 600 GB aggregated VRAM: TP4 on B300/GB200/GB300 and AMD Instinct, or TP8 on B200/H200. MI355X TP4 is smoke-tested; MI300X/MI325X use the gfx942 automatic MoE path pending direct BF16 smoke. - fp8: precision=fp8, vram_minimum_gb=332, model_id=RedHatAI/Inkling-Small-FP8-dynamic RedHatAI FP8 (E4M3) with dynamic per-token activation quantization ## Recommended base flags ```bash vllm serve 'thinkingmachines/Inkling-Small-NVFP4' \ --trust-remote-code \ --tokenizer-mode \ inkling ``` Environment: ```bash VLLM_USE_V2_MODEL_RUNNER='1' ``` ## Guide ## Overview TML Inkling-Small is a natively multimodal 276B-parameter Mixture-of-Experts model from Thinking Machines Lab. It activates 12B parameters per token and uses the same Inkling architecture and vLLM serving configuration as its larger 975B-parameter sibling, while substantially reducing the deployment footprint. Inkling-Small accepts text, image, and audio inputs and generates text. It supports Inkling's relative-attention and short-convolution serving path, tool and reasoning parsers, and MTP speculative decoding. ## Variants - **NVFP4 (default):** Requires at least 180 GB aggregated VRAM. Run W4A4 with TP1 on B300/GB300 or TP2 on B200/GB200. On Hopper, run W4A16 with TP4 on H100 or TP2 on H200; the MoE kernel dequantizes the NVFP4 weights to BF16 on the fly. - **BF16:** Requires at least 600 GB aggregated VRAM. Run TP4 on B300/GB200/GB300 or AMD MI300X/MI325X/MI355X, and TP8 on B200/H200. MI355X TP4 is validated; direct MI300X/MI325X BF16 smoke is pending. - **MXFP4 (AMD):** `EmbeddedLLM/Inkling-Small-MXFP4` keeps the full 1M-token context at TP1 on MI355X or TP2 on MI300X. MI355X runs native AITER MXFP4 MoE; MI300X uses the supported Triton on-the-fly MXFP4 emulation path. ## Prerequisites - **Hardware:** B300/GB300 TP1, B200/GB200/H200 TP2, or H100 TP4 for NVFP4; B300/GB200/GB300/MI300X/MI325X/MI355X TP4 or B200/H200 TP8 for BF16; MI355X TP1 or MI300X/MI325X TP2 for AMD MXFP4. - **vLLM:** A nightly build with Inkling support. - **Audio:** Install the optional `vllm[audio]` extra only when serving audio inputs. ## Client Usage Launch the NVFP4 checkpoint in W4A4 mode on B300: ```bash export VLLM_USE_V2_MODEL_RUNNER=1 export FLASH_ATTENTION_CUTE_DSL_CACHE_ENABLED=1 vllm serve thinkingmachines/Inkling-Small-NVFP4 \ --tokenizer-mode inkling \ --reasoning-parser inkling \ --tool-call-parser inkling \ --enable-auto-tool-choice \ --tensor-parallel-size 1 \ --disable-flashinfer-autotune \ --trust-remote-code ``` Select **GB300** for W4A4 with TP1, or **B200**/**GB200** for W4A4 with TP2. On H100, the builder emits TP4; on H200, it emits TP2. vLLM runs W4A16 with on-the-fly dequantization. A reported 4×H100 deployment served a plain chat request, but JSON-schema structured output returned HTTP 500; see [vLLM #51693](https://github.com/vllm-project/vllm/issues/51693). H100 is not marked verified for the full recipe. On AMD, select the **MXFP4** variant. To fit 1M context, MI300X TP2 and MI355X TP1. MI355X uses `--moe-backend aiter`; MI300X uses `--moe-backend auto` because native AITER W4A4 is gated to gfx950, while gfx942 falls back to `OCP_MXQuantizationEmulationTritonExperts`. Inkling relative attention remains on its model-specific Triton path (`INKLING_GFX950_GLUON=0`), which was faster than the optional gfx950 Gluon path in the serving benchmark. Equivalent MI355X launch: ```bash export VLLM_USE_V2_MODEL_RUNNER=1 export VLLM_ROCM_USE_AITER=1 export VLLM_ROCM_USE_AITER_MOE=1 export INKLING_GFX950_GLUON=0 vllm serve EmbeddedLLM/Inkling-Small-MXFP4 \ --tensor-parallel-size 1 \ --tokenizer-mode inkling \ --reasoning-parser inkling \ --tool-call-parser inkling \ --enable-auto-tool-choice \ --moe-backend aiter \ --block-size 128 \ --trust-remote-code ``` Query the server with the OpenAI SDK: ```python from openai import OpenAI client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1") response = client.chat.completions.create( model="thinkingmachines/Inkling-Small-NVFP4", messages=[{ "role": "user", "content": [ {"type": "text", "text": "Describe this image."}, {"type": "image_url", "image_url": {"url": "https://example.com/photo.jpg"}}, ], }], ) print(response.choices[0].message.content) ``` ## References - [TML Inkling announcement](https://thinkingmachines.ai/news/introducing-inkling/#inkling-small) - [Inkling-Small MXFP4 for AMD](https://huggingface.co/EmbeddedLLM/Inkling-Small-MXFP4) - [TML Inkling recipe](https://docs.vllm.ai/projects/recipes/en/latest/thinkingmachines/Inkling.html) --- # tencent/Hy4-preview on vLLM Source: https://recipes.vllm.ai/tencent/Hy4-preview Provider: Hunyuan (Tencent) Released: 2026-08-27T08:52:40.000Z Updated: 2026-09-28 Architecture: moe Parameters: 770B Context length: 1,048,576 Min vLLM: 0.29.0 Tasks: text Verified hardware: B200, B300, MI325X, MI355X Summary: Tencent Hunyuan Hy4-preview — scaled-up MoE language model (770B total / 49B active) with a 10B MTP layer for speculative decoding, 1M context, and hy_v4 tool/reasoning parsers ## Variants - default: precision=bf16, vram_minimum_gb=1848 Full precision BF16 — 16xB200 or 8xB300 minimum for weights + KV cache - fp8: precision=mxfp8, vram_minimum_gb=924, model_id=tencent/Hy4-preview-FP8 MXFP8-quantized instruct checkpoint (tencent/Hy4-preview-FP8) — fits a single 8xB200 / 8xB300 node with wide headroom. Verified AMD path on 8xMI355X and 8xMI325X. ## Guide # Hy4-preview Usage Guide Hy4-preview is Tencent Hunyuan's new-generation open-source Mixture-of-Experts language model. Its backbone has 770B total parameters with 49B activated per token across 78 layers. The first layer uses a dense FFN; the other 77 layers use 256 routed experts (top-8) and 1 shared expert. A native MTP layer adds 10B total parameters (0.7B activated) for speculative decoding. On the architecture side, inspired by DeepSeek and GLM, the attention module employs Gated DeepSeek Sparse Attention (Gated DSA) with IndexCache for cross-layer sparse index reuse. The residual pathway uses iHC (identity Hyper-Connections) to expand inter-layer information flow. ## vLLM Setup Choose one of the following setup methods. ### Using Docker ```bash docker run --gpus all \ -p 8000:8000 \ --ipc=host \ -e VLLM_ENABLE_HPC_OPS=1 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ vllm/vllm-openai:hy4-preview tencent/Hy4-preview-FP8 \ --tensor-parallel-size 8 \ --speculative-config '{"num_speculative_tokens":3,"method":"mtp"}' \ --attention-backend FLASHMLA_SPARSE \ --tool-call-parser hy_v4 \ --reasoning-parser hy_v4 \ --enable-auto-tool-choice \ --port 8000 \ --served-model-name hy4-preview ``` Build vLLM from source: ```bash uv venv --python 3.12 --seed --managed-python source .venv/bin/activate git clone https://github.com/vllm-project/vllm.git cd vllm uv pip install --editable . --torch-backend=auto ``` ## vLLM Deployment Start the FP8 model with MTP enabled: ```bash export VLLM_ENABLE_HPC_OPS=1 vllm serve tencent/Hy4-preview-FP8 \ --tensor-parallel-size 8 \ --speculative-config '{"num_speculative_tokens":3,"method":"mtp"}' \ --attention-backend FLASHMLA_SPARSE \ --tool-call-parser hy_v4 \ --reasoning-parser hy_v4 \ --enable-auto-tool-choice \ --port 8000 \ --served-model-name hy4-preview ``` ### Serving on 8×AMD MI355X (gfx950) Spec decoding (native MTP, `k=3`) is opt-in. ```bash export VLLM_ROCM_USE_AITER=1 vllm serve tencent/Hy4-preview-FP8 \ --tensor-parallel-size 8 \ --max-model-len 262144 \ --gpu-memory-utilization 0.85 \ --max-num-batched-tokens 8192 \ --tool-call-parser hy_v4 \ --reasoning-parser hy_v4 \ --enable-auto-tool-choice \ --port 8000 \ --served-model-name hy4-preview ``` **Context window.** 262,144 is the default verified window. MXFP8 (`tencent/Hy4-preview-FP8`) also comes up at the full 1,048,576 ceiling at util `0.92` (1.21× concurrency for a 1M request). ### Serving on 8×AMD MI325X (gfx942) Select the FP8 variant. Since gfx942 dequantizes MXFP8 weights to BF16, the context window is not the same as on the MI355X. The verified serve is context 139264, FP8 KV, and the AITER MoE, MHA, RMSNorm, and linear toggles. At GPU memory utilization 0.96 the KV cache holds 530,912 tokens, about 3.81× one 139,264-token request. ```bash export VLLM_ROCM_USE_AITER=1 export VLLM_ROCM_USE_AITER_MOE=1 export VLLM_ROCM_USE_AITER_MHA=1 export VLLM_ROCM_USE_AITER_RMSNORM=1 export VLLM_ROCM_USE_AITER_LINEAR=1 export VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4 export VLLM_USE_RUST_FRONTEND=1 vllm serve tencent/Hy4-preview-FP8 \ --tensor-parallel-size 8 \ --trust-remote-code \ --kv-cache-dtype fp8 \ --max-model-len 139264 \ --max-num-batched-tokens 16384 \ --gpu-memory-utilization 0.96 \ --tool-call-parser hy_v4 \ --enable-auto-tool-choice \ --reasoning-parser hy_v4 ``` ## OpenAI Client Example Tencent recommends `temperature=0.9` and `top_p=1.0`. Reasoning defaults to `high`, which is suitable for math, coding, and other complex tasks. ```python from openai import OpenAI client = OpenAI(base_url="http://127.0.0.1:8000/v1", api_key="EMPTY") response = client.chat.completions.create( model="hy4-preview", messages=[ {"role": "user", "content": "Hello! Can you briefly introduce yourself?"}, ], temperature=0.9, top_p=1.0, ) output_msg = response.choices[0].message print(output_msg.reasoning) # chain-of-thought print(output_msg.content) # final answer ``` For a direct response without deep reasoning, pass `reasoning_effort="no_think"`: ```python response = client.chat.completions.create( model="hy4-preview", messages=[ {"role": "user", "content": "Hello! Can you briefly introduce yourself?"}, ], temperature=0.9, top_p=1.0, extra_body={"chat_template_kwargs": {"reasoning_effort": "no_think"}}, ) print(response.choices[0].message.content) ``` ### cURL Usage ```bash curl http://localhost:8000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "hy4-preview", "messages": [ {"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Hello! Can you briefly introduce yourself?"} ], "temperature": 0.9, "top_p": 1.0 }' ``` ## References - [Hugging Face: Hy4-preview](https://huggingface.co/tencent/Hy4-preview) - [Hugging Face: Hy4-preview-FP8](https://huggingface.co/tencent/Hy4-preview-FP8) - [ModelScope: Hy4-preview](https://modelscope.cn/models/Tencent-Hunyuan/Hy4-preview) - [GitHub: Tencent-Hunyuan/Hy4-preview](https://github.com/Tencent-Hunyuan/Hy4-preview) - [GitCode: Hy4-preview](https://ai.gitcode.com/tencent_hunyuan/Hy4-preview) - [CNB: Hy4-preview](https://cnb.cool/ai-models/tencent/Hy4-preview) - [AngelSlim quantization toolkit](https://github.com/tencent/AngelSlim) --- # Google/gemma-4-26B-A4B-it on vLLM Source: https://recipes.vllm.ai/Google/gemma-4-26B-A4B-it Provider: Google Released: 2026-03-11T21:25:57.000Z Updated: 2026-09-27 Architecture: moe Parameters: 26B Context length: 131,072 Min vLLM: 0.25.0 Tasks: multimodal, text Verified hardware: H100, H200, MI300X, MI325X, MI355X, TPU v6e (Trillium), TPU v7 (Ironwood), Xeon 6, DGX Spark (GB10), DGX Station (GB300), RTX Pro 6000, RTX 5090, Arc Pro B60, Arc Pro B70 Summary: Google's Gemma 4 MoE multimodal model (26B total / 4B active) with 128 fine-grained experts, top-8 routing, thinking mode, and tool-use protocol. ## Variants - default: precision=bf16, vram_minimum_gb=64 Full BF16 — single 80 GB NVIDIA GPU or 1x MI300X/MI325X/MI350X/MI355X or 2x Xeon6/Xeon5 NUMA nodes - fp8: precision=fp8, vram_minimum_gb=32, model_id=RedHatAI/gemma-4-26B-A4B-it-FP8-dynamic FP8 (E4M3) weights with dynamic per-token activation quantization — Hopper or Blackwell - nvfp4: precision=nvfp4, vram_minimum_gb=16, model_id=RedHatAI/gemma-4-26B-A4B-it-NVFP4 NVFP4 (4-bit) quantized weights — requires Blackwell (B200/B300) - nvidia_nvfp4: precision=nvfp4, vram_minimum_gb=16, model_id=nvidia/Gemma-4-26B-A4B-NVFP4 NVIDIA ModelOpt NVFP4 (4-bit) quantized weights — validated on the single-GPU Blackwell desktops (DGX Spark GB10, DGX Station GB300, RTX PRO 6000) ## Guide ## Overview [Gemma 4 26B-A4B](https://huggingface.co/google/gemma-4-26B-A4B-it) is the Mixture-of-Experts member of Google's Gemma 4 family — 26B total parameters with only 4B active per token via 128 fine-grained experts and top-8 routing. It supports text + images natively, structured thinking, function calling, and dynamic vision resolution. Intel Xeon 6 CPU support remains verified. Benchmark-specific TP/DP, CPU binding, sequence limits, and repro-only timeouts remain deployment choices. ### Key Features - **MoE**: 128 fine-grained experts with top-8 routing and custom GELU-activated FFN. - **Multimodal**: Text + images natively (video via custom frame-extraction pipeline). Audio is only supported on the smaller E2B/E4B variants. - **Dual Attention**: Alternating sliding-window (local) and global attention with different head dimensions. - **Thinking Mode**: Structured reasoning via `<|channel>thought\n...` delimiters. - **Function Calling**: Custom tool-call protocol with dedicated special tokens. - **Dynamic Vision Resolution**: Per-request configurable vision token budget (70, 140, 280, 560, 1120 tokens). TPU support is provided through [vLLM TPU](https://github.com/vllm-project/tpu-inference) with recipes for [Trillium](https://github.com/AI-Hypercomputer/tpu-recipes/tree/main/inference/trillium/vLLM/Gemma4) and [Ironwood](https://github.com/AI-Hypercomputer/tpu-recipes/blob/main/inference/ironwood/vLLM/Gemma4/). ## Prerequisites - Hardware: Intel Xeon 6/Xeon 5 CPUs - vLLM >= 0.25.0 ### pip (NVIDIA CUDA) ```bash uv venv source .venv/bin/activate uv pip install -U vllm --pre \ --extra-index-url https://wheels.vllm.ai/nightly/cu129 \ --extra-index-url https://download.pytorch.org/whl/cu129 \ --index-strategy unsafe-best-match ``` ### pip (AMD ROCm: MI300X, MI325X, MI350X, MI355X) Requires Python 3.12, ROCm 7.2.1, glibc >= 2.35 (Ubuntu 22.04+). ```bash uv venv --python 3.12 source .venv/bin/activate uv pip install vllm --pre \ --extra-index-url https://wheels.vllm.ai/rocm/nightly/rocm721 --upgrade ``` ### pip (Intel Xeon 6 CPUs) For Intel and AMD x86 CPUs, follow the [CPU pre-built wheels](https://docs.vllm.ai/en/latest/getting_started/installation/cpu/#pre-built-wheels) installation instructions. ### Docker ```bash docker pull vllm/vllm-openai:gemma4-0505-cu129 # NVIDIA Hopper (H100/H200, CUDA 12.9) docker pull vllm/vllm-openai:gemma4-0505-cu130 # NVIDIA Blackwell (B200/B300, CUDA 13.0) docker pull vllm/vllm-openai-rocm:latest # AMD docker pull vllm/vllm-openai-cpu:latest-x86_64 # For Intel Xeon 6 docker pull vllm/vllm-openai-xpu:latest # Intel XPU (B60 / B70) ``` TPU images are published separately by [vllm-project/tpu-inference](https://github.com/vllm-project/tpu-inference); see the Trillium / Ironwood tpu-recipes below for the pinned tag. ## Deployment Configurations ### 26B MoE on 1x A100/H100 (BF16) ```bash vllm serve google/gemma-4-26B-A4B-it \ --max-model-len 32768 \ --gpu-memory-utilization 0.90 ``` ### Full-Featured Server Launch Enables text, image, thinking, and tool calling: ```bash vllm serve google/gemma-4-26B-A4B-it \ --max-model-len 16384 \ --gpu-memory-utilization 0.90 \ --enable-auto-tool-choice \ --reasoning-parser gemma4 \ --tool-call-parser gemma4 \ --chat-template examples/tool_chat_template_gemma4.jinja \ --limit-mm-per-prompt.image 4 \ --async-scheduling \ --host 0.0.0.0 \ --port 8000 ``` ### Docker (NVIDIA) ```bash docker run -itd --name gemma4-moe \ --ipc=host --network host --shm-size 16G --gpus all \ -v ~/.cache/huggingface:/root/.cache/huggingface \ vllm/vllm-openai:gemma4-0505-cu129 \ --model google/gemma-4-26B-A4B-it \ --max-model-len 32768 \ --gpu-memory-utilization 0.90 \ --host 0.0.0.0 --port 8000 ``` Swap `vllm/vllm-openai:gemma4-0505-cu129` for `vllm/vllm-openai:gemma4-0505-cu130` on Blackwell (B200/B300). ### Docker (AMD MI300X/MI325X/MI350X/MI355X) ```bash docker run -itd --name gemma4-rocm \ --ipc=host --network=host --privileged \ --cap-add=CAP_SYS_ADMIN --device=/dev/kfd --device=/dev/dri \ --group-add=video --cap-add=SYS_PTRACE \ --security-opt=seccomp=unconfined --shm-size 16G \ -v ~/.cache/huggingface:/root/.cache/huggingface \ vllm/vllm-openai-rocm:latest \ --model google/gemma-4-26B-A4B-it \ --host 0.0.0.0 --port 8000 ``` ### Docker (Cloud TPU — Trillium / Ironwood) TPU uses the separate `vllm/vllm-tpu` image (no pip wheel). Pull the tag specified by the upstream [Trillium](https://github.com/AI-Hypercomputer/tpu-recipes/tree/main/inference/trillium/vLLM/Gemma4) or [Ironwood](https://github.com/AI-Hypercomputer/tpu-recipes/blob/main/inference/ironwood/vLLM/Gemma4/) recipe, then run: ```bash docker run -itd --name gemma4-tpu \ --privileged --network host --shm-size 16G \ -v /dev/shm:/dev/shm -e HF_TOKEN=$HF_TOKEN \ vllm/vllm-tpu:latest \ --model google/gemma-4-26B-A4B-it \ --tensor-parallel-size 8 \ --max-model-len 16384 \ --disable_chunked_mm_input \ --host 0.0.0.0 --port 8000 ``` Trillium requires a 4-chip slice minimum; Ironwood runs on a single chip. ### Intel Xeon 6 Deployment via Docker Launch the x86 CPU vLLM Docker container for `google/gemma-4-26B-A4B-it`: ```bash docker run \ --privileged --ipc=host -p 8000:8000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ vllm/vllm-openai-cpu:latest-x86_64 google/gemma-4-26B-A4B-it \ --tensor-parallel-size 1 \ --enable-auto-tool-choice \ --tool-call-parser gemma4 \ --chat-template examples/tool_chat_template_gemma4.jinja \ --reasoning-parser gemma4 ``` For additional Intel Xeon 6 deployment details, see the Intel Software Catalog entries for [Gemma 4 26B-A4B IT](https://aiswcatalog.intel.com/models/google-gemma-4-26b-a4b-it). The following settings are intentionally not prescribed as portable CPU model defaults because they depend on the workload, hardware, or runtime environment: - `--max-num-batched-tokens`: scheduler/throughput tuning. - `--max-num-seqs`: concurrency and scheduler-capacity tuning. - `--gpu-memory-utilization`: platform memory-budget tuning. - `--no-enable-prefix-caching`: workload/benchmark cache-policy tuning. - `VLLM_ENGINE_ITERATION_TIMEOUT_S`: operational runtime timeout. These settings may still appear in validated hardware-specific overrides. Tune them at deployment time based on platform resources, workload shape, and latency/throughput goals. ### Docker (Intel XPU B60 / B70, TP=4, Full-Featured) Validated on 4x Intel Arc Pro B60 / B70 with the official vLLM XPU image. ```bash docker run --device /dev/dri \ -v /dev/dri/by-path:/dev/dri/by-path --shm-size=16g \ --privileged --ipc=host -p 8000:8000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ vllm/vllm-openai-xpu:latest \ google/gemma-4-26B-A4B-it \ --enforce-eager \ --attention-backend flash_attn \ --host 0.0.0.0 \ --port 8000 \ --trust-remote-code \ --enable-auto-tool-choice \ --reasoning-parser gemma4 \ --tool-call-parser gemma4 \ --chat-template examples/tool_chat_template_gemma4.jinja \ --limit-mm-per-prompt.image 4 \ --async-scheduling \ --gpu-memory-utilization 0.9 \ --no-enable-prefix-caching \ --max-num-batched-tokens 8192 \ --max-model-len 5120 \ --tensor-parallel-size 4 ``` ## Client Usage ### Text Generation ```python from openai import OpenAI client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY") response = client.chat.completions.create( model="google/gemma-4-26B-A4B-it", messages=[{"role": "user", "content": "Write a poem about the ocean."}], max_tokens=512, temperature=0.7, ) print(response.choices[0].message.content) ``` ### Image Understanding ```python response = client.chat.completions.create( model="google/gemma-4-26B-A4B-it", messages=[{"role": "user", "content": [ {"type": "image_url", "image_url": {"url": "https://upload.wikimedia.org/wikipedia/commons/thumb/3/3a/Cat03.jpg/1200px-Cat03.jpg"}}, {"type": "text", "text": "Describe this image in detail."}, ]}], max_tokens=1024, ) ``` ### Thinking Mode ```bash vllm serve google/gemma-4-26B-A4B-it \ --max-model-len 16384 \ --reasoning-parser gemma4 \ --tool-call-parser gemma4 \ --enable-auto-tool-choice \ --chat-template examples/tool_chat_template_gemma4.jinja ``` Enable per-request via `extra_body={"chat_template_kwargs": {"enable_thinking": True}}`. ### Dynamic Vision Resolution Supported values: 70, 140, 280 (default), 560, 1120 tokens/image. ```bash vllm serve google/gemma-4-26B-A4B-it \ --mm-processor-kwargs '{"max_soft_tokens": 560}' ``` ## Configuration Tips - Set `--max-model-len` to match your workload. - `--gpu-memory-utilization 0.90-0.95` maximizes KV cache. - Text-only workloads: `--limit-mm-per-prompt '{"image": 0, "audio": 0}'`. - `--async-scheduling` improves throughput. - FP8 KV cache (`--kv-cache-dtype fp8`) saves ~50% KV memory. - For MoE, TEP (tensor-expert parallelism) and DEP (data-expert parallelism) strategies scale better than pure TP at large node counts. ## Quantized Variants Three pre-quantized checkpoints are available: - [`RedHatAI/gemma-4-26B-A4B-it-FP8-dynamic`](https://huggingface.co/RedHatAI/gemma-4-26B-A4B-it-FP8-dynamic) — FP8 (E4M3) weights with dynamic per-token activation quantization; runs on Hopper and Blackwell. - [`RedHatAI/gemma-4-26B-A4B-it-NVFP4`](https://huggingface.co/RedHatAI/gemma-4-26B-A4B-it-NVFP4) — NVFP4 (4-bit) weights; requires Blackwell (B200/B300). - [`nvidia/Gemma-4-26B-A4B-NVFP4`](https://huggingface.co/nvidia/Gemma-4-26B-A4B-NVFP4) — NVIDIA ModelOpt NVFP4 (4-bit) weights; this is the checkpoint validated on the single-GPU Blackwell desktops below. Pick them from the **Variant** dropdown above, or pass the repo id directly to `vllm serve`. ### NVFP4 on the Blackwell desktops [`nvidia/Gemma-4-26B-A4B-NVFP4`](https://huggingface.co/nvidia/Gemma-4-26B-A4B-NVFP4) has been run end to end on DGX Spark (GB10), DGX Station (GB300) and RTX PRO 6000, in each case on one GPU at TP1 with MTP drafting against [`google/gemma-4-26B-A4B-it-assistant`](https://huggingface.co/google/gemma-4-26B-A4B-it-assistant). Select the **NVIDIA NVFP4** variant, enable **Spec Decoding**, and switch the hardware pill — memory utilization, batch size, prefill scheduling and the drafter's MoE backend all differ per box and are encoded per hardware, so the generated command is the tested one. Highlights: - **DGX Spark (GB10)** — 128 GB shared with the CPU, so `--gpu-memory-utilization 0.8` and an 8-way batch. Serve from `eugr/spark-vllm:nightly-20260704`: vLLM 0.24 and the generic nightly both fail on this checkpoint, so the pip tab is hidden on this hardware. - **DGX Station (GB300)** — 252 GB on one GPU, so `--gpu-memory-utilization 0.92` with chunked prefill and a 64-way batch. - **RTX PRO 6000** — 96 GB, `--gpu-memory-utilization 0.92` and an 8-way batch. All three run the full 262,144-token context, above the 131,072 in `config.json`, and pass `"moe_backend":"triton"` in `--speculative-config` because the drafter's default fused MoE path is not built in these images. The GB300 and RTX PRO 6000 runs used an internal NVIDIA nightly vLLM build. As noted under Speculative Decoding below, Gemma 4 MTP has not landed in a stable release, so the default `vllm/vllm-openai:latest` tag will not serve these commands with Spec Decoding enabled — use a nightly wheel or a nightly image on those two boxes. ## Throughput vs Latency | Goal | TP | `--max-num-seqs` | Notes | |------|----|------------------|-------| | Max throughput | 1-2 | 256-512 | Best tok/s per GPU | | Min latency | 4-8 | 8-16 | Best TTFT/TPOT | | Balanced | 2 | 128 | Mixed workloads | ## Speculative Decoding (MTP) Enable the **Spec Decoding** feature toggle (above) or add `--speculative-config` manually to use MTP drafting with the [assistant model](https://huggingface.co/google/gemma-4-26B-A4B-it-assistant). Recommended `num_speculative_tokens`: 4 for this model. See the [Gemma 4 usage guide](../../Google/Gemma4) for details and benchmarks. > **Note:** MTP speculative decoding for Gemma 4 is only available on the vLLM nightly build — it has not yet landed in a stable release. Install via the nightly wheel (`uv pip install -U vllm --pre --extra-index-url https://wheels.vllm.ai/nightly/cu129 …`) or use the `vllm/vllm-openai:gemma4-0505-cu129` / `vllm/vllm-openai:gemma4-0505-cu130` images above; the standard `:latest` stable tag does not include this feature. ## References - [Model card](https://huggingface.co/google/gemma-4-26B-A4B-it) - [FP8 variant](https://huggingface.co/RedHatAI/gemma-4-26B-A4B-it-FP8-dynamic) - [NVFP4 variant](https://huggingface.co/RedHatAI/gemma-4-26B-A4B-it-NVFP4) - [NVIDIA ModelOpt NVFP4 variant](https://huggingface.co/nvidia/Gemma-4-26B-A4B-NVFP4) - [Gemma docs](https://ai.google.dev/gemma/docs) - [vLLM Gemma 4 tool-call template](https://github.com/vllm-project/vllm/blob/main/examples/tool_chat_template_gemma4.jinja) - [TPU recipes: Trillium](https://github.com/AI-Hypercomputer/tpu-recipes/tree/main/inference/trillium/vLLM/Gemma4) - [TPU recipes: Ironwood](https://github.com/AI-Hypercomputer/tpu-recipes/blob/main/inference/ironwood/vLLM/Gemma4/) - [Xeon 6 recipe: Gemma 4 26B-A4B IT](https://aiswcatalog.intel.com/models/google-gemma-4-26b-a4b-it) --- # Google/diffusiongemma-26B-A4B-it on vLLM Source: https://recipes.vllm.ai/Google/diffusiongemma-26B-A4B-it Provider: Google Released: 2026-06-09T12:40:12.000Z Updated: 2026-09-23 Architecture: moe Parameters: 26B Context length: 262,144 Min vLLM: 0.24.0 Tasks: multimodal, text Verified hardware: H100, H200, DGX Spark (GB10), Xeon 6, Arc Pro B60, Arc Pro B70 Summary: Google's DiffusionGemma — a block-diffusion language model built on Gemma 4's MoE backbone (26B total / 4B active). Generates tokens via iterative denoising over a fixed-length canvas rather than left-to-right autoregressive decoding, enabling higher throughput with parallel block generation. ## Variants - default: precision=bf16, vram_minimum_gb=64 Full BF16 — single 80 GB NVIDIA GPU (H100/H200). Requires --max-num-seqs 4 to avoid OOM from diffusion state buffers. - fp8: precision=fp8, vram_minimum_gb=32, model_id=RedHatAI/diffusiongemma-26B-A4B-it-FP8-dynamic FP8 (E4M3) weights with dynamic per-token activation quantization - nvfp4: precision=nvfp4, vram_minimum_gb=24, model_id=RedHatAI/diffusiongemma-26B-A4B-it-NVFP4 NVFP4 (4-bit) quantized weights - nvidia_nvfp4: precision=nvfp4, vram_minimum_gb=24, model_id=nvidia/diffusiongemma-26B-A4B-it-NVFP4 NVIDIA NVFP4 (4-bit) quantized weights ## Recommended base flags ```bash vllm serve 'google/diffusiongemma-26B-A4B-it' \ --max-num-seqs \ 4 ``` ## Guide ## Overview [DiffusionGemma 26B-A4B](https://huggingface.co/google/diffusiongemma-26B-A4B-it) is a block-diffusion language model built on Gemma 4's MoE backbone. Instead of generating tokens one at a time (autoregressive), it generates blocks of 256 tokens simultaneously via iterative denoising over a fixed canvas — trading higher time-to-first-token for significantly higher per-request generation throughput. ### Key Architecture Features - **Block Diffusion**: Generates tokens in 256-token canvas blocks via iterative denoising (up to 48 steps per block). - **MoE Backbone**: Same Gemma 4 architecture — 128 fine-grained experts with top-8 routing, 26B total / 4B active parameters. - **Entropy-Bound Sampling**: Uses an entropy-based sampler (`diffusion_sampler: entropy_bound`, `entropy_bound: 0.1`) for the denoising process. - **Multimodal**: Supports text + images via the Gemma 4 vision encoder. - **Thinking Mode**: Supports structured reasoning via `<|channel>thought\n...` delimiters. - **Function Calling**: Supports Gemma 4's tool-call protocol (works best in thinking mode). ### Important Deployment Notes DiffusionGemma requires several specific flags that differ from standard Gemma 4: | Flag | Why | |------|-----| | `--max-num-seqs 4` | The diffusion state buffers (`self_conditioning_probs`) pre-allocate `max_seqs × canvas_length × vocab_size` tensors. With Gemma's 262K vocab and canvas_length=256, higher values cause OOM. | | `--generation-config vllm` | The checkpoint's `generation_config.json` sets `max_tokens: 256`, which would override per-request limits. This flag ignores it. | | `--gpu-memory-utilization 0.85` | Leaves headroom for activation memory during denoising. | | `--hf-overrides '{"diffusion_sampler":"entropy_bound","diffusion_entropy_bound":0.1}'` | Configures the entropy-bound denoising sampler. | | `--diffusion-config '{"canvas_length": 256}'` | Sets the canvas block size for generation. | ## Prerequisites DiffusionGemma requires a vLLM build with diffusion model support, available in the Gemma docker image: ```bash docker pull vllm/vllm-openai:gemma ``` ## Deployment ### Single GPU (H100/H200, BF16) ```bash docker run -itd --name diffusiongemma \ --ipc=host --network host --gpus all \ -v ~/.cache/huggingface:/root/.cache/huggingface \ vllm/vllm-openai:gemma \ --model google/diffusiongemma-26B-A4B-it \ --max-model-len 262144 \ --max-num-seqs 4 \ --gpu-memory-utilization 0.85 \ --host 0.0.0.0 --port 8000 ``` ### Intel Xeon 6 (CPU, BF16) DiffusionGemma runs on Intel Xeon 6 CPUs using vLLM's CPU backend with AMX BF16 acceleration. The generated example uses TP=1. Adjust `--tensor-parallel-size` based on the detected NUMA topology and deployment requirements. ```bash docker run \ --privileged --ipc=host -p 8000:8000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ vllm/vllm-openai-cpu:latest-x86_64 google/diffusiongemma-26B-A4B-it \ --max-num-seqs 4 \ --tensor-parallel-size 1 \ --enable-auto-tool-choice \ --tool-call-parser gemma4 \ --chat-template examples/tool_chat_template_gemma4.jinja \ --reasoning-parser gemma4 ``` ### Intel XPU (B60 / B70, BF16) Validated on 4x Intel Arc Pro B60 / B70 (B60 24 GB, B70 32 GB per card) with the official vLLM XPU image `vllm/vllm-openai-xpu:latest`. `--enforce-eager` and `--attention-backend TRITON_ATTN` are required on this backend. Select Arc Pro B60/B70 in the hardware picker above for the exact command. ### Full-Featured Server (text + image + thinking + tools) ```bash docker run -itd --name diffusiongemma \ --ipc=host --network host --gpus all \ -v ~/.cache/huggingface:/root/.cache/huggingface \ vllm/vllm-openai:gemma \ --model google/diffusiongemma-26B-A4B-it \ --max-model-len 262144 \ --max-num-seqs 4 \ --gpu-memory-utilization 0.85 \ --mm-processor-kwargs '{"max_soft_tokens": 1120}' \ --limit-mm-per-prompt '{"image": 7}' \ --enable-auto-tool-choice \ --tool-call-parser gemma4 \ --reasoning-parser gemma4 \ --host 0.0.0.0 --port 8000 ``` ### Jev-Style Structured Decisions (Nightly) vLLM's structured-read support for DiffusionGemma can answer yes/no, multiple-choice, and scored questions with a confidence value. The example server seeds a canvas with the response template, leaves only the answer slots noisy, and reads a probability distribution from every slot after one denoising step. It exposes the Jev-compatible `/v1/systemone` endpoint; this is an example server rather than a standard vLLM endpoint. Use the pinned nightly image that contains the merged structured-read support: ```bash docker run -d --name dgemma \ --gpus '"device=0"' \ --ipc=host \ -p 8000:8000 \ -p 8011:8011 \ -v "$HOME/.cache/huggingface:/root/.cache/huggingface" \ vllm/vllm-openai:nightly-e9757321527ca1ecd514c07c1418dd2c53da3d19 \ google/diffusiongemma-26B-A4B-it \ --served-model-name dgemma \ --diffusion-config '{"canvas_length":64}' \ --max-logprobs 32 \ --enable-prefix-caching ``` Wait for the vLLM server to become healthy, then start the example server in the same container: ```bash until curl -fsS localhost:8000/health >/dev/null; do sleep 2; done docker exec -d dgemma \ python /vllm-workspace/examples/features/structured_diffusion/structured_server.py \ --upstream http://127.0.0.1:8000 \ --tokenizer google/diffusiongemma-26B-A4B-it \ --canvas 64 ``` After the example server becomes healthy, send a Jev-style decision request: ```bash until curl -fsS localhost:8011/health >/dev/null; do sleep 2; done curl -sS localhost:8011/v1/systemone \ -H 'content-type: application/json' \ -d '{ "model": "jev-latest", "state": {"ticket": "Everything is down, demo at noon."}, "questions": { "urgent": { "type": "noul", "instructions": "Needs a reply within the hour?" } } }' ``` ## Client Usage ### Text Generation ```python from openai import OpenAI client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY") response = client.chat.completions.create( model="google/diffusiongemma-26B-A4B-it", messages=[{"role": "user", "content": "Write a poem about the ocean."}], max_tokens=512, temperature=0.7, ) print(response.choices[0].message.content) ``` ### Offline Inference ```python from vllm import LLM, SamplingParams from transformers import AutoTokenizer model_path = "google/diffusiongemma-26B-A4B-it" tokenizer = AutoTokenizer.from_pretrained(model_path) llm = LLM( model=model_path, max_num_seqs=4, gpu_memory_utilization=0.85, hf_overrides={ "diffusion_sampler": "entropy_bound", "diffusion_entropy_bound": 0.1, }, diffusion_config={"canvas_length": 256}, ) messages = [{"role": "user", "content": "Explain quantum computing in simple terms."}] prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True) outputs = llm.generate(prompt, SamplingParams(temperature=0.0, max_tokens=1024)) print(outputs[0].outputs[0].text) ``` ### Thinking Mode Launch the server with `--reasoning-parser gemma4`, then enable per-request: ```python response = client.chat.completions.create( model="google/diffusiongemma-26B-A4B-it", messages=[{"role": "user", "content": "What is the derivative of x^3 * ln(x)?"}], max_tokens=32768, extra_body={"chat_template_kwargs": {"enable_thinking": True}}, ) ``` ## Performance Characteristics Compared to the autoregressive Gemma 4 26B-A4B baseline (single H100, concurrency=1, SPEED-Bench): | Metric | Gemma 4 26B-A4B (baseline) | DiffusionGemma 26B-A4B | |--------|---------------------------|------------------------| | Output TPS | 199 tok/s | 375 tok/s (**1.9×**) | | E2E Request Time (mean) | 2.87s | 0.88s (**3.3× faster**) | | TTFT (mean) | 53ms | 489ms (higher due to canvas denoising setup) | | Per-request Gen TPS (mean) | 205 tok/s | 1,282 tok/s (**6.2×**) | The diffusion model trades higher time-to-first-token for significantly higher generation throughput. It generates fewer total tokens per prompt on average (shorter outputs) but completes requests much faster. ## Known Limitations - **TTFT**: ~10× higher than autoregressive baseline — the model must denoise an entire canvas block before emitting the first token. - **`--max-num-seqs`**: Must be kept low (≤4) due to the large diffusion state tensors. Higher values cause CUDA OOM. - **Audio**: Not supported — the diffusion checkpoints do not include an audio encoder. - **Function Calling**: Works best in thinking mode. Non-thinking mode may answer directly instead of emitting tool calls. ## References - [Model card](https://huggingface.co/google/diffusiongemma-26B-A4B-it) - [vLLM structured-read support for DiffusionGemma](https://github.com/vllm-project/vllm/pull/57250) - [djev-run Cloud Run deployment](https://github.com/taeold/djev-run) - [Gemma 4 26B-A4B (autoregressive baseline)](https://huggingface.co/google/gemma-4-26B-A4B-it) - [Gemma docs](https://ai.google.dev/gemma/docs) --- # Qwen/Qwen3-30B-A3B on vLLM Source: https://recipes.vllm.ai/Qwen/Qwen3-30B-A3B Provider: Qwen Released: 2025-04-27T03:43:05.000Z Updated: 2026-09-23 Architecture: moe Parameters: 30.5B Context length: 40,960 Min vLLM: 0.8.5 Tasks: text Verified hardware: Xeon 6, Ascend 910C, Arc Pro B70 Summary: Qwen3 MoE model with 30.5B total and 3.3B active parameters, validated for BF16 serving on Intel Xeon 6 and Intel Arc Pro B70 and W8A8 serving on Ascend A3. ## Variants - default: precision=bf16, vram_minimum_gb=74 Full precision BF16 - ascend_w8a8: precision=int8, vram_minimum_gb=37, model_id=Eco-Tech/Qwen3-30B-A3B-w8a8 ModelSlim INT8 W8A8 checkpoint for Ascend 910C, verified with TP4 + expert parallelism. ## Guide ## Overview Qwen3-30B-A3B is a mixture-of-experts Qwen3 model with 30.5B total parameters and 3.3B active parameters per token. This recipe includes an Intel Xeon 6 CPU configuration validated with BF16 serving and an Ascend A3 W8A8 variant. ## Prerequisites - Hardware: 4x Xeon 6 CPUs, or 4x Intel Arc Pro B70 (32 GB per card). - vLLM >= 0.8.5 ### pip (Intel Xeon 6 CPUs) For Intel and AMD x86 CPUs, follow the [CPU pre-built wheels](https://docs.vllm.ai/en/latest/getting_started/installation/cpu/#pre-built-wheels) installation instructions. ### Docker (Intel Xeon 6 CPUs) ```bash docker pull vllm/vllm-openai-cpu:latest-x86_64 ``` ### Docker (Intel Arc Pro B70) ```bash docker pull vllm/vllm-openai-xpu:latest ``` ## Intel Xeon 6 Choose the tensor parallel size based on the system topology and deployment requirements. Add `--tensor-parallel-size ` when needed; the recipe does not prescribe a Xeon 6 TP value or hard-code NUMA node IDs. ```bash vllm serve Qwen/Qwen3-30B-A3B \ --max-num-batched-tokens 16384 \ --gpu-memory-utilization 0.8 ``` Docker: ```bash docker run \ --privileged --ipc=host -p 8000:8000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ vllm/vllm-openai-cpu:latest-x86_64 Qwen/Qwen3-30B-A3B \ --tensor-parallel-size 1 \ --max-num-batched-tokens 16384 \ --gpu-memory-utilization 0.8 \ --enable-auto-tool-choice \ --tool-call-parser hermes \ --reasoning-parser qwen3 ``` ## Intel Arc Pro B70 (XPU) Validated on 4x Intel Arc Pro B70 (32 GB per card) with the official vLLM XPU image `vllm/vllm-openai-xpu:latest`, BF16, TP=4. ```bash docker run --device /dev/dri \ -v /dev/dri/by-path:/dev/dri/by-path --shm-size=16g \ --privileged --ipc=host -p 8000:8000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ vllm/vllm-openai-xpu:latest \ Qwen/Qwen3-30B-A3B \ --tensor-parallel-size 4 \ --block-size 64 \ --enforce-eager \ --no-enable-prefix-caching \ --disable-sliding-window \ --max-model-len 9472 \ --max-num-batched-tokens 8192 \ --gpu-memory-utilization 0.9 ``` ## Client Usage ```python from openai import OpenAI client = OpenAI(base_url="http://localhost:8000/v1", api_key="unused") response = client.chat.completions.create( model="Qwen/Qwen3-30B-A3B", messages=[{"role": "user", "content": "Explain tensor parallelism briefly."}], ) print(response.choices[0].message.content) ``` ## References - [Model card](https://huggingface.co/Qwen/Qwen3-30B-A3B) - [Ascend A3 W8A8 checkpoint](https://www.modelscope.cn/models/Eco-Tech/Qwen3-30B-A3B-w8a8) - [vLLM Ascend](https://github.com/vllm-project/vllm-ascend) - [vLLM CPU installation](https://docs.vllm.ai/en/latest/getting_started/installation/cpu/) --- # meta-llama/Llama-3.3-70B-Instruct on vLLM Source: https://recipes.vllm.ai/meta-llama/Llama-3.3-70B-Instruct Provider: Meta Released: 2024-11-26T16:08:47.000Z Updated: 2026-09-22 Architecture: dense Parameters: 70B Context length: 131,072 Min vLLM: 0.12.0 Tasks: text Verified hardware: H100, H200, B200, GB200 NVL4, MI300X, MI325X, MI355X, TPU v6e (Trillium), Xeon 6, Arc Pro B70 Summary: Llama 3.3 70B dense model with NVIDIA FP8/FP4 quantized variants for Hopper and Blackwell GPUs ## Variants - default: precision=bf16, vram_minimum_gb=170 Full precision BF16 - fp8: precision=fp8, vram_minimum_gb=84, model_id=nvidia/Llama-3.3-70B-Instruct-FP8 NVIDIA FP8 quantization for Hopper and Blackwell - redhat_fp8: precision=fp8, vram_minimum_gb=84, model_id=RedHatAI/Llama-3.3-70B-Instruct-FP8-dynamic RedHatAI FP8 (E4M3) with dynamic per-token activation quantization - nvfp4: precision=nvfp4, vram_minimum_gb=42, model_id=nvidia/Llama-3.3-70B-Instruct-NVFP4 NVIDIA NVFP4 quantized weights for Blackwell GPUs - redhat_fp8_block: precision=fp8, vram_minimum_gb=84, model_id=RedHatAI/Llama-3.3-70B-Instruct-FP8-block RedHatAI FP8 block quantization - redhat_nvfp4: precision=nvfp4, vram_minimum_gb=46, model_id=RedHatAI/Llama-3.3-70B-Instruct-NVFP4 RedHatAI NVFP4 quantization ## Guide ## Overview Llama 3.3 70B Instruct is Meta's 70-billion parameter dense language model. NVIDIA provides FP8 and FP4 quantized variants optimized for Hopper (H100/H200) and Blackwell (B200/GB200) GPUs. FP4 is Blackwell-only and provides the best VRAM efficiency. TPU support is provided through [vLLM TPU](https://github.com/vllm-project/tpu-inference) with a recipe for [Trillium](https://github.com/AI-Hypercomputer/tpu-recipes/tree/main/inference/trillium/vLLM/Llama3.x). ## Prerequisites - Hardware: 1x H100/H200 (FP8), 1x B200 (FP4), 1x MI300X/MI325X/MI355X, 2x GPUs, 8x Xeon6/Xeon5 NUMA nodes, or 4x Intel Arc Pro B70 (online FP8) - vLLM >= 0.12.0 - CUDA Driver >= 575 for GPUs - Docker with NVIDIA Container Toolkit (recommended) for GPUs ### AMD ROCm The ROCm wheel requires Python 3.12, ROCm 7.0+, and glibc >= 2.35. Use the ROCm Docker image when your host environment does not meet those requirements. ```bash uv venv --python 3.12 source .venv/bin/activate uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm ``` Launch on 1x MI300X / MI325X / MI355X with AITER enabled: ```bash export VLLM_ROCM_USE_AITER=1 vllm serve meta-llama/Llama-3.3-70B-Instruct \ --tensor-parallel-size 1 ``` The first AITER launch may take several minutes while ROCm kernels are compiled and cached. ### pip (Intel Xeon 6 CPUs) For Intel and AMD x86 CPUs, follow the [CPU pre-built wheels](https://docs.vllm.ai/en/latest/getting_started/installation/cpu/#pre-built-wheels) installation instructions. ### Docker (Intel Xeon 6 CPUs) ```bash docker pull vllm/vllm-openai-cpu:latest-x86_64 # For Intel Xeon 6 ``` ### Docker (Cloud TPU — Trillium) TPU uses the separate `vllm/vllm-tpu` image (no pip wheel). Pull the tag specified by the upstream [Trillium recipe](https://github.com/AI-Hypercomputer/tpu-recipes/tree/main/inference/trillium/vLLM/Llama3.x), then run: ```bash docker run -itd --name llama33-tpu \ --privileged --network host --shm-size 16G \ -v /dev/shm:/dev/shm -e HF_TOKEN=$HF_TOKEN \ vllm/vllm-tpu:latest \ --model meta-llama/Llama-3.3-70B-Instruct \ --tensor-parallel-size 8 \ --max-model-len 16384 \ --host 0.0.0.0 --port 8000 ``` Trillium requires a 4-chip slice minimum. ### Intel Xeon 6 Deployment via Docker Launch the x86 CPU vLLM Docker container for `meta-llama/Llama-3.3-70B-Instruct`: ```bash docker run \ --privileged --ipc=host -p 8000:8000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ vllm/vllm-openai-cpu:latest-x86_64 meta-llama/Llama-3.3-70B-Instruct \ --tensor-parallel-size 8 ``` TP/DP remain user/deployment choices rather than recipe defaults. The following validation settings are not model requirements and are intentionally not portable CPU recipe defaults: - `--max-num-batched-tokens` and `--max-num-seqs`: scheduler batch/concurrency tuning for workload shape, latency targets, and platform capacity. - `--gpu-memory-utilization`: platform memory-budget tuning. - `--no-enable-prefix-caching`: workload/benchmark cache-policy tuning. - `VLLM_ENGINE_ITERATION_TIMEOUT_S`: runtime operational timeout tuning. Hardware-specific overrides may still use these settings when they are part of a validated platform configuration. ## Intel Arc Pro B70 (XPU) Validated on 4x Intel Arc Pro B70 (32 GB per card) with the official vLLM XPU image `vllm/vllm-openai-xpu:latest`, TP=4. BF16 weights (~140 GB) exceed the 128 GB node, so the validated configuration quantizes the base checkpoint online to FP8 (`--quantization fp8`) with an FP8 KV cache. ```bash docker run --device /dev/dri \ -v /dev/dri/by-path:/dev/dri/by-path --shm-size=16g \ --privileged --ipc=host -p 8000:8000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ -e HF_TOKEN=$HF_TOKEN \ -e PYTHONUNBUFFERED=1 \ -e TORCH_LLM_ALLREDUCE=1 \ -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 \ -e VLLM_WORKER_MULTIPROC_METHOD=spawn \ vllm/vllm-openai-xpu:latest \ meta-llama/Llama-3.3-70B-Instruct \ --tensor-parallel-size 4 \ --block-size 64 \ --enforce-eager \ --no-enable-prefix-caching \ --disable-sliding-window \ --max-model-len 9472 \ --max-num-batched-tokens 8192 \ --gpu-memory-utilization 0.9 \ --quantization fp8 \ --kv-cache-dtype fp8 ``` ## Client Usage ```python from openai import OpenAI client = OpenAI(base_url="http://localhost:8000/v1", api_key="unused") response = client.chat.completions.create( model="nvidia/Llama-3.3-70B-Instruct-FP8", messages=[{"role": "user", "content": "Hello, how are you?"}], ) print(response.choices[0].message.content) ``` ## Troubleshooting **FP4 variant not loading:** FP4 is only supported on Blackwell (compute capability 10.0). Use FP8 on Hopper. **OOM with BF16 on single GPU:** Use the FP8 variant (~70 GB) or FP4 variant (~40 GB) to fit on a single GPU. ## References - [Llama-3.3-70B-Instruct](https://huggingface.co/meta-llama/Llama-3.3-70B-Instruct) - [NVIDIA FP8 variant](https://huggingface.co/nvidia/Llama-3.3-70B-Instruct-FP8) - [NVIDIA FP4 variant](https://huggingface.co/nvidia/Llama-3.3-70B-Instruct-FP4) - [TPU recipes: Trillium](https://github.com/AI-Hypercomputer/tpu-recipes/tree/main/inference/trillium/vLLM/Llama3.x) --- # openai/gpt-oss-120b on vLLM Source: https://recipes.vllm.ai/openai/gpt-oss-120b Provider: OpenAI Released: 2025-08-04T22:33:06.000Z Updated: 2026-09-22 Architecture: moe Parameters: 120B Context length: 131,072 Min vLLM: 0.10.1 Tasks: text Verified hardware: H100, H200, B200, MI300X, MI325X, MI355X, TPU v7 (Ironwood), Arc Pro B70 Summary: OpenAI's gpt-oss family (20B / 120B) with MXFP4 MoE, attention-sinks, built-in tools via Responses API ## Variants - default: precision=mxfp4, vram_minimum_gb=96 gpt-oss-120b with MXFP4 MoE; fits on 1xA100 80GB, scales to TP 2/4/8 - amd_fp8: precision=mxfp4, vram_minimum_gb=80, model_id=amd/gpt-oss-120b-w-mxfp4-a-fp8 Quark-quantized MXFP4 weights with FP8 activations for MI355X (gfx950) ## Guide ## Overview `gpt-oss-20b` and `gpt-oss-120b` are open-weight reasoning models from OpenAI. vLLM supports NVIDIA H100/H200/B200, AMD MI300X/MI325X/MI355X, and Radeon AI PRO R9700, with ongoing work for Ampere/Ada/RTX 5090. Optimizations: - Flexible parallelism (TP 2/4/8) - Attention kernels for attention-sinks and sliding-window shapes - Asynchronous scheduling for CPU/GPU overlap ## Prerequisites - Hardware: NVIDIA H100/H200/B200 (or A100 80GB for single-GPU), AMD MI300+ - vLLM >= 0.10.1 - CUDA >= 12.8 if building from source (must match between install and serving) ### Install vLLM ```bash uv venv source .venv/bin/activate uv pip install vllm --torch-backend=auto ``` Docker quickstart: ```bash docker run --gpus all -p 8000:8000 --ipc=host vllm/vllm-openai --model openai/gpt-oss-20b ``` AMD ROCm wheels: ```bash uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm ``` Intel XPU (Arc Pro B70): ```bash docker pull vllm/vllm-openai-xpu:latest ``` ## Launch commands A100 (single card, default TRITON_ATTN + Marlin MXFP4 MoE): ```bash vllm serve openai/gpt-oss-120b vllm serve openai/gpt-oss-120b --tensor-parallel-size 4 ``` Blackwell (B200) with FlashInfer MXFP4+MXFP8 MoE: ```bash vllm serve openai/gpt-oss-120b --config GPT-OSS_Blackwell.yaml --tensor-parallel-size 1 ``` On vLLM v0.28.0+ the FlashInfer TRTLLM MXFP4 MoE path is selected automatically on Blackwell; it can be pinned explicitly with `--moe-backend flashinfer_trtllm --quantization-config.moe.activation mxfp8`. Hopper (H100/H200): same as Blackwell without `kv-cache-dtype` and without the FlashInfer MoE flags. TPU v7 (Ironwood), verified on a `tpu7x-standard-4t` host (topology `2x2x1`, TP=2 × DP=4). Served via the OpenAI API-server entrypoint from the nightly TPU image (e.g. `vllm/vllm-tpu:nightly-20260316-e778980-57a314d`): ```bash export TPU_BACKEND_TYPE=jax export MODEL_IMPL_TYPE=vllm export HF_HOME=/data export HUGGING_FACE_HUB_TOKEN= vllm serve openai/gpt-oss-120b \ --host 0.0.0.0 \ --port 8000 \ --seed 42 \ --tensor-parallel-size 2 \ --data-parallel-size 4 \ --max-model-len 9216 \ --max-num-batched-tokens 16384 \ --max-num-seqs 2048 \ --gpu-memory-utilization 0.86 \ --kv-cache-dtype fp8 \ --load-format runai_streamer \ --no-enable-prefix-caching \ --async-scheduling ``` AMD MI300X/MI325X: ```bash export HSA_NO_SCRATCH_RECLAIM=1 export AMDGCN_USE_BUFFER_OPS=0 export VLLM_ROCM_USE_AITER=1 export VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4 vllm serve openai/gpt-oss-120b \ --tensor-parallel-size 8 \ --attention-backend ROCM_AITER_UNIFIED_ATTN \ -cc.pass_config.fuse_rope_kvcache=True \ -cc.use_inductor_graph_partition=True \ --gpu-memory-utilization 0.95 \ --block-size 64 ``` Intel Arc Pro B70 (4x 32 GB, TP=4), MXFP4 weights via the official XPU image: ```bash docker run --device /dev/dri \ -v /dev/dri/by-path:/dev/dri/by-path --shm-size=16g \ --privileged --ipc=host -p 8000:8000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ -e TORCH_LLM_ALLREDUCE=1 \ -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 \ -e VLLM_WORKER_MULTIPROC_METHOD=spawn \ vllm/vllm-openai-xpu:latest \ openai/gpt-oss-120b \ --tensor-parallel-size 4 \ --block-size 64 \ --enforce-eager \ --disable-sliding-window \ --no-enable-prefix-caching \ --max-model-len 9216 \ --max-num-batched-tokens 9216 \ --gpu-memory-utilization 0.9 ``` ## Tool Use `/v1/responses` endpoint supports built-in tools (browsing, python, MCP). Setup requires `uv pip install gpt-oss` and either docker (for Python sandbox) or `PYTHON_EXECUTION_BACKEND=dangerously_use_uv`. For demo tools: ```bash vllm serve ... --tool-server demo ``` For user-defined function calling: ```bash vllm serve ... --tool-call-parser openai --enable-auto-tool-choice ``` ## Client Usage ```python from openai import OpenAI client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1") response = client.chat.completions.create( model="openai/gpt-oss-120b", messages=[{"role": "user", "content": "Explain sinks attention."}], ) print(response.choices[0].message.content) ``` ## Accuracy Evaluation OpenAI recommends evaluating with the gpt-oss reference library: ```bash vllm serve openai/gpt-oss-120b \ --tensor_parallel_size 8 --max-model-len 131072 \ --max-num-batched-tokens 10240 --max-num-seqs 128 \ --gpu-memory-utilization 0.85 --no-enable-prefix-caching mkdir -p /tmp/gpqa_openai OPENAI_API_KEY=empty python -m gpt_oss.evals \ --model openai/gpt-oss-120b --eval gpqa --n-threads 128 ``` Reproduced scores (120B): Low 65.3 / 51.2; Mid 72.4 / 79.6; High 79.4 / 93.0 (GPQA / AIME25). ## Troubleshooting - **Attention sinks dtype error on Blackwell:** pass `--quantization-config.moe.activation mxfp8` and `--kv-cache-dtype fp8`. - **`tl.language not defined`:** make sure no extra Triton (e.g., pytorch-triton) is installed. - **H100 TP1 OOM:** `--gpu-memory-utilization 0.95 --max-num-batched-tokens 1024`. - **Harmony vocab download failure:** pre-download tiktoken files and set `TIKTOKEN_ENCODINGS_BASE`. ## Known Limitations - Responses API: streaming is basic, annotations/citations unsupported, usage accounting returns zeros. - Function calling currently supports only `tool_choice="auto"`. ## References - [gpt-oss-120b](https://huggingface.co/openai/gpt-oss-120b) - [gpt-oss-20b](https://huggingface.co/openai/gpt-oss-20b) - [amd/gpt-oss-120b-w-mxfp4-a-fp8](https://huggingface.co/amd/gpt-oss-120b-w-mxfp4-a-fp8) - [Eagle3 draft model](https://huggingface.co/nvidia/gpt-oss-120b-Eagle3-v3) - [gpt-oss-120b on Ironwood (TPU v7)](https://github.com/AI-Hypercomputer/tpu-recipes/tree/main/inference/ironwood/vLLM/GPT-OSS) --- # bharatgenai/Param2-17B-A2.4B-Thinking on vLLM Source: https://recipes.vllm.ai/bharatgenai/Param2-17B-A2.4B-Thinking Provider: BharatGen Released: 2026-02-16T19:11:58.000Z Updated: 2026-09-20 Architecture: moe Parameters: 17B Context length: 32,768 Min vLLM: 0.20.0 Tasks: text Verified hardware: H100, L40S Summary: BharatGen's hybrid-MoE reasoning model for English, Hindi and 21 Indian languages, with thinking traces and Hermes-style tool calling ## Variants - default: precision=bf16, vram_minimum_gb=41 Official BF16 checkpoint; approximately 34.3 GB of repository weights, served with TP1 ## Recommended base flags ```bash vllm serve 'bharatgenai/Param2-17B-A2.4B-Thinking' \ --trust-remote-code \ --async-scheduling ``` ## Guide ## Overview `bharatgenai/Param2-17B-A2.4B-Thinking` is BharatGen's early post-training checkpoint of Param-2-17B, a hybrid Mixture-of-Experts model with **17B total parameters** and **2.4B activated per token**. Each sparse block routes to 6 of 64 experts and additionally keeps **2 shared experts** always active; the first layer is dense (`first_k_dense_replace: 1`) before the MoE stack begins. The shared-expert design is aimed at stable cross-lingual representations for Indian languages. Beyond English and Hindi it covers 21 scheduled Indian languages — Assamese, Bengali, Bodo, Dogri, Gujarati, Kannada, Konkani, Kashmiri, Maithili, Malayalam, Manipuri, Marathi, Nepali, Oriya, Punjabi, Sanskrit, Santali, Sindhi, Tamil, Telugu and Urdu — and was pretrained on roughly 22.6T tokens. The checkpoint is a **thinking** model: its chat template always opens an assistant turn with a `` block, and tool calls are emitted as Hermes-style `{"name": ..., "arguments": ...}` JSON. That is why the two Features above map onto the `qwen3` reasoning parser and the `hermes` tool-call parser. ## Prerequisites - **vLLM:** 0.20.0 or newer. `Param2MoEForCausalLM` was registered upstream in [PR #38000](https://github.com/vllm-project/vllm/pull/38000) and the tensor-parallel attention-head fix landed in [PR #39707](https://github.com/vllm-project/vllm/pull/39707); both ship in v0.20.0. Nothing extra needs to be installed. - **`--trust-remote-code` is required.** vLLM implements the model natively (`vllm/model_executor/models/param2moe.py`), but `model_type: param2moe` is not registered with Transformers, so the config still loads from the repo's `configuration_param2moe.py`. - **Hardware:** one GPU is enough. The BF16 weights are ~34.3 GB, so an H100 80GB leaves ample room for the 32,768-token KV cache. A single 48 GB **L40S** also runs the same configuration: at the default `--gpu-memory-utilization 0.9` roughly 8–9 GB remains for KV, and this model's KV cache is small (21 layers × 8 KV heads × 64 head dim ≈ 42 KB per token), so about 6 concurrent sequences fit at the full 32,768-token window — more if you cap `--max-model-len`. - **License:** the checkpoint is released under the BharatGen **Non-Commercial** license — review it before deploying. ## Launching the Server This is the exact configuration verified on both H100 and L40S: ```bash vllm serve bharatgenai/Param2-17B-A2.4B-Thinking \ --trust-remote-code \ --async-scheduling \ --tensor-parallel-size 1 \ --enable-auto-tool-choice \ --tool-call-parser hermes \ --reasoning-parser qwen3 ``` To pin the server to one specific GPU on a multi-GPU node, prefix the command with `CUDA_VISIBLE_DEVICES=`; add `--port ` if 8000 is taken. `--async-scheduling` overlaps scheduling with the model forward pass and cuts host overhead between decode steps. It is not required — drop it if you hit an incompatibility with another feature you enable. TP1 is the recommended layout: the model fits on one GPU and a 2.4B active-parameter forward pass gains little from sharding. To raise aggregate throughput, run several independent TP1 replicas behind a load balancer rather than increasing TP — especially on L40S, which has no NVLink, so cross-GPU TP would fall back to PCIe. ## Client Usage Reasoning traces arrive in `message.reasoning` once `--reasoning-parser qwen3` is active: ```python from openai import OpenAI client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY") response = client.chat.completions.create( model="bharatgenai/Param2-17B-A2.4B-Thinking", messages=[ {"role": "user", "content": "एक कक्षा में 30 छात्र हैं। पूरी कक्षा का औसत 70 है। शीर्ष 10 का औसत 85 और निचले 10 का औसत 50 है। मध्य 10 छात्रों का औसत क्या है?"} ], temperature=0, ) msg = response.choices[0].message print(msg.reasoning) print(msg.content) ``` The model card recommends **deterministic decoding** (`temperature=0`) for evaluations, structured tasks and production use; sampling defaults of `temperature=0.7`, `top_p=0.9`, `top_k=50` are suggested for open-ended generation. The card also notes the model is strong at structured output (Markdown, tables, JSON), and that clear task/format/constraint instructions in the prompt matter noticeably. ### Skipping the thinking block The chat template accepts `enable_thinking`. Setting it to `false` prefills an empty `` block so the model answers directly: ```python response = client.chat.completions.create( model="bharatgenai/Param2-17B-A2.4B-Thinking", messages=[{"role": "user", "content": "Name the capital of Maharashtra."}], extra_body={"chat_template_kwargs": {"enable_thinking": False}}, ) ``` ### Tool calling With the Tool Calling feature on, pass `tools` as usual and read `message.tool_calls`: ```python response = client.chat.completions.create( model="bharatgenai/Param2-17B-A2.4B-Thinking", messages=[{"role": "user", "content": "What is the weather in Pune?"}], tools=[{ "type": "function", "function": { "name": "get_weather", "description": "Get the current weather for a city", "parameters": { "type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"], }, }, }], tool_choice="auto", ) print(response.choices[0].message.tool_calls) ``` ## Benchmarking ```bash vllm bench serve \ --model bharatgenai/Param2-17B-A2.4B-Thinking \ --dataset-name random \ --random-input-len 2048 \ --random-output-len 1024 \ --num-prompts 200 ``` Because thinking traces count toward the output budget, keep `--random-output-len` generous when benchmarking with reasoning enabled, or benchmark with `enable_thinking: false` to measure the non-reasoning path. ## Notes and Limitations - **Context:** the native window is 32,768 tokens. The model card states a 128k-context variant is planned but not yet released, so do not raise `--max-model-len` past 32,768 on this checkpoint. - **No MTP head:** `num_nextn_predict_layers: 0`, so there is no speculative-decoding draft shipped with this checkpoint. - **Early post-training checkpoint:** BharatGen describes the release as an early post-training snapshot intended for downstream use and further fine-tuning. - **Verification scope:** the `verified` H100 and L40S badges cover the BF16 checkpoint at TP1 with the launch command above. Other GPUs are untested rather than known-bad. - **L40S headroom:** at 48 GB the weights alone take ~34.3 GB, so KV cache is the binding constraint rather than weight capacity. Lower `--max-model-len` if you need more concurrency than the full context window allows. ## References - [Param2-17B-A2.4B-Thinking on Hugging Face](https://huggingface.co/bharatgenai/Param2-17B-A2.4B-Thinking) - [BharatGen on Hugging Face](https://huggingface.co/bharatgenai) - [vLLM support PR #38000 — Add support for BharatGen's Param2MoE model](https://github.com/vllm-project/vllm/pull/38000) - [vLLM PR #39707 — Fix attention-head mismatch under tensor parallelism for param2moe](https://github.com/vllm-project/vllm/pull/39707) - [vLLM supported models — `Param2MoEForCausalLM`](https://docs.vllm.ai/en/latest/models/supported_models.html) --- # Qwen/Qwen-Image-2.1 on vLLM Source: https://recipes.vllm.ai/Qwen/Qwen-Image-2.1 Provider: Qwen Released: 2026-09-14T03:47:26.000Z Updated: 2026-09-19 Architecture: dense Parameters: 16B Min vLLM: 0.29.0 Tasks: omni Verified hardware: GB300 NVL4 Summary: Unified text-to-image and image-conditioned generation. A 7.1B single-stream DiT with block-causal attention and an exact cross-step prefix KV cache, paired with a Qwen3-VL-8B text encoder and a 16x RGBA autoencoder. ## Variants - default: precision=bf16, vram_minimum_gb=40 Checkpoint precision. 33.1 GB of weights. Measured 34.0 GB peak on one GB300 at 1024x1024 with --vae-use-tiling (40 steps, CFG off); the contributors report ~40 GB on GB200. Sized by the repo formula (ceil(33.1 GB x 1.2) = 40), which the 34.0 GB measurement sits under. - fp8: precision=fp8, vram_minimum_gb=40 Online FP8 scoped to the DiT, holding the composition-sensitive `img_mlp` layers in BF16 (64 of 160 block linears quantized). 32 GB peak against 34 GB in BF16, and no faster. Quantizing the text encoder instead saves more — about 28 GB — and is one more key in the same JSON. ## Guide ## Overview vLLM-Omni accelerates Qwen-Image 2.1 through cross-step prefix KV cache reuse and dedicated CUDA Graphs, reducing redundant computation and kernel launch overhead. Request-level and step-level continuous batching improve GPU utilization and throughput, with phase-aware prefill and decode scheduling. It also supports tensor and Ulysses sequence parallelism, distributed VAE decoding with adaptive OOM recovery, FP8 weights and prefix KV storage, and CPU offloading for varying memory budgets. One pipeline class, `QwenImage21Pipeline`, serves both text-to-image and image-conditioned generation (editing). Prompt and reference images are encoded together by a Qwen3-VL vision-language model, so an edit request takes the same serving path as plain text-to-image, and there is no separate edit model to load. | Component | Class | On disk | |---|---|---| | Text encoder | `Qwen3VLForConditionalGeneration` | bf16, 17.5 GB | | DiT | `QwenImage21Transformer2DModel` | bf16, 14.2 GB | | Autoencoder | `AutoencoderKLQwenImage21` | fp32, 1.4 GB | The autoencoder compresses 16x into a 64-channel latent and carries 4 channels in and out, so RGBA transparency survives the round trip. ## Sampling defaults Sample at **40 steps with classifier-free guidance off** — `num_inference_steps=40` and `true_cfg_scale=1.0` in the reference implementation's pipeline signature ([diffusers#14804](https://github.com/huggingface/diffusers/pull/14804)). ```bash --num-inference-steps 40 --cfg-scale 1.0 ``` `--cfg-scale` above 1 engages only with a negative prompt; otherwise it is ignored with a warning. When it engages, the DiT runs twice per step, roughly doubling latency. ## Prerequisites Support is not in a tagged release and, as of this writing, not yet merged (vllm-project/vllm-omni#7759). The pipeline itself is known-good: brought up from PR #7759 on one NVIDIA GB300, it generated a 1024x1024 PNG in **4.5 s** over 40 steps at **34.0 GB** peak. ```bash git clone https://github.com/vllm-project/vllm-omni.git && cd vllm-omni git fetch origin pull/7759/head:qwen-image-2.1 && git checkout qwen-image-2.1 uv venv --python 3.12 --seed && source .venv/bin/activate uv pip install vllm==0.29.0 --torch-backend=auto && uv pip install -e . ``` ## Offline inference ```bash # text-to-image python examples/offline_inference/text_to_image/text_to_image.py \ --model Qwen/Qwen-Image-2.1 \ --prompt "A ceramic teapot on a wooden table" \ --output t2i.png --num-inference-steps 40 --cfg-scale 1.0 # image-conditioned python examples/offline_inference/image_to_image/image_edit.py \ --model Qwen/Qwen-Image-2.1 --color-format RGBA --seed 42 \ --image input1.png input2.png \ --prompt "Combine these images into a single scene" \ --output edit.png --num-inference-steps 40 --cfg-scale 1.0 ``` Up to 4 reference images per request; a fifth is rejected with a 400. Each is resized independently to ~1024x1024 of area, so they may differ in aspect ratio, and each is tagged ``, `` in the order given, so an ordinal reference in the prompt binds a phrase to a photo. `--color-format RGBA` preserves transparency (the shared example defaults to RGB). Width and height are floored to a multiple of 32, never rejected. When omitted on an edit request the output size derives from the last reference image's aspect ratio at ~1024x1024. ## Online serving ```bash vllm serve Qwen/Qwen-Image-2.1 --omni --port 8091 ``` Add `--step-execution --max-num-seqs 8` to batch requests at the same KV-cache phase. Admission is all-or-nothing: once a wave is past its first denoising step, a new arrival waits for it to finish, even if `max-num-seqs` has room. Send `num_inference_steps` on every request; the server's own default is 50, not the 40 this checkpoint wants. `true_cfg_scale` defaults to 4.0 but only applies alongside a `negative_prompt` — send one without the other and you get guidance at 4.0 and roughly double the compute. ### Text to image Send JSON to `/v1/images/generations`. The picture comes back base64-encoded at `.data[0].b64_json`. ```bash curl -X POST http://localhost:8091/v1/images/generations \ -H "Content-Type: application/json" \ -d '{"model": "Qwen/Qwen-Image-2.1", "prompt": "A ceramic teapot on a wooden table", "size": "1024x1024", "num_inference_steps": 40, "true_cfg_scale": 1.0, "seed": 42}' ``` ### Image editing Send a **form**, not JSON, to `/v1/images/edits`. The picture is a file upload and everything else is a form field. A JSON body to this endpoint resets the connection. Repeat `-F image=@...` for more pictures, up to four. ```bash curl -X POST http://localhost:8091/v1/images/edits \ -F image=@plate.png \ -F 'prompt=Write the words "FRESH BASIL" on this plate in dark green lettering' \ -F model=Qwen/Qwen-Image-2.1 \ -F size=1024x1024 \ -F num_inference_steps=40 \ -F true_cfg_scale=1.0 \ -F seed=42 ``` The reply has the same shape as text to image: the picture is at `.data[0].b64_json`. If you would rather send JSON, the chat endpoint also accepts pictures. It puts the result somewhere different — `.choices[0].message.content[0].image_url.url`: ```bash curl -X POST http://localhost:8091/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{"model": "Qwen/Qwen-Image-2.1", "messages": [{"role": "user", "content": [ {"type": "image_url", "image_url": {"url": "data:image/png;base64,"}}, {"type": "text", "text": "Write the words \"FRESH BASIL\" on this plate"} ]}], "modalities": ["image"], "extra_body": {"size": "1024x1024", "num_inference_steps": 40, "true_cfg_scale": 1.0, "seed": 42}}' ``` ## Prefix KV cache On automatically; rebuilt per generation, so nothing leaks between requests. The prompt and any reference images are cached after the first denoising step and reused for the rest, which is exact rather than approximate. It pays off in proportion to how much prefix there is: repeating a prompt costs about a third of the first submission, four reference images leave only about a fifth of the sequence to recompute, and plain short-prompt text-to-image has almost nothing to cache. Changing the tail of a prompt loses the saving. For long-prompt or multi-image workloads it can be stored in FP8 via the stage `extras` key `prefix_kv_cache_dtype` — on the command line through `--stage-overrides '{"0":{"extras":{"prefix_kv_cache_dtype":"fp8_v"}}}'`, or as a stage field in a deploy YAML. `"fp8_v"` quantizes V only, `"fp8"` quantizes K too — the contributors measured ~41 dB and ~35 dB PSNR against BF16. A quantized cache returns less of the speed-up than one in the native dtype, and is not CUDA-graph capturable, so those requests fall back to eager decode. ## CUDA graphs Fixed-shape decode steps are captured automatically (`enforce_eager=False` by default); prefill stays eager. Each image layout gets its own entry and prefix K/V is copied into static buffers before replay, which costs memory. TP, SP/ring, HSDP, offload and cache hooks, compiled blocks, quantized KV caches, dynamic LoRA and padded text masks each force eager decode. Disable with `--enforce-eager`; the AR engine's `compilation_config.cudagraph_mode` does not reach this path. Expect the first request after startup to cost about 1.3x a later one. `--enforce-eager` does not remove that cost, so it is not graph capture and disabling graphs will not flatten startup. ## Quantization Online FP8, per component: ```python from vllm_omni import Omni omni = Omni(model="Qwen/Qwen-Image-2.1", quantization_config={ "transformer": {"method": "fp8", "ignored_layers": ["img_mlp"]}, "text_encoder": {"method": "fp8"}, }) ``` Only the DiT's block-internal linears are eligible; boundary projections stay BF16. Measured by the vLLM-Omni contributors on GB200 at 1024x1024, 50 steps: | Config | Quantized | Avg PSNR vs BF16 | Peak memory | |---|---|---|---| | BF16 | 0 / 160 | — | 40.0 GB | | FP8, all layers | 160 / 160 | 26.1 dB | 33.4 GB | | FP8, `ignored_layers=["img_mlp"]` | 64 / 160 | 29.7 dB | 38.3 GB | Measured here on one GB300 at 1024x1024, 40 steps, same prompt and seed in each case: | Config | Peak memory | Time per image | |---|---|---| | BF16 | 34.0 GB | 3282-4492 ms | | DiT FP8, `img_mlp` kept BF16 | 32.0-32.7 GB | 3356-4706 ms | | Text-encoder FP8 | **27.5-28.1 GB** | 3427-4770 ms | The text encoder is the bigger saving by a wide margin: about 6 GB against about 2 GB. Neither runs faster. Keeping `img_mlp` in BF16 protects composition but gives back most of the saving. The larger win is the text encoder: in a separate GB200 run, peak memory went ~41.0 GiB in BF16 to ~34.4 GiB with text-encoder FP8, and ~28.1 GiB with the DiT too. Its vision tower and unused `lm_head` are excluded by construction, but reference-image tokens still cross the quantized language-model linears, so text-encoder FP8 does reach the edit path. On Blackwell FP8 buys memory, not speed. ## Limitations - Cache backends (`cache_dit`, `tea_cache`) unsupported — the model's own prefix cache conflicts with step-skipping hooks. - Sequence parallelism is Ulysses only — no ring attention, no pipeline parallelism. - True CFG only; there is no `guidance_scale`. - `/v1/images/edits` takes **multipart/form-data**, not JSON. Send the picture as a file upload and the settings as form fields. Posting a JSON body to it resets the connection, which is easy to mistake for a broken endpoint. - Passing a picture as a `data:` URL in the `url` form field is limited to **1024 KB per part**. A 512x512 PNG from this model is about 1 MB, so it usually fails with `Part exceeded maximum size of 1024KB`. File uploads have no such limit. Base64 inside a form field only suits small images. - `/v1/chat/completions` with `image_url` content parts works too, and returns the picture at `.choices[0].message.content[0].image_url.url`. - NVIDIA GPUs only. - VAE tiling (`--vae-use-tiling`) decodes in 512px tiles at 384px stride, halving on CUDA OOM down to 128px; tile sizes are not exposed as engine arguments. ## References - [In-repo recipe](https://github.com/vllm-project/vllm-omni/blob/qwen-image-2.1/recipes/Qwen/Qwen-Image-2.1.md) - [Feature support matrix](https://github.com/vllm-project/vllm-omni/blob/qwen-image-2.1/docs/user_guide/diffusion_features.md) --- # zai-org/GLM-4.7 on vLLM Source: https://recipes.vllm.ai/zai-org/GLM-4.7 Provider: GLM (Z-AI) Released: 2025-12-22T07:45:52.000Z Updated: 2026-09-18 Architecture: moe Parameters: 358B Context length: 202,752 Min vLLM: 0.24.0 Tasks: text Verified hardware: H200, MI300X, MI325X, MI355X Summary: GLM-4.7 MoE language model (~358B total parameters) with MTP speculative decoding, updated tool call parser, and reasoning support ## Variants - default: precision=bf16, vram_minimum_gb=859 Full precision BF16 on 8xH200 or equivalent - fp8: precision=fp8, vram_minimum_gb=430, model_id=zai-org/GLM-4.7-FP8 Native FP8 checkpoint with minimal accuracy loss - nvfp4: precision=nvfp4, vram_minimum_gb=215, model_id=nvidia/GLM-4.7-NVFP4 NVIDIA NVFP4 quantized weights for Blackwell GPUs ## Recommended base flags ```bash vllm serve 'zai-org/GLM-4.7' \ --trust-remote-code ``` ## Guide ## Overview GLM-4.7 is the latest GLM-4.X MoE release from Z-AI. It introduces the `glm47` tool call parser and uses the matching `glm47` reasoning parser. Built-in Multi-Token Prediction (MTP) layers enable speculative decoding for throughput gains on decode-heavy workloads. A smaller `zai-org/GLM-4.7-Flash` variant is also available for lower-latency scenarios. ## Prerequisites - **vLLM version:** nightly recommended for GLM-4.7 (until packaged in a stable release) - **Hardware:** 4x-8x H200 (FP8), AMD MI300X / MI325X / MI355X for ROCm - **Python:** 3.10 - 3.13 (3.12 required for ROCm wheels) ### Install vLLM (NVIDIA, nightly) ```bash uv venv source .venv/bin/activate uv pip install -U vllm --pre --extra-index-url https://wheels.vllm.ai/nightly uv pip install git+https://github.com/huggingface/transformers.git ``` ### Install vLLM (AMD ROCm) ```bash uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm ``` ## Launching the Server ### Tensor Parallel + MTP (FP8 on 4xH200) ```bash vllm serve zai-org/GLM-4.7-FP8 \ --tensor-parallel-size 4 \ --speculative-config.method mtp \ --speculative-config.num_speculative_tokens 1 \ --tool-call-parser glm47 \ --reasoning-parser glm47 \ --enable-auto-tool-choice ``` ### AMD ROCm ```bash SAFETENSORS_FAST_GPU=1 \ vllm serve zai-org/GLM-4.7 \ --tensor-parallel-size 8 \ --gpu-memory-utilization 0.9 \ --no-enable-prefix-caching \ --trust-remote-code ``` ### Tuning Tips - `--max-model-len=65536` is a sensible default; max is 128K. - `--max-num-batched-tokens=32768` for prompt-heavy workloads; reduce to 8K-16K for latency-sensitive. - Use `--gpu-memory-utilization=0.95` to maximize KV cache. ## Client Usage ```python from openai import OpenAI client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1") resp = client.chat.completions.create( model="zai-org/GLM-4.7-FP8", messages=[{"role": "user", "content": "Hello!"}], max_tokens=512, ) print(resp.choices[0].message.content) ``` ## Benchmarking ```bash vllm bench serve \ --model zai-org/GLM-4.7-FP8 \ --dataset-name random \ --random-input-len 8000 \ --random-output-len 1000 \ --request-rate 10000 \ --num-prompts 16 \ --ignore-eos ``` ## Troubleshooting - **Parser mismatch:** GLM-4.7 uses `--tool-call-parser glm47` and `--reasoning-parser glm47`. - **MTP acceptance:** 1 speculative token gives ~90%+ acceptance and best throughput. ## References - [Model card](https://huggingface.co/zai-org/GLM-4.7) - [FP8 checkpoint](https://huggingface.co/zai-org/GLM-4.7-FP8) - [GLM-4.7-Flash](https://huggingface.co/zai-org/GLM-4.7-Flash) - [vLLM docs](https://docs.vllm.ai/) --- # MiniMaxAI/MiniMax-M2.5 on vLLM Source: https://recipes.vllm.ai/MiniMaxAI/MiniMax-M2.5 Provider: MiniMax Released: 2026-02-12T06:05:24.000Z Updated: 2026-09-14 Architecture: moe Parameters: 230B Context length: 196,608 Min vLLM: 0.20.2 Tasks: text Verified hardware: H100, H200, MI300X, MI325X, MI355X Summary: MiniMax M2.5 MoE language model (230B total / 10B active) for coding, agent toolchains, and long-context reasoning — native FP8 checkpoint ## Variants - default: precision=fp8, vram_minimum_gb=276 Native FP8 checkpoint — 4x H200/H20/H100 or 4x A100/A800 for weights, plus KV cache headroom - nvfp4: precision=nvfp4, vram_minimum_gb=138, model_id=nvidia/MiniMax-M2.5-NVFP4 NVIDIA NVFP4 quantized weights for Blackwell GPUs - redhat_nvfp4: precision=nvfp4, vram_minimum_gb=146, model_id=RedHatAI/MiniMax-M2.5-NVFP4 RedHatAI NVFP4 quantization ## Recommended base flags ```bash vllm serve 'MiniMaxAI/MiniMax-M2.5' \ --trust-remote-code ``` ## Guide ## Overview [MiniMax-M2.5](https://huggingface.co/MiniMaxAI/MiniMax-M2.5) is part of the MiniMax M2 series of advanced MoE language models. It retains the M2 architecture (10B active, 230B total) and a 196K context per sequence. ## Prerequisites - **OS:** Linux - **Python:** 3.10 - 3.13 - **NVIDIA:** compute capability >= 7.0; ~220 GB for weights + 240 GB per 1M context tokens - **AMD:** MI300X / MI325X / MI350X / MI355X with ROCm 7.0+ ### Install vLLM (NVIDIA) ```bash uv venv source .venv/bin/activate uv pip install -U vllm --torch-backend auto ``` ### Docker (dedicated M2-series image) ```bash docker run --gpus all \ -p 8000:8000 \ --ipc=host \ -v ~/.cache/huggingface:/root/.cache/huggingface \ -e PYTHONNOUSERSITE=1 \ -e VLLM_USE_DEEP_GEMM=0 \ -e VLLM_BLOCKSCALE_FP8_GEMM_FLASHINFER=0 \ -e VLLM_FLOAT32_MATMUL_PRECISION=high \ vllm/vllm-openai:v0.28.0 MiniMaxAI/MiniMax-M2.5 \ --tensor-parallel-size 4 \ --max-num-seqs 512 \ --max-num-batched-tokens 32768 \ --kv-cache-dtype fp8 \ --moe-backend triton \ --attention-backend FLASHINFER \ --enable-flashinfer-autotune \ --tool-call-parser minimax_m2 \ --reasoning-parser minimax_m2 \ --enable-auto-tool-choice \ --compilation-config '{"cudagraph_mode":"PIECEWISE"}' \ --trust-remote-code ``` ## Launching the Server ### NVIDIA H200 — TP4 ```bash PYTHONNOUSERSITE=1 \ VLLM_USE_DEEP_GEMM=0 \ VLLM_BLOCKSCALE_FP8_GEMM_FLASHINFER=0 \ VLLM_FLOAT32_MATMUL_PRECISION=high \ vllm serve MiniMaxAI/MiniMax-M2.5 \ --tensor-parallel-size 4 \ --max-num-seqs 512 \ --max-num-batched-tokens 32768 \ --kv-cache-dtype fp8 \ --moe-backend triton \ --attention-backend FLASHINFER \ --enable-flashinfer-autotune \ --tool-call-parser minimax_m2 \ --reasoning-parser minimax_m2 \ --compilation-config '{"cudagraph_mode":"PIECEWISE"}' \ --enable-auto-tool-choice \ --trust-remote-code ``` For H200 FP8, choose the attention backend by sequence length for best performance: shorter sequences (e.g. 1024) should keep the command above with `--attention-backend FLASHINFER` and `--enable-flashinfer-autotune`, while longer input sequences (e.g. 8192) should remove those flags to use the auto-selected FlashAttention backend. Pure TP8 is not supported. For >4 GPUs use DP+EP or TP+EP. ### TP4+EP (recommended for H100) ```bash vllm serve MiniMaxAI/MiniMax-M2.5 \ --tensor-parallel-size 4 \ --enable-expert-parallel \ --tool-call-parser minimax_m2 \ --reasoning-parser minimax_m2 \ --compilation-config '{"cudagraph_mode":"PIECEWISE"}' \ --enable-auto-tool-choice ``` ### AMD ROCm — TP2 ```bash VLLM_ROCM_USE_AITER=1 vllm serve MiniMaxAI/MiniMax-M2.5 \ --tensor-parallel-size 2 \ --tool-call-parser minimax_m2 \ --reasoning-parser minimax_m2 \ --enable-auto-tool-choice \ --trust-remote-code ``` ## Benchmarking ```bash vllm bench serve \ --backend vllm \ --model MiniMaxAI/MiniMax-M2.5 \ --endpoint /v1/completions \ --dataset-name random \ --random-input 2048 \ --random-output 1024 \ --max-concurrency 10 \ --num-prompt 100 ``` ## Troubleshooting - See [MiniMax-M2](./MiniMax-M2.yaml) for shared troubleshooting notes (latest vLLM fusion behavior, nightly vs stable, DeepGEMM, AITER). ## References - [Model card](https://huggingface.co/MiniMaxAI/MiniMax-M2.5) - [MiniMax](https://www.minimax.io/) --- # MiniMaxAI/MiniMax-M2.7 on vLLM Source: https://recipes.vllm.ai/MiniMaxAI/MiniMax-M2.7 Provider: MiniMax Released: 2026-04-09T03:37:12.000Z Updated: 2026-09-14 Architecture: moe Parameters: 230B Context length: 196,608 Min vLLM: 0.20.0 Tasks: text Verified hardware: H100, H200, MI300X, MI325X, MI355X, DGX Station (GB300), RTX Pro 6000 2x Summary: MiniMax M2.7 MoE language model (230B total / 10B active) — latest M2 release for coding, agent toolchains, and long-context reasoning with native FP8 ## Variants - default: precision=fp8, vram_minimum_gb=276 Native FP8 checkpoint — 4x H200/H20/H100 or 4x A100/A800 for weights, plus KV cache headroom - nvfp4: precision=nvfp4, vram_minimum_gb=138, model_id=nvidia/MiniMax-M2.7-NVFP4 NVIDIA NVFP4 quantized weights for Blackwell GPUs ## Recommended base flags ```bash vllm serve 'MiniMaxAI/MiniMax-M2.7' \ --trust-remote-code ``` ## Guide ## Overview [MiniMax-M2.7](https://huggingface.co/MiniMaxAI/MiniMax-M2.7) is the latest release in the MiniMax M2 series. Like earlier M2 variants, it ships with 10B active parameters out of 230B total, and supports a 196K context per sequence. MiniMax has verified M2.7 accuracy on AIME25, GPQA-D, and GSM8K at vLLM commit `0f3ce4c74b1875791d6604e006b6e905fde9f698`. ## Prerequisites - **OS:** Linux - **Python:** 3.10 - 3.13 - **NVIDIA:** compute capability >= 7.0; ~220 GB for weights + 240 GB per 1M context tokens - **AMD:** MI300X / MI325X / MI350X / MI355X with ROCm 7.0+ ### Install vLLM (NVIDIA) ```bash uv pip install -U vllm --extra-index-url https://wheels.vllm.ai/nightly ``` ### Install vLLM (AMD ROCm) ```bash uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/ ``` ### Docker (dedicated M2-series image) ```bash docker run --gpus all \ -p 8000:8000 \ --ipc=host \ -v ~/.cache/huggingface:/root/.cache/huggingface \ vllm/vllm-openai:minimax27 MiniMaxAI/MiniMax-M2.7 \ --tensor-parallel-size 4 \ --tool-call-parser minimax_m2 \ --reasoning-parser minimax_m2 \ --enable-auto-tool-choice \ --trust-remote-code ``` ## Launching the Server ### NVIDIA — TP4 (4x H200/H20/H100 or 4x A100/A800) ```bash vllm serve MiniMaxAI/MiniMax-M2.7 \ --tensor-parallel-size 4 \ --tool-call-parser minimax_m2 \ --reasoning-parser minimax_m2 \ --enable-auto-tool-choice \ --trust-remote-code ``` Pure TP8 is not supported. For >4 GPUs use DP+EP or TP+EP: ### DP8+EP ```bash vllm serve MiniMaxAI/MiniMax-M2.7 \ --data-parallel-size 8 \ --enable-expert-parallel \ --tool-call-parser minimax_m2 \ --reasoning-parser minimax_m2 \ --enable-auto-tool-choice ``` ### TP4+EP (recommended for H100) ```bash vllm serve MiniMaxAI/MiniMax-M2.7 \ --tensor-parallel-size 4 \ --enable-expert-parallel \ --tool-call-parser minimax_m2 \ --reasoning-parser minimax_m2 \ --enable-auto-tool-choice ``` ### TP8+EP ```bash vllm serve MiniMaxAI/MiniMax-M2.7 \ --tensor-parallel-size 8 \ --enable-expert-parallel \ --tool-call-parser minimax_m2 \ --reasoning-parser minimax_m2 \ --enable-auto-tool-choice ``` ### AMD ROCm — TP2 ```bash VLLM_ROCM_USE_AITER=1 vllm serve MiniMaxAI/MiniMax-M2.7 \ --tensor-parallel-size 2 \ --tool-call-parser minimax_m2 \ --reasoning-parser minimax_m2 \ --enable-auto-tool-choice \ --trust-remote-code ``` ## Benchmarking ```bash vllm bench serve \ --backend vllm \ --model MiniMaxAI/MiniMax-M2.7 \ --endpoint /v1/completions \ --dataset-name random \ --random-input 2048 \ --random-output 1024 \ --max-concurrency 10 \ --num-prompt 100 ``` ## Troubleshooting - **MiniMax QK norm fusion:** Latest vLLM applies the fusion automatically at runtime when the fused CUDA op is available; older vLLM builds used an explicit `fuse_minimax_qk_norm` compile pass. - **Corrupted output on stable release:** Upgrade to a nightly after commit `cf3eacfe58fa9e745c2854782ada884a9f992cf7`. - **Verified accuracy:** Use the pinned commit `0f3ce4c74b1875791d6604e006b6e905fde9f698` if reproducing MiniMax's reported results. - **DeepGEMM:** vLLM uses DeepGEMM by default; install via [install_deepgemm.sh](https://github.com/vllm-project/vllm/blob/main/tools/install_deepgemm.sh) if missing. - **AITER first launch:** Initial AMD launch JIT-compiles optimized kernels; subsequent launches reuse cached kernels. ## References - [Model card](https://huggingface.co/MiniMaxAI/MiniMax-M2.7) - [MiniMax](https://www.minimax.io/) - [vLLM PR #43410 (automatic MiniMax QK norm fusion)](https://github.com/vllm-project/vllm/pull/43410) --- # Qwen/Qwen3.5-0.8B on vLLM Source: https://recipes.vllm.ai/Qwen/Qwen3.5-0.8B Provider: Qwen Released: 2026-02-28T23:57:01.000Z Updated: 2026-09-14 Architecture: dense Parameters: 0.8B Context length: 262,144 Min vLLM: 0.17.0 Tasks: multimodal, text Verified hardware: Arc Pro B60, Arc Pro B70 Summary: Qwen3.5 tiny dense multimodal model (0.8B) — ultra-low-VRAM / edge serving with 262K context ## Variants - default: precision=bf16, vram_minimum_gb=2 Full precision BF16 — runs on any modern GPU ## Recommended base flags ```bash vllm serve 'Qwen/Qwen3.5-0.8B' \ --trust-remote-code ``` ## Guide ## Overview [Qwen3.5-0.8B](https://huggingface.co/Qwen/Qwen3.5-0.8B) is the smallest member of the Qwen3.5 family — same hybrid gated delta networks architecture and 262K context, at a size suited to edge devices or as a draft model for speculative decoding with larger Qwen3.5 checkpoints. ## Prerequisites - **vLLM version:** >= 0.17.0 - **Hardware:** any modern GPU (>=4 GB VRAM) or Intel Arc Pro B60/B70 ### Install vLLM ```bash uv venv source .venv/bin/activate uv pip install -U vllm --torch-backend=auto ``` ### Docker ```bash docker pull vllm/vllm-openai-xpu:latest # Intel XPU (B60 / B70) ``` ## Launching the Server ```bash vllm serve Qwen/Qwen3.5-0.8B \ --max-model-len 262144 \ --reasoning-parser qwen3 ``` ### Docker (Intel XPU B60 / B70) Validated on 1× Intel Arc Pro B60 / B70 (B60 24 GB, B70 32 GB per card) with the official vLLM XPU image `vllm/vllm-openai-xpu:latest`. ```bash docker run --device /dev/dri \ -v /dev/dri/by-path:/dev/dri/by-path --shm-size=16g \ --privileged --ipc=host -p 8000:8000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ vllm/vllm-openai-xpu:latest \ Qwen/Qwen3.5-0.8B \ --reasoning-parser qwen3 \ --enforce-eager ``` ## Client Usage ```python from openai import OpenAI client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1") resp = client.chat.completions.create( model="Qwen/Qwen3.5-0.8B", messages=[{"role": "user", "content": "Hi!"}], max_tokens=64, ) print(resp.choices[0].message.content) ``` ## References - [Model card](https://huggingface.co/Qwen/Qwen3.5-0.8B) - [Base checkpoint](https://huggingface.co/Qwen/Qwen3.5-0.8B-Base) --- # Qwen/Qwen3.5-27B on vLLM Source: https://recipes.vllm.ai/Qwen/Qwen3.5-27B Provider: Qwen Released: 2026-02-24T09:41:56.000Z Updated: 2026-09-14 Architecture: dense Parameters: 27B Context length: 262,144 Min vLLM: 0.17.0 Tasks: multimodal, text Verified hardware: Arc Pro B60, Arc Pro B70 Summary: Qwen3.5 dense multimodal model (27B) with gated delta networks hybrid attention, MTP, and 262K context ## Variants - default: precision=bf16, vram_minimum_gb=65 Full precision BF16 — fits on 1x H200, 2x H100, 4x Intel Arc Pro B60, 2x Intel Arc Pro B70 - fp8: precision=fp8, vram_minimum_gb=33, model_id=Qwen/Qwen3.5-27B-FP8 Qwen official FP8 checkpoint — single 40 GB GPU - gptq_int4: precision=int4, vram_minimum_gb=17, model_id=Qwen/Qwen3.5-27B-GPTQ-Int4 GPTQ Int4 checkpoint — fits on a single 24 GB GPU ## Recommended base flags ```bash vllm serve 'Qwen/Qwen3.5-27B' \ --trust-remote-code ``` ## Guide ## Overview [Qwen3.5-27B](https://huggingface.co/Qwen/Qwen3.5-27B) is the flagship dense model of the Qwen3.5 family. It uses the same gated delta networks hybrid attention as its MoE siblings, supports vision+text input, and natively serves 262K context. MTP (multi-token prediction) is supported out of the box for low-latency decoding. ## Prerequisites - **vLLM version:** >= 0.17.0 - **Hardware (BF16):** 1x H200, 2x H100, 4x Intel Arc Pro B60, 2x Intel Arc Pro B70 - **Hardware (FP8):** single 40 GB GPU (H100/H200/L40S) - **Hardware (Int4):** single 24 GB GPU ### Install vLLM ```bash uv venv source .venv/bin/activate uv pip install -U vllm --torch-backend=auto ``` ### Docker ```bash docker pull vllm/vllm-openai-xpu:latest # Intel XPU (B60 / B70) ``` ## Launching the Server ### Single-GPU FP8 ```bash vllm serve Qwen/Qwen3.5-27B-FP8 \ --max-model-len 262144 \ --reasoning-parser qwen3 ``` ### BF16 on 2xH100 (TP2) ```bash vllm serve Qwen/Qwen3.5-27B \ --tensor-parallel-size 2 \ --max-model-len 262144 \ --reasoning-parser qwen3 ``` ### MTP speculative decoding ```bash vllm serve Qwen/Qwen3.5-27B-FP8 \ --speculative-config '{"method": "mtp", "num_speculative_tokens": 1}' \ --reasoning-parser qwen3 ``` ### Text-only (skip vision encoder) ```bash vllm serve Qwen/Qwen3.5-27B-FP8 \ --language-model-only \ --reasoning-parser qwen3 \ --enable-prefix-caching ``` ### Docker (Intel XPU B60 / B70) Validated on 4× Intel Arc Pro B60 or 2x Intel Arc Pro B70 (B60 24 GB, B70 32 GB per card) with the official vLLM XPU image `vllm/vllm-openai-xpu:latest`. ```bash docker run --device /dev/dri \ -v /dev/dri/by-path:/dev/dri/by-path --shm-size=16g \ --privileged --ipc=host -p 8000:8000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ vllm/vllm-openai-xpu:latest \ Qwen/Qwen3.5-27B \ --reasoning-parser qwen3 \ --tensor-parallel-size 4 \ --max-model-len 8192 \ --enforce-eager ``` ## Client Usage ```python from openai import OpenAI client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1") resp = client.chat.completions.create( model="Qwen/Qwen3.5-27B", messages=[{"role": "user", "content": "Write a haiku about gated delta networks."}], max_tokens=256, ) print(resp.choices[0].message.content) ``` ## Troubleshooting - **CUDA graph / Mamba cache size error:** reduce `--max-cudagraph-capture-size` (default 512). See [vLLM PR #34571](https://github.com/vllm-project/vllm/pull/34571). - **Disable reasoning:** add `--default-chat-template-kwargs '{"enable_thinking": false}'`. - **Prefix Caching (Mamba):** currently experimental in "align" mode. ## References - [Model card](https://huggingface.co/Qwen/Qwen3.5-27B) - [FP8 checkpoint](https://huggingface.co/Qwen/Qwen3.5-27B-FP8) - [GPTQ-Int4 checkpoint](https://huggingface.co/Qwen/Qwen3.5-27B-GPTQ-Int4) - [Qwen3.5-397B-A17B recipe](../Qwen3.5-397B-A17B) --- # Qwen/Qwen3.5-2B on vLLM Source: https://recipes.vllm.ai/Qwen/Qwen3.5-2B Provider: Qwen Released: 2026-02-28T23:56:16.000Z Updated: 2026-09-14 Architecture: dense Parameters: 2B Context length: 262,144 Min vLLM: 0.17.0 Tasks: multimodal, text Verified hardware: Arc Pro B60, Arc Pro B70 Summary: Qwen3.5 mini dense multimodal model (2B) — edge / low-VRAM serving with 262K context ## Variants - default: precision=bf16, vram_minimum_gb=5 Full precision BF16 — fits on an 8 GB GPU or Intel Arc Pro B60/B70 ## Recommended base flags ```bash vllm serve 'Qwen/Qwen3.5-2B' \ --trust-remote-code ``` ## Guide ## Overview [Qwen3.5-2B](https://huggingface.co/Qwen/Qwen3.5-2B) is a miniature dense Qwen3.5 model — the full gated delta networks architecture, vision encoder, and 262K context, in a form small enough for 8 GB consumer GPUs or edge inference. ## Prerequisites - **vLLM version:** >= 0.17.0 - **Hardware:** single 8 GB GPU or Intel Arc Pro B60/B70 ### Install vLLM ```bash uv venv source .venv/bin/activate uv pip install -U vllm --torch-backend=auto ``` ### Docker ```bash docker pull vllm/vllm-openai-xpu:latest # Intel XPU (B60 / B70) ``` ## Launching the Server ```bash vllm serve Qwen/Qwen3.5-2B \ --max-model-len 262144 \ --reasoning-parser qwen3 ``` ### Docker (Intel XPU B60 / B70) Validated on 1× Intel Arc Pro B60 / B70 (B60 24 GB, B70 32 GB per card) with the official vLLM XPU image `vllm/vllm-openai-xpu:latest`. ```bash docker run --device /dev/dri \ -v /dev/dri/by-path:/dev/dri/by-path --shm-size=16g \ --privileged --ipc=host -p 8000:8000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ vllm/vllm-openai-xpu:latest \ Qwen/Qwen3.5-2B \ --reasoning-parser qwen3 \ --enforce-eager ``` ## Client Usage ```python from openai import OpenAI client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1") resp = client.chat.completions.create( model="Qwen/Qwen3.5-2B", messages=[{"role": "user", "content": "Hi!"}], max_tokens=64, ) print(resp.choices[0].message.content) ``` ## References - [Model card](https://huggingface.co/Qwen/Qwen3.5-2B) - [Base checkpoint](https://huggingface.co/Qwen/Qwen3.5-2B-Base) --- # Qwen/Qwen3.5-4B on vLLM Source: https://recipes.vllm.ai/Qwen/Qwen3.5-4B Provider: Qwen Released: 2026-02-27T14:45:03.000Z Updated: 2026-09-14 Architecture: dense Parameters: 4B Context length: 262,144 Min vLLM: 0.17.0 Tasks: multimodal, text Verified hardware: Xeon 6, Arc Pro B60, Arc Pro B70 Summary: Qwen3.5 compact dense multimodal model (4B) — fits on 16 GB consumer GPUs with full 262K context or one Xeon 6 NUMA node or Intel Arc Pro B60/B70 ## Variants - default: precision=bf16, vram_minimum_gb=10 Full precision BF16 — fits on a single 16 GB GPU or one Xeon 6 NUMA node or one Intel Arc Pro B60/B70 - fp8: precision=fp8, vram_minimum_gb=5, model_id=RedHatAI/Qwen3.5-4B-FP8-dynamic RedHatAI FP8 (E4M3) with dynamic per-token activation quantization ## Recommended base flags ```bash vllm serve 'Qwen/Qwen3.5-4B' \ --trust-remote-code ``` ## Guide ## Overview [Qwen3.5-4B](https://huggingface.co/Qwen/Qwen3.5-4B) is the compact dense entry in the Qwen3.5 family — same gated delta networks architecture, vision encoder, 262K context, and MTP decoding as the larger siblings, sized for 16 GB consumer GPUs. ## Prerequisites - **vLLM version:** >= 0.17.0 - **Hardware:** single 16 GB GPU (RTX 4080 / L4 / A10 / T4-24GB) or one Xeon 6 NUMA node or one Intel Arc Pro B60/B70 ### Install vLLM ```bash uv venv source .venv/bin/activate uv pip install -U vllm --torch-backend=auto ``` ### pip (Intel Xeon 6 CPUs) For Intel and AMD x86 CPUs, follow the [CPU pre-built wheels](https://docs.vllm.ai/en/latest/getting_started/installation/cpu/#pre-built-wheels) installation instructions. ### Docker ```bash docker pull vllm/vllm-openai-xpu:latest # Intel XPU (B60 / B70) ``` ## Launching the Server ```bash vllm serve Qwen/Qwen3.5-4B \ --max-model-len 262144 \ --reasoning-parser qwen3 ``` ### Intel Xeon 6 Deployment via Docker Launch the x86 CPU vLLM Docker container for `Qwen/Qwen3.5-4B`: ```bash docker run \ --privileged --ipc=host -p 8000:8000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ vllm/vllm-openai-cpu:latest-x86_64 Qwen/Qwen3.5-4B \ --trust-remote-code \ --tensor-parallel-size 1 \ --enable-auto-tool-choice \ --tool-call-parser qwen3_coder \ --reasoning-parser qwen3 ``` ### Docker (Intel XPU B60 / B70) Validated on 1× Intel Arc Pro B60 / B70 (B60 24 GB, B70 32 GB per card) with the official vLLM XPU image `vllm/vllm-openai-xpu:latest`. ```bash docker run --device /dev/dri \ -v /dev/dri/by-path:/dev/dri/by-path --shm-size=16g \ --privileged --ipc=host -p 8000:8000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ vllm/vllm-openai-xpu:latest \ Qwen/Qwen3.5-4B \ --reasoning-parser qwen3 \ --enforce-eager ``` ### MTP speculative decoding ```bash vllm serve Qwen/Qwen3.5-4B \ --speculative-config '{"method": "mtp", "num_speculative_tokens": 1}' \ --reasoning-parser qwen3 ``` ## Client Usage ```python from openai import OpenAI client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1") resp = client.chat.completions.create( model="Qwen/Qwen3.5-4B", messages=[{"role": "user", "content": "Hello!"}], max_tokens=128, ) print(resp.choices[0].message.content) ``` ## Troubleshooting - **CUDA graph / Mamba cache size error:** reduce `--max-cudagraph-capture-size`. - **Disable reasoning:** add `--default-chat-template-kwargs '{"enable_thinking": false}'`. ## References - [Model card](https://huggingface.co/Qwen/Qwen3.5-4B) - [Base checkpoint](https://huggingface.co/Qwen/Qwen3.5-4B-Base) --- # Qwen/Qwen3.5-9B on vLLM Source: https://recipes.vllm.ai/Qwen/Qwen3.5-9B Provider: Qwen Released: 2026-02-27T12:58:26.000Z Updated: 2026-09-14 Architecture: dense Parameters: 9B Context length: 262,144 Min vLLM: 0.17.0 Tasks: multimodal, text Verified hardware: Arc Pro B60, Arc Pro B70 Summary: Qwen3.5 dense multimodal model (9B) with gated delta networks hybrid attention, MTP, and 262K context ## Variants - default: precision=bf16, vram_minimum_gb=22 Full precision BF16 — single 24 GB GPU or Intel Arc Pro B60/B70 - fp8: precision=fp8, vram_minimum_gb=11, model_id=RedHatAI/Qwen3.5-9B-FP8-dynamic RedHatAI FP8 (E4M3) with dynamic per-token activation quantization ## Recommended base flags ```bash vllm serve 'Qwen/Qwen3.5-9B' \ --trust-remote-code ``` ## Guide ## Overview [Qwen3.5-9B](https://huggingface.co/Qwen/Qwen3.5-9B) is a dense multimodal model from the Qwen3.5 family — same gated delta networks hybrid attention, vision encoder, 262K context, and MTP support as its larger siblings, but sized to fit comfortably on a single 24 GB GPU. ## Prerequisites - **vLLM version:** >= 0.17.0 - **Hardware:** single 24 GB GPU (RTX 4090 / L4 / A10G / H100) or Intel Arc Pro B60/B70 ### Install vLLM ```bash uv venv source .venv/bin/activate uv pip install -U vllm --torch-backend=auto ``` ### Docker ```bash docker pull vllm/vllm-openai-xpu:latest # Intel XPU (B60 / B70) ``` ## Launching the Server ### Single-GPU BF16 ```bash vllm serve Qwen/Qwen3.5-9B \ --max-model-len 262144 \ --reasoning-parser qwen3 ``` ### MTP speculative decoding ```bash vllm serve Qwen/Qwen3.5-9B \ --speculative-config '{"method": "mtp", "num_speculative_tokens": 1}' \ --reasoning-parser qwen3 ``` ### Docker (Intel XPU B60 / B70) Validated on 1× Intel Arc Pro B60 / B70 (B60 24 GB, B70 32 GB per card) with the official vLLM XPU image `vllm/vllm-openai-xpu:latest`. ```bash docker run --device /dev/dri \ -v /dev/dri/by-path:/dev/dri/by-path --shm-size=16g \ --privileged --ipc=host -p 8000:8000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ vllm/vllm-openai-xpu:latest \ Qwen/Qwen3.5-9B \ --reasoning-parser qwen3 \ --max-model-len 8192 \ --enforce-eager ``` ## Client Usage ```python from openai import OpenAI client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1") resp = client.chat.completions.create( model="Qwen/Qwen3.5-9B", messages=[{"role": "user", "content": "Hello!"}], max_tokens=128, ) print(resp.choices[0].message.content) ``` ## Troubleshooting - **CUDA graph / Mamba cache size error:** reduce `--max-cudagraph-capture-size` (default 512). See [vLLM PR #34571](https://github.com/vllm-project/vllm/pull/34571). - **Disable reasoning:** add `--default-chat-template-kwargs '{"enable_thinking": false}'`. ## References - [Model card](https://huggingface.co/Qwen/Qwen3.5-9B) - [Base checkpoint](https://huggingface.co/Qwen/Qwen3.5-9B-Base) --- # Qwen/Qwen3.6-35B-A3B on vLLM Source: https://recipes.vllm.ai/Qwen/Qwen3.6-35B-A3B Provider: Qwen Released: 2026-04-15T05:59:19.000Z Updated: 2026-09-14 Architecture: moe Parameters: 35B Context length: 262,144 Min vLLM: 0.17.0 Tasks: multimodal, text Verified hardware: DGX Spark (GB10), H100, H200, MI300X, MI325X, MI355X, RTX Pro 6000, DGX Station (GB300), RTX 5090, Arc Pro B60, Arc Pro B70 Summary: Smaller Qwen3.6 multimodal MoE model (35B total / 3B active) with BF16, FP8, and NVIDIA NVFP4 variants ## Variants - default: precision=bf16, vram_minimum_gb=84 Full precision BF16 — fits on 1x H200, 2x H100, 4x Intel Arc Pro B60 / B70 - fp8: precision=fp8, vram_minimum_gb=42, model_id=Qwen/Qwen3.6-35B-A3B-FP8 Qwen official FP8 checkpoint — single-GPU serving - nvfp4: precision=nvfp4, vram_minimum_gb=21, model_id=nvidia/Qwen3.6-35B-A3B-NVFP4 NVIDIA ModelOpt NVFP4 checkpoint for Blackwell GPUs, including DGX Spark - redhat_nvfp4: precision=nvfp4, vram_minimum_gb=21, model_id=RedHatAI/Qwen3.6-35B-A3B-NVFP4 RedHatAI compressed-tensors NVFP4 quantization for Blackwell GPUs - redhat_fp8: precision=fp8, vram_minimum_gb=42, model_id=RedHatAI/Qwen3.6-35B-A3B-FP8-dynamic RedHatAI FP8 (E4M3) with dynamic per-token activation quantization - redhat_fp8_plain: precision=fp8, vram_minimum_gb=42, model_id=RedHatAI/Qwen3.6-35B-A3B-FP8 RedHatAI FP8 quantization ## Recommended base flags ```bash vllm serve 'Qwen/Qwen3.6-35B-A3B' \ --trust-remote-code ``` ## Guide ## Overview [Qwen3.6-35B-A3B](https://huggingface.co/Qwen/Qwen3.6-35B-A3B) is the smaller sibling of Qwen3.5, sharing the same gated-delta-networks MoE architecture but with 35B total parameters and 3B activated (256 experts, 8 routed + 1 shared). The recipe covers the BF16 base model, Qwen's official FP8 checkpoint, and NVIDIA's ModelOpt NVFP4 checkpoint. ## Prerequisites - **vLLM version:** >= 0.17.0 - **NVFP4 vLLM version:** >= 0.28.0. 0.24.0 loads the checkpoint, but FlashInfer only selects its XQA decode kernel on SM120 GPUs from 0.28.0. - **Hardware (BF16):** 1x H200, 2x H100, 4x Intel Arc Pro B60 / B70 - **Hardware (FP8):** single H100/H200 or 1x MI300X/MI325X/MI355X - **Hardware (NVFP4):** NVIDIA Blackwell GPUs, including DGX Spark (GB10) ## NVFP4 on Blackwell All NVFP4 tuning lives in `variants.nvfp4.hardware_overrides`, keyed by GPU profile, so the exact flag set for a box is readable straight from the recipe YAML (or from `by_hardware` in the JSON API) without going through the command builder. Four profiles are tuned and verified. DGX Spark (GB10) and DGX Station (GB300) run the checkpoint on the default loader with FlashInfer attention, `--moe-backend marlin` and a 3-token MTP draft on Triton; Spark shares unified memory with the host, so `--gpu-memory-utilization` is capped at 0.5. The SM120 parts (RTX PRO 6000, RTX 5090) load through `--quantization modelopt_fp4` with FlashInfer's TRT-LLM attention kernels, which need `--block-size 128` and `VLLM_HAS_FLASHINFER_CUBIN=1`, and sustain a deeper 4-token MTP draft on `flashinfer_cutlass`. The RTX 5090's 32 GB caps context at 64K. Datacenter Blackwell (B200/GB200/B300/GB300) intentionally carries no overrides — it runs vLLM's own NVFP4 defaults, which select the native FP4 path rather than the Marlin W4A16 kernel the workstation parts need. ### Measured on DGX Spark (GB10) The GB10 profile above was re-measured on 2026-08-31 with aiperf 0.12.0 on vLLM 0.28.0 (Ubuntu 24.04.4, driver 580.159.03, CUDA 13.0), one GB10 at TP1, serving from `vllm/vllm-openai:v0.28.0`. On SPEED-Bench at 8K input / 256 output tokens and concurrency 1, it sustained 6,265 prefill tok/s and 97.7 decode tok/s at a 3.95 s mean end-to-end latency. The flag set is unchanged — the sweep confirms the tuning rather than revising it, so the command the builder emits for GB10 is the one that produced these numbers. ### Docker (Intel XPU B60 / B70) Validated on 4× Intel Arc Pro B60 / B70 (B60 24 GB, B70 32 GB per card) with the official vLLM XPU image `vllm/vllm-openai-xpu:latest`. ```bash docker run --device /dev/dri \ -v /dev/dri/by-path:/dev/dri/by-path --shm-size=16g \ --privileged --ipc=host -p 8000:8000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ vllm/vllm-openai-xpu:latest \ Qwen/Qwen3.6-35B-A3B \ --reasoning-parser qwen3 \ --tensor-parallel-size 4 \ --max-model-len 8192 \ --enforce-eager ``` ## Processing Ultra-Long Texts Qwen3.6-35B-A3B natively supports `262,144` tokens. To serve longer inputs, turn on the **Long Context** feature above: it applies the model card's YaRN RoPE configuration via `--hf-overrides` and raises `--max-model-len` to ~1,010,000 tokens. Static YaRN holds the scaling factor constant regardless of input length, so leave it off unless you actually serve long prompts; for a ~524K workload, halve the card's `factor` to `2.0`. See the [model card](https://huggingface.co/Qwen/Qwen3.6-35B-A3B#processing-ultra-long-texts) for the full parameter reference. ## Client Usage ```python from openai import OpenAI client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1") resp = client.chat.completions.create( model="Qwen/Qwen3.6-35B-A3B", messages=[{"role": "user", "content": "Explain gated delta networks in one paragraph."}], max_tokens=512, ) print(resp.choices[0].message.content) ``` ## Troubleshooting - **CUDA graph / Mamba cache size error:** reduce `--max-cudagraph-capture-size` (default 512). See [vLLM PR #34571](https://github.com/vllm-project/vllm/pull/34571). - **Reasoning disable:** add `--default-chat-template-kwargs '{"enable_thinking": false}'`. - **Prefix Caching (Mamba):** currently experimental in "align" mode. ## References - [Qwen3.6-35B-A3B on Hugging Face](https://huggingface.co/Qwen/Qwen3.6-35B-A3B) - [FP8 checkpoint](https://huggingface.co/Qwen/Qwen3.6-35B-A3B-FP8) - [NVIDIA NVFP4 checkpoint](https://huggingface.co/nvidia/Qwen3.6-35B-A3B-NVFP4) - [Qwen3.5 recipe (sibling 397B-A17B flagship)](../Qwen3.5-397B-A17B) --- # Qwen/Qwen3.8-2.4T-A95B on vLLM Source: https://recipes.vllm.ai/Qwen/Qwen3.8-2.4T-A95B Provider: Qwen Released: 2026-08-08T01:50:52.000Z Updated: 2026-09-14 Architecture: moe Parameters: 2.4T Context length: 262,144 Min vLLM: nightly Tasks: text Verified hardware: B300, GB300 NVL4 Summary: 2.4T-parameter hybrid-attention MoE (~95B active) with linear attention on 69 of 92 layers, 512 routed experts, a built-in MTP draft head, 262K native context window and extensible to 1M context ## Variants - default: precision=bf16, vram_minimum_gb=5871 Full-precision BF16 — 4.45 TiB of weights: 24 GPUs, i.e. 3x 8xB300 nodes or 6 GB300 NVL4 trays. - fp8: precision=fp8, vram_minimum_gb=2996, model_id=Qwen/Qwen3.8-2.4T-A95B-FP8 Official block-scaled FP8 checkpoint — 2.27 TiB: 16 GPUs, i.e. 2x 8xB300 nodes or 4 GB300 NVL4 trays (3 trays hold it, but TP12 is invalid — 64 attention heads are not divisible by 12). - mxfp4: precision=mxfp4, vram_minimum_gb=1917, model_id=Inferact/Qwen3.8-2.4T-A95B-MXFP4 MXFP4 build for AMD Instinct — 1.45 TiB: 8 GPUs, a single 8xMI355X or 8xMI325X node. - nvfp4: precision=nvfp4, vram_minimum_gb=1737, model_id=Inferact/Qwen3.8-2.4T-A95B-NVFP4 NVFP4 build with 4-bit activations for NVIDIA Blackwell — 1.32 TiB, the smallest footprint: 8 GPUs, i.e. one 8xB300 node or 2 GB300 NVL4 trays. - nvfp4_reap25: precision=nvfp4, vram_minimum_gb=1373, model_id=RedHatAI/Qwen3.8-2.4T-A95B-NVFP4-REAP-25 NVFP4 quantization + 25% uniform expert pruning via REAP — 4.25x compression with ~99% accuracy recovery ## Recommended base flags Environment: ```bash VLLM_ENGINE_READY_TIMEOUT_S='3600' ``` ## Guide ## Overview Qwen3.8-2.4T-A95B is a 2.4-trillion-parameter Mixture-of-Experts model with roughly 95B parameters active per token — 512 routed experts with 10 active, plus one shared expert, over a 92-layer hybrid-attention backbone. **The layer mix is the interesting part.** Only 23 of the 92 layers run full attention (`full_attention_interval: 4`); the other 69 run linear attention with a constant recurrent state. This is a text-only causal LM: the architecture is `Qwen3_5MoeForCausalLM`, `config.json` has no `vision_config`, and the weight index contains zero vision tensors. ## Prerequisites - **vLLM: nightly.** for the best performance - **transformers >= 5.4.0**, per the version recorded in `config.json`. ```bash uv venv && source .venv/bin/activate uv pip install -U vllm --extra-index-url https://wheels.vllm.ai/nightly uv pip install -U "transformers>=5.4.0" ``` ## Choosing a variant Counted in **GPUs, not nodes** — a "node" is 8 GPUs on B300/MI355X/H200 but only 4 on the GB200/GB300 NVL4 trays, and that difference decides whether a variant fits. | Variant | Weights | Sized | B300 (268 GB) | MI355X (288 GB) | H200 (141 GB) | GB300 tray (4 GPU) | |---|---:|---:|---:|---:|---:|---:| | BF16 | 4.45 TiB | 5871 GB | 24 GPUs | 24 GPUs | 48 GPUs | 6 trays | | FP8 | 2.27 TiB | 2996 GB | 16 GPUs | 16 GPUs | 32 GPUs | **4 trays (TP16)** | | MXFP4 (AMD) | 1.45 TiB | 1917 GB | — | **8 GPUs** | 16 GPUs | — | | NVFP4 W4A4 (NVIDIA) | 1.32 TiB | 1737 GB | **8 GPUs** | — | 16 GPUs | **2 trays (TP8)** | | NVFP4-REAP-25 (RedHatAI) | 1.04 TiB | 1373 GB | **8 GPUs** | — | 16 GPUs | **2 trays (TP8)** | **TP must divide the 64 attention heads**, so only 1/2/4/8/16/32 are legal. This rounds some layouts up: FP8 needs 2325 GiB, which is 3 GB300 trays by capacity, but TP12 is invalid — so FP8 is a **16-GPU / 4-tray** deployment on GB300, not 12. ## Launch commands ### Low latency NVFP4 W4A4, TP8 (one 8xB300 node, or two GB300 trays): ```bash vllm serve Inferact/Qwen3.8-2.4T-A95B-NVFP4 \ --tensor-parallel-size 8 \ --max-model-len 262144 \ --kv-cache-dtype fp8 \ --reasoning-parser qwen3 \ --enable-auto-tool-choice --tool-call-parser qwen3_coder ``` FP8, TP16 across two 8-GPU nodes. Same command on both, changing only `--node-rank`; rank > 0 additionally takes `--headless` (it must not start an API server). ```bash vllm serve Qwen/Qwen3.8-2.4T-A95B-FP8 \ --tensor-parallel-size 16 \ --nnodes 2 --node-rank 0 --master-addr $HEAD_ADDR \ --max-model-len 262144 \ --kv-cache-dtype fp8 \ --reasoning-parser qwen3 ``` Add `--speculative-config '{"method":"mtp","num_speculative_tokens":3}'` for the latency numbers below — MTP-3 is worth roughly 2.3x on per-user output rate. MXFP4 on 8xMI355X — swap the model id and drop `--kv-cache-dtype fp8` unless you have confirmed the ROCm path supports it in your build. ### High throughput Expert parallelism, not plain TP, and a one-sided NVLink all2all. The recipe sets `--all2all-backend flashinfer_nvlink_one_sided` automatically on the EP strategies when the hardware is Blackwell. The MoE kernel differs by precision, so set it explicitly: - **FP8, TP4·DP4 + EP16** (`multi_node_tp_dp`, 4 GB300 trays): add `--moe-backend flashinfer_trtllm --enable-expert-parallel`. See . - **NVFP4, DEP16** (`multi_node_dep`, 4 trays): add `--moe-backend flashinfer_cutedsl`. These MoE backends are deliberately NOT pinned in the variant args: they are measured for the EP layouts above, and the plain-TP path has not been measured with them. vLLM auto-selects `FLASHINFER_TRTLLM` for NVFP4 under TP, which is what the TP8 numbers below were taken with. ### Pipeline parallelism `multi_node_tp_pp` **works** on this architecture — verified 2026-08-13 on GB300 with TP4 x PP3 across 3 trays on the FP8 checkpoint: it loads, captures full CUDA graphs (51 piecewise + 35 full-decode), and generates correctly. Earlier revisions of this recipe warned that hybrid linear-attention models had historically lagged on PP support; that caveat is resolved. It remains a fallback rather than a default — PP exists for GPU counts that pure TP cannot express (12 GPUs is 3 trays but TP12 is illegal), and TP or EP is faster whenever the GPU count is a power of two. ## Performance Two operating points, because they want different layouts. Latency wants TP and speculative decoding; throughput wants expert parallelism. **Low latency** — output tokens/s per user: | Variant | Layout | no MTP | MTP-3 | with kernel opts | |---|---|---:|---:|---:| | FP8 | TP16 | 130 | **307** | 150 / 360 (target) | | NVFP4 | TP8 | 133 | **304** | — | **High throughput** — total (input+output) tokens/s per GPU: | Variant | Layout | Backends | total TPS/GPU | |---|---|---|---:| | FP8 | TP4·DP4 + EP16 | one-sided all2all + trtllm-gen MoE + activation opt | up to **3200** | | NVFP4 | DEP16 | one-sided all2all + cute-dsl MoE | up to **4300** | The pending kernel work behind the "target" column is gdn decode fusion, split-k GEMM, and all-reduce fusion. **Verification run** (what earned `gb300: verified`) — NVFP4 W4A4, TP8 across 2 trays, 8k in / 1k out, vLLM `0.1.dev19754+g3a0914114`, no EP, default MoE backend: | Concurrency | TPS/user | total TPS/GPU | TPOT | |---:|---:|---:|---:| | 1 | 101 | 108 | 9.9 ms | | 32 | 41.9 | 1338 | 23.9 ms | | 64 | 26.9 | 1715 | 37.2 ms | | 128 | 18.3 | 2170 | 54.7 ms | That 2170 against 4300 is the cost of plain TP with the default all2all — it is the measurement the EP + one-sided-NVLink defaults above are based on. Three things that moved the numbers materially: - **Use MTP depth 3, not 1.** MTP-1 was measured at 64.8% acceptance and is not worth it: +3.4% at concurrency 1 and a *loss* above it (−9% at 128, −23% at 256), because the draft pass displaces real work once the batch is compute-bound. Depth 3 is where speculation pays (~2.3x per-user). - **`--load-format fastsafetensors --safetensors-load-strategy lazy`** cut weight load from 545 s to 306 s (−44%) on a 1.32 TiB checkpoint over shared storage. - **Size `--max-model-len` to the workload.** At 262144 the engine reserved KV for 25 concurrent requests; at 9240 (8k in + 1k out) the same 70 GiB of KV serves 506. ## Client usage `generation_config.json` ships `temperature: 1.0`, `top_p: 0.95`, `top_k: 20`. ```python from openai import OpenAI client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1", timeout=3600) resp = client.chat.completions.create( model="Qwen/Qwen3.8-2.4T-A95B", messages=[{"role": "user", "content": "Give me three primes above 100."}], temperature=1.0, top_p=0.95, max_tokens=2048, ) print(resp.choices[0].message.content) ``` ## Processing Ultra-Long Texts The model has 262k native context length and it can be extended to 1M with --max-model-len flag. The best value can be picked based on the use case and GPU ram usage tradeoff. For example, to enable full 1M context length on this model: ```bash VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 vllm serve Qwen/Qwen3.8-2.4T-A95B \ --max-model-len 1010000 \ --hf-overrides '{"max_position_embeddings": 1010000}' \ ... ``` ## Troubleshooting **CUDA graph / recurrent-state cache size error.** On hybrid models an `assert num_cache_lines >= batch` failure means the CUDA-graph capture size exceeds the state cache. Reduce `--max-cudagraph-capture-size` (default 512), or fall back to `--compilation-config '{"cudagraph_mode":"FULL_DECODE_ONLY"}'` if full capture misbehaves on this architecture. See . **Engine start times out.** Raise `VLLM_ENGINE_READY_TIMEOUT_S` (set to 3600 by this recipe) and prefer a real `/v1/chat/completions` probe over `/health` for readiness — loading terabytes plus kernel JIT takes minutes. The linear backend feature "--linear-backend flashinfer_cutedsl" is Docker only until FlashInfer version upgrade in the vllm build. The NVFP4 variant sets that flag, so on a pip install either use the Docker image pinned above or drop the flag. ## References - Model card: https://huggingface.co/Qwen/Qwen3.8-2.4T-A95B - FP8 checkpoint: https://huggingface.co/Qwen/Qwen3.8-2.4T-A95B-FP8 - MXFP4 build (AMD): https://huggingface.co/Inferact/Qwen3.8-2.4T-A95B-MXFP4 - NVFP4 W4A4 build (NVIDIA): https://huggingface.co/Inferact/Qwen3.8-2.4T-A95B-NVFP4 - NVFP4-REAP-25 build (RedHatAI): https://huggingface.co/RedHatAI/Qwen3.8-2.4T-A95B-NVFP4-REAP-25 - REAP: Router-weighted Expert Activation Pruning: https://arxiv.org/abs/2510.13999 - vLLM documentation: https://docs.vllm.ai/ --- # deepseek-ai/DeepSeek-V3.1 on vLLM Source: https://recipes.vllm.ai/deepseek-ai/DeepSeek-V3.1 Provider: DeepSeek Released: 2025-08-21T02:37:52.000Z Updated: 2026-09-14 Architecture: moe Parameters: 671B Context length: 163,840 Min vLLM: 0.12.0 Tasks: text Verified hardware: H200, MI300X, MI325X, MI355X Summary: DeepSeek-V3.1 is a hybrid MoE model that supports dynamic switching between thinking and non-thinking modes, with tool calling and function execution. ## Variants - default: precision=fp8, vram_minimum_gb=805 Native FP8 weights on 8xH200 (or H20) with 141GB per GPU - nvfp4: precision=nvfp4, vram_minimum_gb=403, model_id=nvidia/DeepSeek-V3.1-NVFP4 NVIDIA NVFP4 quantized weights for Blackwell GPUs ## Recommended base flags ```bash vllm serve 'deepseek-ai/DeepSeek-V3.1' \ --trust-remote-code \ --enable-expert-parallel ``` ## Guide ## Overview DeepSeek-V3.1 is a hybrid MoE model that supports both thinking and non-thinking modes. You can dynamically switch between the two modes from the client by passing `extra_body={"chat_template_kwargs": {"thinking": True|False}}`. ## Prerequisites - **Hardware**: 8x H200/H20 (CUDA) or 8x MI300X / MI325X / MI355X (ROCm) - **vLLM**: Current stable release ### CUDA ```bash uv venv source .venv/bin/activate uv pip install -U vllm --torch-backend auto ``` ### ROCm (MI300X, MI325X, MI355X) Requires Python 3.12, ROCm 7.2.1, and glibc >= 2.35. ```bash uv venv --python 3.12 source .venv/bin/activate uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/ ``` ## Launching DeepSeek-V3.1 ### 8xH200 / 8xH20 (CUDA) ```bash vllm serve deepseek-ai/DeepSeek-V3.1 \ --enable-expert-parallel \ --tensor-parallel-size 8 \ --served-model-name ds31 ``` ### 8xMI300X / MI325X / MI355X (ROCm) ```bash export SAFETENSORS_FAST_GPU=1 export VLLM_ROCM_USE_AITER=1 export VLLM_ROCM_USE_AITER_MOE=1 vllm serve deepseek-ai/DeepSeek-V3.1 \ --tensor-parallel-size 8 \ --enable-expert-parallel ``` ### Function calling vLLM supports user-defined tool calling for DeepSeek-V3.1. Add these flags when launching the server. The example chat template ships in the official container and can also be downloaded from the vLLM repo: [`tool_chat_template_deepseekv31.jinja`](https://github.com/vllm-project/vllm/blob/main/examples/tool_chat_template_deepseekv31.jinja). ```bash vllm serve ... \ --enable-auto-tool-choice \ --tool-call-parser deepseek_v31 \ --chat-template examples/tool_chat_template_deepseekv31.jinja ``` ## Client Usage ### OpenAI Python SDK Control thinking mode via `extra_body={"chat_template_kwargs": {"thinking": False}}` (or `True` to enable thinking). ```python from openai import OpenAI client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1") model = client.models.list().data[0].id messages = [ {"role": "system", "content": "You are a helpful assistant"}, {"role": "user", "content": "Who are you?"}, {"role": "assistant", "content": "HmmI am DeepSeek"}, {"role": "user", "content": "9.11 and 9.8, which is greater?"}, ] response = client.chat.completions.create( model=model, messages=messages, extra_body={"chat_template_kwargs": {"thinking": False}}, ) print(response.choices[0].message.content) ``` When `thinking=True`, output includes a `` segment delimiting chain-of-thought; when `thinking=False`, the model produces a direct answer without the thinking segment. ### curl ```bash curl http://localhost:8000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "ds31", "messages": [ {"role": "user", "content": "9.11 and 9.8, which is greater?"} ], "chat_template_kwargs": {"thinking": true} }' ``` ## References - [DeepSeek-V3.1 on Hugging Face](https://huggingface.co/deepseek-ai/DeepSeek-V3.1) - [vLLM tool chat template for DeepSeek-V3.1](https://github.com/vllm-project/vllm/blob/main/examples/tool_chat_template_deepseekv31.jinja) --- # deepseek-ai/DeepSeek-V3.2-Exp on vLLM Source: https://recipes.vllm.ai/deepseek-ai/DeepSeek-V3.2-Exp Provider: DeepSeek Released: 2025-09-29T06:07:26.000Z Updated: 2026-09-14 Architecture: moe Parameters: 671B Context length: 163,840 Min vLLM: 0.12.0 Tasks: text Verified hardware: H200, MI300X, MI325X, MI355X Summary: Experimental DeepSeek-V3.2 preview with sparse attention (MQA-like logits) and FP8 KV cache; architecture matches DeepSeek-V3.1 except for the sparse attention mechanism. ## Variants - default: precision=fp8, vram_minimum_gb=805 Native FP8 weights on 8xH200 (or H20, or 8xB200) with FP8 KV cache default ## Recommended base flags ```bash vllm serve 'deepseek-ai/DeepSeek-V3.2-Exp' \ --trust-remote-code ``` ## Guide ## Overview DeepSeek-V3.2-Exp is a sparse-attention MoE preview. Its main architecture is similar to DeepSeek-V3.1, with a sparse attention mechanism. This guide covers deployment on NVIDIA Hopper/Blackwell GPUs and AMD MI300X / MI325X / MI355X GPUs. ## Prerequisites - **Hardware**: 8x H200/H20/B200 (CUDA) or 8x MI300X / MI325X / MI355X (ROCm) - **vLLM**: Current stable release - **DeepGEMM** (CUDA only): Required for MQA logits computation (and optionally for MoE) ### CUDA ```bash source .venv/bin/activate uv pip install -U vllm --torch-backend auto uv pip install git+https://github.com/deepseek-ai/DeepGEMM.git@v2.1.1.post3 --no-build-isolation ``` ### ROCm (MI300X, MI325X, MI355X) Requires Python 3.12, ROCm 7.2.1, and glibc >= 2.35. If your environment does not meet these requirements, use the Docker-based setup described in the [documentation](https://docs.vllm.ai/en/latest/getting_started/installation/gpu/#pre-built-images). ```bash uv venv --python 3.12 source .venv/bin/activate uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/ ``` Note: DeepGEMM is used both for MoE and for MQA logits computation. It is required for MQA logits on CUDA. To disable the MoE path only, set `VLLM_USE_DEEP_GEMM=0`. Some users report better performance with `VLLM_USE_DEEP_GEMM=0` (e.g. on H20), and this also skips the long warmup. ## Launching DeepSeek-V3.2-Exp ### 8xH200 / 8xH20 (CUDA) Using the recommended EP/DP mode: ```bash vllm serve deepseek-ai/DeepSeek-V3.2-Exp -dp 8 --enable-expert-parallel ``` Using tensor parallel: ```bash vllm serve deepseek-ai/DeepSeek-V3.2-Exp -tp 8 ``` ### 8xB200 (CUDA) Same as 8xH200 / 8xH20. ### 8xMI300X / MI325X / MI355X (ROCm) ```bash export SAFETENSORS_FAST_GPU=1 export VLLM_ROCM_USE_AITER=1 export VLLM_ROCM_USE_AITER_MOE=1 vllm serve deepseek-ai/DeepSeek-V3.2-Exp \ --tensor-parallel-size 8 \ --max-num-batched-tokens 32768 \ --trust-remote-code \ --kv-cache-dtype bfloat16 \ --block-size 1 ``` ## Accuracy Benchmarking ```bash lm-eval --model local-completions --tasks gsm8k \ --model_args model=deepseek-ai/DeepSeek-V3.2-Exp,base_url=http://127.0.0.1:8000/v1/completions,num_concurrent=100,max_retries=3,tokenized_requests=False ``` Reported GSM8K score: `0.9591` (5-shot) and `0.9538` (20-shot). ## Performance Tips 1. The kernels are mainly optimized for TP=1, so it is recommended to run this model under EP/DP mode (e.g. DP=8, EP=8, TP=1). If you hit any errors or hangs, try tensor parallel instead. Simple TP works and is more robust, but the performance is not optimal. 2. The default config uses a custom `fp8` KV cache. You can also use `bfloat16` KV cache by specifying `kv_cache_dtype=bfloat16`. FP8 allows more tokens to be cached but incurs quantization/dequantization overhead. Use `bfloat16` for short requests and `fp8` for long requests. ## Troubleshooting - **`CUDA error (flashmla-src/csrc/smxx/mla_combine.cu:201): invalid configuration argument`**: This may be caused by too large a batch size. Try `--max-num-seqs 256` or smaller (default is 1024). - For thinking-mode toggling, refer to the DeepSeek-V3.1 recipe (`deepseek-ai/DeepSeek-V3.1`). ## References - [DeepSeek-V3.2-Exp on Hugging Face](https://huggingface.co/deepseek-ai/DeepSeek-V3.2-Exp) - [End-to-end tutorial (Jupyter Notebook)](https://github.com/vllm-project/recipes/blob/main/DeepSeek/DeepSeek_v3_2_vLLM_getting_started_guide.ipynb) --- # deepseek-ai/DeepSeek-V3.2 on vLLM Source: https://recipes.vllm.ai/deepseek-ai/DeepSeek-V3.2 Provider: DeepSeek Released: 2025-12-01T02:34:49.000Z Updated: 2026-09-14 Architecture: moe Parameters: 671B Context length: 163,840 Min vLLM: 0.18.0 Tasks: text Verified hardware: H100, H200 Summary: DeepSeek V3.2 MoE model with MLA attention, sparse attention, and scalable RL for strong reasoning and agent capabilities. ## Variants - default: precision=fp8, vram_minimum_gb=805 Native FP8 checkpoint (F8_E4M3) - nvfp4: precision=nvfp4, vram_minimum_gb=403, model_id=nvidia/DeepSeek-V3.2-NVFP4 NVIDIA FP4 quantized variant with FP8 KV cache for reduced VRAM usage. - redhat_nvfp4: precision=nvfp4, vram_minimum_gb=430, model_id=RedHatAI/DeepSeek-V3.2-NVFP4-FP8-BLOCK RedHatAI NVFP4+FP8 block mixed-precision checkpoint ## Recommended base flags ```bash vllm serve 'deepseek-ai/DeepSeek-V3.2' \ --trust-remote-code \ --disable-flashinfer-autotune ``` ## Guide ## Overview DeepSeek-V3.2 is a Mixture-of-Experts model that balances computational efficiency with strong reasoning and agent capabilities through three technical innovations: DeepSeek Sparse Attention (DSA) for efficient long-context processing, a scalable reinforcement learning framework achieving GPT-5-level performance, and a large-scale agentic task synthesis pipeline for robust tool-use generalization. ## Prerequisites - **Hardware**: Minimum 8x H100/H200 80GB GPUs (BF16) or 3x H200 (NVFP4 variant). - **vLLM**: Version 0.18.0 or later (nightly recommended). - **Python**: 3.10+ - **CUDA**: 12.x or later (CUDA 13.x may require extra env vars; see Troubleshooting). - **Disk**: ~1.3 TB for BF16 weights; ~350 GB for NVFP4 variant. - **DeepGEMM** (recommended): ```bash uv pip install git+https://github.com/deepseek-ai/DeepGEMM.git@v2.1.1.post3 --no-build-isolation ``` Note: Set `VLLM_USE_DEEP_GEMM=0` to disable MoE DeepGEMM if you experience issues (e.g., on H20 GPUs) or want to skip the long warmup. ## Client Usage Launch the server: ```bash vllm serve deepseek-ai/DeepSeek-V3.2 \ --tensor-parallel-size 8 \ --trust-remote-code \ --disable-flashinfer-autotune \ --tokenizer-mode deepseek_v32 \ --tool-call-parser deepseek_v32 \ --enable-auto-tool-choice \ --reasoning-parser deepseek_v3 ``` Use the OpenAI Python SDK to interact with the server: ```python from openai import OpenAI client = OpenAI( api_key="your-api-key", base_url="http://localhost:8000/v1", ) # Standard chat response = client.chat.completions.create( model="deepseek-ai/DeepSeek-V3.2", messages=[{"role": "user", "content": "Hello!"}], ) # Thinking / reasoning mode response = client.chat.completions.create( model="deepseek-ai/DeepSeek-V3.2", messages=[{"role": "user", "content": "Solve this step by step..."}], extra_body={"chat_template_kwargs": {"thinking": True}}, ) ``` ## Troubleshooting **`ptxas fatal: Value 'sm_110a' is not defined for option 'gpu-name'`** This can occur on CUDA 13.x. Fix by exporting: ```bash export TRITON_PTXAS_PATH=/usr/local/cuda/bin/ptxas export PATH=/usr/local/cuda/bin:$PATH export LD_LIBRARY_PATH=/usr/local/cuda/lib64:${LD_LIBRARY_PATH:-} ``` **TP=8 performance on Hopper/Blackwell** Avoid `-tp 8` with FlashMLA-Sparse. Due to kernel restrictions, TP=8 yields only 16 heads per rank but is padded to 64, causing overhead. Prefer TP=2 (Hopper) or TP=1 (Blackwell) with DP/EP mode: `vllm serve deepseek-ai/DeepSeek-V3.2 -dp 8 --enable-expert-parallel`. **DeepGEMM warmup too slow** Set `VLLM_USE_DEEP_GEMM=0` to disable MoE DeepGEMM and skip the long warmup. ## References - [DeepSeek-V3.2 on Hugging Face](https://huggingface.co/deepseek-ai/DeepSeek-V3.2) - [DeepSeek API Documentation](https://api-docs.deepseek.com/) - [vLLM Data Parallel Deployment Guide](https://docs.vllm.ai/en/latest/serving/data_parallel_deployment.html) --- # deepseek-ai/DeepSeek-V4-Flash-Vision-Exp on vLLM Source: https://recipes.vllm.ai/deepseek-ai/DeepSeek-V4-Flash-Vision-Exp Provider: DeepSeek Released: 2026-08-31T06:16:18.000Z Updated: 2026-09-14 Architecture: moe Parameters: 285B Context length: 1,048,576 Min vLLM: 0.30.0 Tasks: text, multimodal Verified hardware: GB200 NVL4, MI350X, Ascend 950DT 8x, Ascend 950PR 8x Summary: DeepSeek's first experimental multimodal V4 model — the V4-Flash MoE backbone plus a 32-layer vision tower, 1M context, and a fused DSpark draft module. ## Variants - default: precision=fp8, vram_minimum_gb=202 Native FP4+FP8 mixed checkpoint (MoE experts FP4, attention/norm/router FP8) with a BF16 vision tower, aligner, and the fused DSpark draft module. ## Recommended base flags ```bash vllm serve 'deepseek-ai/DeepSeek-V4-Flash-Vision-Exp' \ --kv-cache-dtype \ fp8 \ --block-size \ 256 ``` ## Guide ## Overview DeepSeek-V4-Flash-Vision-Exp is DeepSeek's first multimodal model in the V4 family. It keeps the DeepSeek-V4-Flash language backbone — 43 layers, 256 routed experts with 6 active per token, Compressed Sparse Attention plus manifold-constrained hyper-connections, a 1,048,576-token context window — and adds a 32-layer / 1024-dim ViT with a two-layer aligner (~0.5B parameters on top of the 284B backbone). Weights are **FP4+FP8 mixed**: MoE experts in FP4, the remaining attention / norm / router params in FP8, vision tower in BF16. The checkpoint is 48 shards / ~168 GB and carries the fused **DSpark** draft module. DeepSeek reports it beats DeepSeek-V4-Flash-0731 on multimodal agent benchmarks (ApexBench 36.5 vs 26.2, Agents' Last Exam 27.3 vs 25.2) while holding text-agent parity (Terminal Bench 2.1 at 83.9, DeepSWE at 59.3). ## Prerequisites Vision support is not in a stable vLLM release yet — it lives in [vllm-project/vllm#54566](https://github.com/vllm-project/vllm/pull/54566). Until that lands, use the pinned `vllm/vllm-openai:deepseekv4-flash-vision` image; the official wheel routes this checkpoint to the text-only class and fails on the vision tensors. - **Weights:** ~168 GB on disk (48 shards). Budget ~202 GB of VRAM before KV cache. - **Hardware:** NVIDIA (verified on GB200) and AMD MI350X (gfx950); XPU still has no vision implementation. - **Tokenizer / chat template:** nothing to pass. The repo ships no Jinja template (prompt encoding lives in `encoding/`); vLLM resolves `--tokenizer-mode` to `deepseek_v4` automatically, which is what turns OpenAI content blocks into `<｜deepseek_image｜>` placeholders. ## Launching the server ### Verified configuration — one GB200 NVL4 tray (TP4 + EP) ```bash export VLLM_FLASHINFER_AUTOTUNE_SKIP_OPS="trtllm_fp4_block_scale_moe,flashinfer::trtllm_fp4_block_scale_moe" vllm serve deepseek-ai/DeepSeek-V4-Flash-Vision-Exp \ --tensor-parallel-size 4 \ --enable-expert-parallel \ --kv-cache-dtype fp8 \ --block-size 256 \ --max-model-len 32768 \ --speculative-config '{"method":"dspark","model":"deepseek-ai/DeepSeek-V4-Flash-Vision-Exp","num_speculative_tokens":3,"draft_sample_method":"probabilistic","enable_adaptive_verification":true}' \ --allowed-local-media-path /mnt/lustre ``` Three things there are environment-specific, which is why the builder above leaves them out: - `VLLM_FLASHINFER_AUTOTUNE_SKIP_OPS` excludes the FP4 block-scale MoE op from FlashInfer's startup autotuning. Export it if autotuning that op fails or costs more startup time than it earns back on your FlashInfer build; it is a tuning knob, not a requirement. The Advanced row's `--disable-flashinfer-autotune` turns autotuning off wholesale instead. - `--max-model-len 32768` bounds KV cache for the benchmark. The checkpoint advertises 1M tokens; that is **not** what was measured, and 1M-token KV needs far more memory than the weights do. Raise it deliberately, or use the Advanced row's `--max-model-len auto`. - `--allowed-local-media-path` is only needed for `file://` image URLs. Drop it if clients send `http(s)://` or base64 data URLs. ### 8-GPU nodes (H200 / B200 / B300) Same **Tensor + Expert Parallel** strategy; the builder emits `--tensor-parallel-size 8 --enable-expert-parallel`. The language backbone is the one the [DeepSeek-V4-Flash](/deepseek-ai/DeepSeek-V4-Flash) recipe already runs there, but the vision path has no published run on those GPUs — expect to tune `--gpu-memory-utilization` and `--max-num-seqs`. The Blackwell `--attention_config.indexer_kv_dtype mxfp4` / `--moe-backend deep_gemm_mega_moe` tweaks the text recipe applies are deliberately not set here: the verified vision run did not use them. ### AMD MI350X (gfx950) ROCm vision support is merged (#55107 on the #54566 base); a `vllm/vllm-openai-rocm:nightly` built on/after 2026-09-11 runs the checkpoint natively. Three env vars are required: - `VLLM_ROCM_USE_AITER=1` — the sparse-attention indexer's ROCm path runs only on AITER; without it the engine aborts at KV-cache profiling. - `VLLM_USE_BREAKABLE_CUDAGRAPH=1` — the multimodal wrapper is not torch-compiled, so the default `FULL_AND_PIECEWISE` capture cannot attach without breakable graphs. - `VLLM_USE_V2_MODEL_RUNNER=1` — required for DSpark (the V1 ROCm runner refuses it); harmless for plain serving. Verified single-node launch: ```bash docker run -d --name dsv4-vision \ --ipc host --network host --privileged --shm-size 128g \ --device /dev/kfd --device /dev/dri --group-add video \ -v ~/.cache/huggingface:/root/.cache/huggingface \ -e VLLM_ROCM_USE_AITER=1 \ -e VLLM_USE_BREAKABLE_CUDAGRAPH=1 \ -e VLLM_USE_V2_MODEL_RUNNER=1 \ vllm/vllm-openai-rocm:nightly \ deepseek-ai/DeepSeek-V4-Flash-Vision-Exp \ --tensor-parallel-size 2 \ --kv-cache-dtype fp8 --block-size 256 \ --gpu-memory-utilization 0.9 \ --max-model-len 32768 ``` Observed backends: AITER Flash-Attention for the ViT, DeepSeek `fp8_ds_mla` KV cache, FP8 Lightning-Indexer cache, `DEEPSEEK_SPARSE_SWA` (block 256), and the `ROCM_FLASHMLA_SPARSE_DSV4` MLA backend. TP2 loads 75.2 GiB/card (1.76M-token KV pool at 131K context); TP4 loads 38.8 GiB/card. **Stable v0.30.0 and single-card TP1** (verified 2026-10-02 on one MI350X). The stable `vllm/vllm-openai-rocm:v0.30.0` image (ROCm 7.2.3) already contains #54566 and #55107 and runs the checkpoint natively — text, vision and DSpark — with the same three env vars. On v0.30.0 only, also pass `--compilation-config '{"cudagraph_mode":"FULL_DECODE_ONLY"}'`: v0.30.0's ROCm platform still defaults DeepSeek V4 on gfx950 + ModelRunner V2 to eager (the #52644 workaround, removed by #57229 after v0.30.0 was branched), so without it decode runs without CUDA graphs. Batch 1 at TP1: 12 tok/s eager vs 94 tok/s with `FULL_DECODE_ONLY`, and 137–164 tok/s with DSpark (k=3, adaptive verification off per the AMD override). Against eager in the same run, a 25-item known-answer check scored 25/25 and decode-vs-prefill logprobs agreed within eager's range. Nightlies after 2026-09-17 and v0.31+ don't need the override. TP1 fits one 288 GB card (147.9 GiB of weights). ## Reasoning modes Same three-tier control as the rest of the V4 family, driven from `chat_template_kwargs` rather than a flag — `encoding/encoding_dsv4.py` names the effort levels `low` / `high` / `max`. DeepSeek evaluated this model at `reasoning_effort: "max"` with `temperature = 1.0, top_p = 0.95`. ```python from openai import OpenAI client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY") model = "deepseek-ai/DeepSeek-V4-Flash-Vision-Exp" resp = client.chat.completions.create( model=model, messages=[{"role": "user", "content": "What is 17*19? Return only the final integer."}], temperature=1.0, top_p=0.95, extra_body={ "chat_template_kwargs": {"thinking": True, "reasoning_effort": "max"}, }, ) print(resp.choices[0].message.reasoning) print(resp.choices[0].message.content) ``` ## Client usage — image input Standard OpenAI multi-part content. Multiple images per request are allowed with no cap, so the practical limit is `--max-model-len`. ```python from openai import OpenAI client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY") resp = client.chat.completions.create( model="deepseek-ai/DeepSeek-V4-Flash-Vision-Exp", messages=[{ "role": "user", "content": [ {"type": "image_url", "image_url": {"url": "https://example.com/chart.png"}}, {"type": "text", "text": "Read the y-axis label and the highest bar's value."}, ], }], max_tokens=512, ) print(resp.choices[0].message.content) ``` ```bash curl http://localhost:8000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "deepseek-ai/DeepSeek-V4-Flash-Vision-Exp", "messages": [{ "role": "user", "content": [ {"type": "image_url", "image_url": {"url": "https://example.com/chart.png"}}, {"type": "text", "text": "Describe the image."} ] }], "max_tokens": 512 }' ``` Image preprocessing is fixed by the checkpoint config (the processor takes no kwargs): each image costs at most 387 prompt tokens (`vision_max_n_token: 384` plus the compressor-alignment pad), images below `vision_min_pixels: 147456` are upscaled, and aspect ratios beyond 8:1 are cropped. An image's exact token count depends on where it sits in the prompt, since the compressor pads each image to a 4-token block boundary. ## Speculative decoding DSpark drafts from the module fused into the checkpoint, so `--speculative-config` names the target repo as its own `model` — there is no separate draft repo to download. Measured over the full OCRBench run on GB200 (mixed text+image traffic): **2.99** tokens/forward mean acceptance length, **66.3%** overall acceptance (83.9% / 66.5% / 50.7% by draft position). Image tokens in the prompt do not degrade acceptance — the drafter reads image content through the target's hidden states. Stay at `num_speculative_tokens: 3` unless you verify acceptance yourself; the checkpoint's trained DSpark block width is 5, but 3 is the depth with published numbers. ## Benchmarking OCRBench, full 1000 samples, on the verified GB200 configuration: **835/1000 (83.5%)**, 0 request errors, 312.6 s (~3.2 samples/s). Strongest categories are Key Information Extraction (92.5%), Doc-oriented VQA (90.0%) and Scene Text VQA (89.5%); weakest are handwritten math (49.0%) and handwriting recognition (58.0%). For throughput: ```bash vllm bench serve \ --model deepseek-ai/DeepSeek-V4-Flash-Vision-Exp \ --dataset-name random \ --random-input-len 4096 \ --random-output-len 1024 \ --num-prompts 200 ``` ## Limitations - **Pre-release.** Flags and defaults can change before #54566 lands; re-check before deploying, and keep the image pinned. - **XPU has no vision path** — it raises `NotImplementedError` at model construction. - **GB200 (NVIDIA) and MI350X (AMD) are the hardware with a verified vision run.** Every other pill is the repo's fail-open default, not a tested claim. - **PD disaggregation is not offered**, and the CPU/filesystem KV-offload pills stay off — neither has a verified run to record. The Mooncake KV-store pills follow the repo's fail-open default and are offered, but untested here. - **1M context is advertised, not measured.** The reference run used 32K. ## References - [Model card](https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-Vision-Exp) - [vLLM support PR #54566](https://github.com/vllm-project/vllm/pull/54566) - [vLLM ROCm vision PR #55107](https://github.com/vllm-project/vllm/pull/55107) - [DeepSeek-V4-Flash recipe](/deepseek-ai/DeepSeek-V4-Flash) — the text-only sibling - [DeepSpec (DSpark)](https://github.com/deepseek-ai/DeepSpec) --- # internlm/Intern-S2-397B on vLLM Source: https://recipes.vllm.ai/internlm/Intern-S2-397B Provider: InternLM Released: 2026-09-13T03:20:35.000Z Updated: 2026-09-14 Architecture: moe Parameters: 397B Context length: 262,144 Min vLLM: 0.22.1 Tasks: multimodal, text Summary: Flagship scientific multimodal MoE (397B total / 17B active) on the Qwen3.5 hybrid linear/full-attention architecture — 262K context, MTP-accelerated reasoning. BF16 and FP8 checkpoints. ## Variants - default: precision=bf16, vram_minimum_gb=969 Full precision BF16 — requires 8x H200 or equivalent - fp8: precision=fp8, vram_minimum_gb=488, model_id=internlm/Intern-S2-397B-FP8 Official FP8 checkpoint — the configuration validated in the deployment guide on 8x H100/H200 ## Recommended base flags ```bash vllm serve 'internlm/Intern-S2-397B' \ --trust-remote-code ``` Environment: ```bash VLLM_DEEP_GEMM_WARMUP='skip' VLLM_USE_DEEP_GEMM='0' ``` ## Guide ## Overview [Intern-S2-397B](https://huggingface.co/internlm/Intern-S2-397B) is Shanghai AI Laboratory's flagship multimodal foundation model for scientific intelligence and long-horizon agents. It packs 397B total / 17B active parameters across 512 experts on the Qwen3.5 MoE architecture (`Qwen3_5MoeForConditionalGeneration`) with hybrid linear/full attention, a 262K context window, and a built-in shared-weight MTP head for speculative decoding. An official FP8 checkpoint ([internlm/Intern-S2-397B-FP8](https://huggingface.co/internlm/Intern-S2-397B-FP8)) halves the memory footprint and is the configuration the vendor validates. Beyond chat and reasoning, it handles image and video inputs and improves agent capabilities for scientific workflows. Time-series inference exists but is currently LMDeploy-only (see Troubleshooting). ## Prerequisites - **vLLM version:** 0.22.1 or newer (per the official deployment guide) - **Hardware (FP8, recommended):** 8x H100 or 8x H200 - **Hardware (BF16):** 8x H200 (141 GB each) - **Trust remote code:** required — the repo ships a custom tokenizer (`InternS1Tokenizer`) - **MoE kernel env:** the deployment guide disables DeepGEMM (`VLLM_USE_DEEP_GEMM=0`, already set by this recipe) and selects the FlashInfer TRTLLM MoE backend on NVIDIA (applied automatically when you pick an NVIDIA GPU above) ### Install vLLM ```bash uv venv source .venv/bin/activate uv pip install -U "vllm>=0.22.1" --torch-backend=auto ``` ## Launch commands ### Recommended — official FP8 checkpoint, TP=8 ```bash vllm serve internlm/Intern-S2-397B-FP8 \ --trust-remote-code \ --tensor-parallel-size 8 \ --reasoning-parser qwen3 \ --enable-auto-tool-choice \ --tool-call-parser qwen3_coder \ --mm-encoder-tp-mode data ``` ### BF16 (8x H200) ```bash vllm serve internlm/Intern-S2-397B \ --trust-remote-code \ --tensor-parallel-size 8 \ --reasoning-parser qwen3 \ --enable-auto-tool-choice \ --tool-call-parser qwen3_coder \ --mm-encoder-tp-mode data ``` ### With MTP speculative decoding ```bash vllm serve internlm/Intern-S2-397B-FP8 \ --trust-remote-code \ --tensor-parallel-size 8 \ --reasoning-parser qwen3 \ --enable-auto-tool-choice \ --tool-call-parser qwen3_coder \ --mm-encoder-tp-mode data \ --speculative-config '{"method":"mtp","num_speculative_tokens":3}' ``` ### Long-context (YaRN, up to ~1M) The base config sets `max_position_embeddings = 262144`. For longer contexts, turn on the **Long Context** feature above — it applies the deployment guide's YaRN RoPE override (factor 4.0) and raises `--max-model-len` to 1,010,000. Static YaRN holds the scaling factor constant regardless of input length, so leave it off unless you actually serve long prompts; for a ~524K workload, halve the `factor` to `2.0`. ## Client Usage Recommended sampling parameters from the model card: ```python from openai import OpenAI client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1") resp = client.chat.completions.create( model="internlm/Intern-S2-397B-FP8", messages=[{"role": "user", "content": "Design a synthesis route for paracetamol."}], temperature=0.8, top_p=0.95, max_tokens=32768, extra_body={ "top_k": 50, "min_p": 0.0, "spaces_between_special_tokens": False, }, ) print(resp.choices[0].message.content) ``` ### Toggle thinking mode Thinking is enabled by default. Disable it per request: ```python resp = client.chat.completions.create( model="internlm/Intern-S2-397B-FP8", messages=[{"role": "user", "content": "What is AGI?"}], temperature=0.8, top_p=0.95, extra_body={"chat_template_kwargs": {"enable_thinking": False}}, ) ``` > The model card notes: do **not** disable thinking mode for agentic tasks. ## Troubleshooting - **OOM at full 262K context:** drop `--max-model-len` to 65536 or 131072, or lower `--gpu-memory-utilization` headroom. - **DeepGEMM warmup failures / slow startup:** this recipe already exports `VLLM_DEEP_GEMM_WARMUP=skip` and `VLLM_USE_DEEP_GEMM=0` per the deployment guide — if you launch outside the builder, set them yourself. - **Time-series inputs** (`time_series_url`): currently supported only on LMDeploy, not vLLM — see the model card. - **Wrong parser name:** LMDeploy examples use `--tool-call-parser interns2-preview`; on vLLM the correct parser is `qwen3_coder` (already wired into the Tool Calling feature above). ## References - [Model card](https://huggingface.co/internlm/Intern-S2-397B) - [FP8 checkpoint](https://huggingface.co/internlm/Intern-S2-397B-FP8) - [Deployment guide](https://huggingface.co/internlm/Intern-S2-397B-FP8/blob/main/deployment_guide.md) - [Intern-S1 GitHub](https://github.com/InternLM/Intern-S1) - [vLLM documentation](https://docs.vllm.ai/) --- # tencent/Hy3-preview on vLLM Source: https://recipes.vllm.ai/tencent/Hy3-preview Provider: Hunyuan (Tencent) Released: 2026-04-13T06:07:57.000Z Updated: 2026-09-14 Architecture: moe Parameters: 295B Context length: 262,144 Min vLLM: 0.20.0 Tasks: text Verified hardware: H200, MI300X, MI325X, MI355X Summary: Tencent Hunyuan Hy3-preview — scaled-up MoE language model (295B total / 21B active) with a 3.8B MTP layer for speculative decoding, 256K context, and hy_v3 tool/reasoning parsers ## Variants - default: precision=bf16, vram_minimum_gb=708 Full precision BF16 — 8×H200 or 8×H20-3e(141GB) minimum for weights + KV cache ## Recommended base flags ```bash vllm serve 'tencent/Hy3-preview' \ --trust-remote-code ``` ## Guide # Hy3-preview Usage Guide Hy3-preview is Tencent Hunyuan's latest open-source Mixture-of-Experts language model: 295B total parameters with 21B activated per token, plus a 3.8B MTP layer for speculative decoding. 80 transformer layers, 192 routed experts (top-8) + 1 shared expert, GQA with 64 heads over 8 KV heads, 256K context. A pretrained base checkpoint is published at [tencent/Hy3-preview-Base](https://huggingface.co/tencent/Hy3-preview-Base); this recipe covers the instruct model. ## Setup Choose one of the following setup methods. ### Using Docker ```bash docker run --gpus all \ -p 8000:8000 \ --ipc=host \ -v ~/.cache/huggingface:/root/.cache/huggingface \ vllm/vllm-openai:hy3-preview tencent/Hy3-preview \ --tensor-parallel-size 8 \ --tool-call-parser hy_v3 \ --reasoning-parser hy_v3 \ --enable-auto-tool-choice \ --served-model-name hy3-preview ``` ### Installing from source ```bash uv venv --python 3.12 --seed --managed-python source .venv/bin/activate git clone https://github.com/vllm-project/vllm.git cd vllm uv pip install --editable . --torch-backend=auto ``` ## Model Deployment To serve Hy3-preview on 8 GPUs, use H20-3e(141GB), H200, AMD MI300X/MI325X (192 GB), AMD MI350X/MI355X (288 GB), or other GPUs with larger memory capacity. Smaller-memory 8-GPU configurations (8×H100 80GB, 8×A100 80GB) do not fit the BF16 weights plus KV cache — use multi-node TP for those. ### Serving on 8×AMD MI300X / MI325X / MI350X / MI355X Hy3-preview support is being added in vLLM PR [#40681](https://github.com/vllm-project/vllm/pull/40681). Until it merges, AMD users must build vLLM from the PR branch inside the published ROCm vLLM nightly image (`rocm/vllm-dev:nightly`). See the AMD install note in `hardware_overrides.amd` above for the full reproducer. Once vLLM is installed, serve with the standard launcher plus the AITER environment variables (the recipe's `hardware_overrides.amd.extra_env` applies these automatically when the AMD profile is selected on the recipe site): ```bash export VLLM_ROCM_USE_AITER=1 export VLLM_ROCM_USE_AITER_MOE=1 export VLLM_ROCM_USE_AITER_MHA=1 export VLLM_ROCM_USE_AITER_RMSNORM=1 export VLLM_ROCM_USE_AITER_LINEAR=1 vllm serve tencent/Hy3-preview \ --tensor-parallel-size 8 \ --tool-call-parser hy_v3 \ --reasoning-parser hy_v3 \ --enable-auto-tool-choice \ --served-model-name hy3-preview \ --gpu-memory-utilization 0.90 ``` MTP (recommended on AMD for lower latency, same flags as the NVIDIA path): ```bash vllm serve tencent/Hy3-preview \ --tensor-parallel-size 8 \ --speculative-config.method mtp \ --speculative-config.num_speculative_tokens 1 \ --tool-call-parser hy_v3 \ --reasoning-parser hy_v3 \ --enable-auto-tool-choice \ --served-model-name hy3-preview \ --gpu-memory-utilization 0.90 ``` ### Serving on 8×H200 or 8×H20-3e(141GB) Without Multi-Token Prediction (MTP): ```bash vllm serve tencent/Hy3-preview \ --tensor-parallel-size 8 \ --tool-call-parser hy_v3 \ --reasoning-parser hy_v3 \ --enable-auto-tool-choice \ --served-model-name hy3-preview ``` With MTP (recommended for lower latency): ```bash vllm serve tencent/Hy3-preview \ --tensor-parallel-size 8 \ --speculative-config.method mtp \ --speculative-config.num_speculative_tokens 1 \ --tool-call-parser hy_v3 \ --reasoning-parser hy_v3 \ --enable-auto-tool-choice \ --served-model-name hy3-preview ``` ## Sampling and Reasoning Modes Tencent's recommended sampling parameters: `temperature=0.9`, `top_p=1.0`. Reasoning is controlled via `chat_template_kwargs.reasoning_effort`: | Value | Behavior | |:---|:---| | `no_think` (default) | Direct response, no chain-of-thought | | `low` | Light reasoning | | `high` | Deep chain-of-thought for math/coding/complex reasoning | When tools are registered, set `interleaved_thinking: true` to allow the model to think between tool calls. ### OpenAI Client Example ```bash uv pip install -U openai ``` ```python from openai import OpenAI client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY") messages = [ {"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Hello."}, ] # Direct response (default). resp = client.chat.completions.create( model="hy3-preview", messages=messages, temperature=0.9, top_p=1.0, max_tokens=4096, ) print(resp.choices[0].message.content) # Deep reasoning: set reasoning_effort (and interleaved_thinking if using tools). resp_think = client.chat.completions.create( model="hy3-preview", messages=messages, temperature=0.9, top_p=1.0, max_tokens=4096, extra_body={ "chat_template_kwargs": { "reasoning_effort": "high", "interleaved_thinking": True, }, }, ) output_msg = resp_think.choices[0].message print(output_msg.reasoning) # chain-of-thought print(output_msg.content) # final answer ``` ### cURL Usage ```bash curl http://localhost:8000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "hy3-preview", "messages": [ {"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Hello."} ], "temperature": 0.9, "top_p": 1.0, "max_tokens": 4096 }' ``` ## Benchmarking For benchmarking, disable prefix caching by adding `--no-enable-prefix-caching` to the server command. The following uses 8×H20-3e(141GB) as an example. ```bash vllm bench serve \ --model tencent/Hy3-preview \ --dataset-name random \ --random-input-len 8192 \ --random-output-len 1024 \ --max-concurrency 32 \ --num-prompts 160 \ --served-model-name hy3-preview ``` Representative output: ```shell ============ Serving Benchmark Result ============ Successful requests: 160 Failed requests: 0 Maximum request concurrency: 32 Benchmark duration (s): 280.58 Total input tokens: 1310720 Total generated tokens: 163840 Request throughput (req/s): 0.57 Output token throughput (tok/s): 583.93 Peak output token throughput (tok/s): 1024.00 Peak concurrent requests: 36.00 Total token throughput (tok/s): 5255.36 ---------------Time to First Token---------------- Mean TTFT (ms): 4542.41 Median TTFT (ms): 2762.17 P99 TTFT (ms): 21062.96 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 50.34 Median TPOT (ms): 51.77 P99 TPOT (ms): 54.07 ---------------Inter-token Latency---------------- Mean ITL (ms): 50.34 Median ITL (ms): 34.32 P99 ITL (ms): 689.10 ================================================== ``` ## References - [Hugging Face model card](https://huggingface.co/tencent/Hy3-preview) - [Hy3-preview-Base (pretrained)](https://huggingface.co/tencent/Hy3-preview-Base) - [GitHub: Tencent-Hunyuan/Hy3-preview](https://github.com/Tencent-Hunyuan/Hy3-preview) - [Tencent Hunyuan](https://hy.tencent.com/research/hy3) --- # tencent/Hy3 on vLLM Source: https://recipes.vllm.ai/tencent/Hy3 Provider: Hy (Tencent) Released: 2026-07-02T13:23:54.000Z Updated: 2026-09-14 Architecture: moe Parameters: 295B Context length: 262,144 Min vLLM: 0.28.0 Tasks: text Verified hardware: GB300 NVL4, MI300X, MI355X Summary: Tencent Hy3 — scaled-up MoE language model (295B total / 21B active) with a 3.8B MTP layer for speculative decoding, 256K context, and hy_v3 tool/reasoning parsers ## Variants - default: precision=bf16, vram_minimum_gb=708 Full precision BF16 — 8×H200 or 8×H20-3e(141GB) minimum for weights + KV cache - fp8: precision=fp8, vram_minimum_gb=354, model_id=tencent/Hy3-FP8 FP8-quantized instruct checkpoint (tencent/Hy3-FP8) — fits a single 8×H200 / 8×H20-3e node with wide headroom. - nvfp4: precision=nvfp4, vram_minimum_gb=214, model_id=RedHatAI/Hy3-NVFP4-FP8 Mixed-precision quant by Red Hat AI — MoE experts in NVFP4, attention in FP8 block. Halves GPU requirement vs BF16 (fits 4×H100 80GB / 4×H200). ## Guide # Hy3 Usage Guide Hy3 is Tencent's latest open-source Mixture-of-Experts language model: 295B total parameters with 21B activated per token, plus a 3.8B MTP layer for speculative decoding. 80 transformer layers, 192 routed experts (top-8) + 1 shared expert, GQA with 64 heads over 8 KV heads, 256K context. ## Availability Hy3 uses the same model code as [Hy3-preview](https://huggingface.co/tencent/Hy3-preview), so any vLLM build that serves Hy3-preview also serves Hy3. vLLM 0.28.0 includes [#47433](https://github.com/vllm-project/vllm/pull/47433), which adds Tencent's HPC-Ops attention and MoE backends. The HPC-Ops kernels are still built and installed separately (see the HPC-Ops backends section below). ## Setup Choose one of the following setup methods. ### Using Docker ```bash # Switch to the trtllm all-reduce backend to work around an mnnvl workspace-size issue. docker run --gpus all \ -p 8000:8000 \ --ipc=host \ -e VLLM_FLASHINFER_ALLREDUCE_BACKEND=trtllm \ -v ~/.cache/huggingface:/root/.cache/huggingface \ vllm/vllm-openai:hy3 tencent/Hy3 \ --tensor-parallel-size 8 \ --tool-call-parser hy_v3 \ --reasoning-parser hy_v3 \ --enable-auto-tool-choice \ --served-model-name hy3 ``` ### Installing from source ```bash uv venv --python 3.12 --seed --managed-python source .venv/bin/activate git clone https://github.com/vllm-project/vllm.git cd vllm uv pip install --editable . --torch-backend=auto ``` ## Model Deployment To serve Hy3 on 8 GPUs, use H20-3e(141GB), H200, AMD MI300X/MI325X (192 GB), AMD MI350X/MI355X (288 GB), or other GPUs with larger memory capacity. Smaller-memory 8-GPU configurations (8×H100 80GB, 8×A100 80GB) do not fit the BF16 weights plus KV cache — use multi-node TP for those. ### Serving on 8×AMD MI300X / MI325X / MI350X / MI355X Serve with the standard launcher plus the AITER environment variables (the recipe's `hardware_overrides.amd.extra_env` applies these automatically when the AMD profile is selected on the recipe site): ```bash export VLLM_ROCM_USE_AITER=1 # MoE disabled (CK GEMM crash); Triton fallback. See troubleshooting note below. export VLLM_ROCM_USE_AITER_MOE=0 export VLLM_ROCM_USE_AITER_MHA=1 export VLLM_ROCM_USE_AITER_RMSNORM=1 export VLLM_ROCM_USE_AITER_LINEAR=1 vllm serve tencent/Hy3 \ --tensor-parallel-size 8 \ --tool-call-parser hy_v3 \ --reasoning-parser hy_v3 \ --enable-auto-tool-choice \ --served-model-name hy3 \ --gpu-memory-utilization 0.90 ``` > **ROCm troubleshooting — AITER fused-MoE.** With `VLLM_ROCM_USE_AITER_MOE=1` > the engine currently aborts during warmup with > `RuntimeError: wrong! device_gemm with the specified compilation parameters > does not support this GEMM problem` (raised from `aiter` `ck_moe_stage2`). > Workaround: set > `VLLM_ROCM_USE_AITER_MOE=0` to fall back to the Triton MoE kernel; the other > AITER backends (MHA / RMSNorm / Linear) run as-is. The docker run needs the > ROCm device flags `--device /dev/kfd --device /dev/dri --group-add video > --group-add render` (the recipe's NVIDIA `docker run` above is CUDA-only). MTP (recommended on AMD for lower latency, same flags as the NVIDIA path): ```bash vllm serve tencent/Hy3 \ --tensor-parallel-size 8 \ --speculative-config.method mtp \ --speculative-config.num_speculative_tokens 2 \ --tool-call-parser hy_v3 \ --reasoning-parser hy_v3 \ --enable-auto-tool-choice \ --served-model-name hy3 \ --gpu-memory-utilization 0.90 ``` ### Serving on 8×H200 or 8×H20-3e(141GB) Without Multi-Token Prediction (MTP): ```bash # Switch to the trtllm all-reduce backend to work around an mnnvl workspace-size issue. export VLLM_FLASHINFER_ALLREDUCE_BACKEND=trtllm vllm serve tencent/Hy3 \ --tensor-parallel-size 8 \ --tool-call-parser hy_v3 \ --reasoning-parser hy_v3 \ --enable-auto-tool-choice \ --served-model-name hy3 ``` With MTP (recommended for lower latency): ```bash # Switch to the trtllm all-reduce backend to work around an mnnvl workspace-size issue. export VLLM_FLASHINFER_ALLREDUCE_BACKEND=trtllm vllm serve tencent/Hy3 \ --tensor-parallel-size 8 \ --speculative-config.method mtp \ --speculative-config.num_speculative_tokens 2 \ --tool-call-parser hy_v3 \ --reasoning-parser hy_v3 \ --enable-auto-tool-choice \ --served-model-name hy3 ``` ### HPC-Ops backends (Hopper) — maximum throughput Tencent's HPC-Ops attention and MoE kernels ([#47433](https://github.com/vllm-project/vllm/pull/47433)) reduce latency for mixed-length decode and small-batch MoE on Hopper GPUs. They ship as a separate wheel — build and install it first: ```bash git clone https://github.com/Tencent/hpc-ops.git cd hpc-ops make wheel python3 -m pip install dist/*.whl ``` Enable the attention backend with `--attention-backend HPC_ATTN`: ```bash # Switch to the trtllm all-reduce backend to work around an mnnvl workspace-size issue. export VLLM_FLASHINFER_ALLREDUCE_BACKEND=trtllm vllm serve tencent/Hy3 \ --tensor-parallel-size 8 \ --attention-backend HPC_ATTN ``` Enable the fused MoE backend with `--moe-backend hpc`. On the FP8 checkpoint, add the FP8 KV-cache flags `--kv-cache-dtype fp8_e4m3 --block-size 64`: ```bash # Switch to the trtllm all-reduce backend to work around an mnnvl workspace-size issue. export VLLM_FLASHINFER_ALLREDUCE_BACKEND=trtllm vllm serve tencent/Hy3-FP8 \ --tensor-parallel-size 8 \ --moe-backend hpc \ --kv-cache-dtype fp8_e4m3 \ --block-size 64 ``` Both backends combine with the standard `--tool-call-parser` / `--reasoning-parser` and MTP flags shown above. See the [vLLM × HPC-Ops post](https://vllm.ai/blog/2026-07-06-vllm-hpc-ops) for benchmarks and details. ## Sampling and Reasoning Modes Tencent's recommended sampling parameters: `temperature=0.9`, `top_p=1.0`. Reasoning is controlled via `chat_template_kwargs.reasoning_effort`: | Value | Behavior | |:---|:---| | `no_think` (default) | Direct response, no chain-of-thought | | `low` | Light reasoning | | `high` | Deep chain-of-thought for math/coding/complex reasoning | When tools are registered, set `interleaved_thinking: true` to allow the model to think between tool calls. ### OpenAI Client Example ```bash uv pip install -U openai ``` ```python from openai import OpenAI client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY") messages = [ {"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Hello."}, ] # Direct response (default). resp = client.chat.completions.create( model="hy3", messages=messages, temperature=0.9, top_p=1.0, max_tokens=4096, ) print(resp.choices[0].message.content) # Deep reasoning: set reasoning_effort (and interleaved_thinking if using tools). resp_think = client.chat.completions.create( model="hy3", messages=messages, temperature=0.9, top_p=1.0, max_tokens=4096, extra_body={ "chat_template_kwargs": { "reasoning_effort": "high", "interleaved_thinking": True, }, }, ) output_msg = resp_think.choices[0].message print(output_msg.reasoning) # chain-of-thought print(output_msg.content) # final answer ``` ### cURL Usage ```bash curl http://localhost:8000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "hy3", "messages": [ {"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Hello."} ], "temperature": 0.9, "top_p": 1.0, "max_tokens": 4096 }' ``` ## Benchmarking For benchmarking, disable prefix caching by adding `--no-enable-prefix-caching` to the server command. The following benchmarks the FP8 checkpoint (8192→1024, concurrency 32, 160 prompts): ```bash vllm bench serve \ --model tencent/Hy3-FP8 \ --dataset-name random \ --random-input-len 8192 \ --random-output-len 1024 \ --max-concurrency 32 \ --num-prompts 160 \ --served-model-name hy3-fp8 ``` Representative output — Hy3-FP8, TP4 on 4×GB300 (one NVL4 tray), trtllm all-reduce, MTP=2: ```shell ============ Serving Benchmark Result ============ Successful requests: 160 Failed requests: 0 Maximum request concurrency: 32 Benchmark duration (s): 175.37 Request throughput (req/s): 0.91 Output token throughput (tok/s): 934.26 Peak output token throughput (tok/s): 1344.00 Total token throughput (tok/s): 8408.34 Mean TTFT (ms): 2352.29 Median TTFT (ms): 505.64 P99 TTFT (ms): 14538.85 Mean TPOT (ms): 30.11 Median TPOT (ms): 30.86 P99 TPOT (ms): 40.48 Mean ITL (ms): 36.26 Median ITL (ms): 24.09 ================================================== ``` ## References - [Hugging Face model card](https://huggingface.co/tencent/Hy3) - [FP8 checkpoint — tencent/Hy3-FP8](https://huggingface.co/tencent/Hy3-FP8) - [NVFP4-FP8 checkpoint — RedHatAI/Hy3-NVFP4-FP8](https://huggingface.co/RedHatAI/Hy3-NVFP4-FP8) - [GitHub: Tencent-Hunyuan/Hy3](https://github.com/Tencent-Hunyuan/Hy3) - [vLLM × HPC-Ops: high-performance attention & MoE backends](https://vllm.ai/blog/2026-07-06-vllm-hpc-ops) - [Tencent Hy3](https://hy.tencent.com/research/hy3) --- # zai-org/GLM-4.5 on vLLM Source: https://recipes.vllm.ai/zai-org/GLM-4.5 Provider: GLM (Z-AI) Released: 2025-07-20T03:25:36.000Z Updated: 2026-09-14 Architecture: moe Parameters: 358B Context length: 131,072 Min vLLM: 0.11.0 Tasks: text Verified hardware: H200, MI300X, MI325X, MI355X Summary: GLM-4.5 MoE language model (~358B total parameters, BF16) with built-in MTP layers for speculative decoding and native tool calling ## Variants - default: precision=bf16, vram_minimum_gb=859 Full precision BF16 on 8xH200 or equivalent - fp8: precision=fp8, vram_minimum_gb=430, model_id=zai-org/GLM-4.5-FP8 Native FP8 checkpoint with minimal accuracy loss — recommended for cost-efficient serving ## Recommended base flags ```bash vllm serve 'zai-org/GLM-4.5' \ --trust-remote-code ``` ## Guide ## Overview GLM-4.5 is a Mixture-of-Experts language model from Z-AI with ~358B total parameters. The checkpoint ships in both BF16 and native FP8 formats. FP8 models have minimal accuracy loss, so unless you need strict reproducibility for benchmarking, FP8 is the recommended precision for lower-cost serving. All GLM-4.X models include built-in Multi-Token Prediction (MTP) layers that enable speculative decoding for higher generation throughput. ## Prerequisites - **vLLM version:** >= 0.11.0 (latest stable recommended) - **Hardware:** 8x H200 (BF16) or 4x-8x H200 (FP8), AMD MI300X / MI325X / MI355X for ROCm - **Python:** 3.10 - 3.13 (3.12 required for ROCm wheels) ### Install vLLM (NVIDIA) ```bash uv venv source .venv/bin/activate uv pip install -U vllm --torch-backend auto ``` ### Install vLLM (AMD ROCm) ```bash uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm ``` The vLLM wheel for ROCm requires Python 3.12, ROCm 7.0, and glibc >= 2.35. ## Launching the Server ### Tensor Parallel (FP8 on 8 GPUs) ```bash vllm serve zai-org/GLM-4.5-FP8 \ --tensor-parallel-size 8 \ --tool-call-parser glm45 \ --reasoning-parser glm45 \ --enable-auto-tool-choice ``` ### Enabling MTP Speculative Decoding ```bash vllm serve zai-org/GLM-4.5-FP8 \ --tensor-parallel-size 4 \ --speculative-config.method mtp \ --speculative-config.num_speculative_tokens 1 \ --tool-call-parser glm45 \ --reasoning-parser glm45 \ --enable-auto-tool-choice ``` Use `--speculative-config.num_speculative_tokens 1` for optimal throughput. Higher values increase mean acceptance length but drop acceptance rate significantly. ### Tuning Tips - `--max-model-len=65536` works well for most scenarios; max is 128K. - `--max-num-batched-tokens=32768` is a good default for prompt-heavy workloads. Reduce to 16K/8K to cut activation memory and latency. - Set `--gpu-memory-utilization=0.95` to maximize KV cache headroom. ## Client Usage ```python from openai import OpenAI client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1") resp = client.chat.completions.create( model="zai-org/GLM-4.5-FP8", messages=[{"role": "user", "content": "Explain MTP speculative decoding."}], max_tokens=512, ) print(resp.choices[0].message.content) ``` ## Benchmarking Disable prefix caching with `--no-enable-prefix-caching` on the server command, then: ```bash vllm bench serve \ --model zai-org/GLM-4.5-FP8 \ --dataset-name random \ --random-input-len 8000 \ --random-output-len 1000 \ --request-rate 10000 \ --num-prompts 16 \ --ignore-eos ``` ## Troubleshooting - **Tool calling not firing:** Ensure `--tool-call-parser glm45 --enable-auto-tool-choice` are both present. - **MTP memory overhead:** MTP adds memory for draft computations. Monitor GPU memory and reduce `--max-model-len` or `--max-num-batched-tokens` if you OOM. - **Low MTP acceptance:** If acceptance rate is below ~90%, drop speculative tokens to 1. ## References - [Model card](https://huggingface.co/zai-org/GLM-4.5) - [FP8 checkpoint](https://huggingface.co/zai-org/GLM-4.5-FP8) - [vLLM docs](https://docs.vllm.ai/) --- # zai-org/GLM-4.6 on vLLM Source: https://recipes.vllm.ai/zai-org/GLM-4.6 Provider: GLM (Z-AI) Released: 2025-09-29T18:22:51.000Z Updated: 2026-09-14 Architecture: moe Parameters: 357B Context length: 202,752 Min vLLM: 0.11.0 Tasks: text Verified hardware: H200, MI300X, MI325X, MI355X Summary: GLM-4.6 MoE language model (~357B total parameters, BF16) with MTP speculative decoding, native tool calling and reasoning ## Variants - default: precision=bf16, vram_minimum_gb=857 Full precision BF16 on 8xH200 or equivalent - fp8: precision=fp8, vram_minimum_gb=428, model_id=zai-org/GLM-4.6-FP8 Native FP8 checkpoint with minimal accuracy loss — recommended for cost-efficient serving - redhat_fp8: precision=fp8, vram_minimum_gb=429, model_id=RedHatAI/GLM-4.6-FP8-dynamic RedHatAI FP8 (E4M3) with dynamic per-token activation quantization - redhat_fp8_plain: precision=fp8, vram_minimum_gb=429, model_id=RedHatAI/GLM-4.6-FP8 RedHatAI FP8 quantization - nvfp4: precision=nvfp4, vram_minimum_gb=223, model_id=RedHatAI/GLM-4.6-NVFP4 RedHatAI NVFP4 quantization ## Recommended base flags ```bash vllm serve 'zai-org/GLM-4.6' \ --trust-remote-code ``` ## Guide ## Overview GLM-4.6 is the successor to GLM-4.5 with ~357B total parameters. It retains the MoE architecture and built-in Multi-Token Prediction (MTP) layers used for speculative decoding. FP8 is the recommended precision for cost-efficient serving with minimal accuracy loss relative to BF16. ## Prerequisites - **vLLM version:** >= 0.11.0 (latest stable recommended) - **Hardware:** 8x H200 (BF16) or 4x-8x H200 (FP8), AMD MI300X / MI325X / MI355X for ROCm - **Python:** 3.10 - 3.13 (3.12 required for ROCm wheels) ### Install vLLM (NVIDIA) ```bash uv venv source .venv/bin/activate uv pip install -U vllm --torch-backend auto ``` ### Install vLLM (AMD ROCm) ```bash uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm ``` ## Launching the Server ### Tensor Parallel (FP8) ```bash vllm serve zai-org/GLM-4.6-FP8 \ --tensor-parallel-size 8 \ --tool-call-parser glm45 \ --reasoning-parser glm45 \ --enable-auto-tool-choice ``` ### Enabling MTP Speculative Decoding ```bash vllm serve zai-org/GLM-4.6-FP8 \ --tensor-parallel-size 4 \ --speculative-config.method mtp \ --speculative-config.num_speculative_tokens 1 \ --tool-call-parser glm45 \ --reasoning-parser glm45 \ --enable-auto-tool-choice ``` ### Tuning Tips - `--max-model-len=65536` works well for most scenarios; max is 128K. - `--max-num-batched-tokens=32768` is a good default for prompt-heavy workloads. - `--gpu-memory-utilization=0.95` maximizes KV cache headroom. ## Client Usage ```python from openai import OpenAI client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1") resp = client.chat.completions.create( model="zai-org/GLM-4.6-FP8", messages=[{"role": "user", "content": "Summarize MTP speculative decoding."}], max_tokens=512, ) print(resp.choices[0].message.content) ``` ## Benchmarking ```bash vllm bench serve \ --model zai-org/GLM-4.6-FP8 \ --dataset-name random \ --random-input-len 8000 \ --random-output-len 1000 \ --request-rate 10000 \ --num-prompts 16 \ --ignore-eos ``` ## Troubleshooting - **MTP memory overhead:** Monitor GPU memory and tune batch size when enabling MTP. - **Tool calling not firing:** Ensure `--tool-call-parser glm45 --enable-auto-tool-choice` are both present. ## References - [Model card](https://huggingface.co/zai-org/GLM-4.6) - [FP8 checkpoint](https://huggingface.co/zai-org/GLM-4.6-FP8) - [vLLM docs](https://docs.vllm.ai/) --- # pfnet/plamo-3-nict-31b-base on vLLM Source: https://recipes.vllm.ai/pfnet/plamo-3-nict-31b-base Provider: Preferred Networks Released: 2025-10-24T07:30:37.000Z Updated: 2026-09-13 Architecture: dense Parameters: 32B Context length: 4,096 Min vLLM: 0.12.0 Tasks: text Summary: Largest PLaMo 3 NICT Japanese/English base model with interleaved sliding-window and full attention. ## Variants - default: precision=bf16, vram_minimum_gb=77 BF16 base checkpoint. Fits on H100/H200-class GPUs for 4K context; use TP=2 if extra cache headroom is needed. ## Recommended base flags ```bash vllm serve 'pfnet/plamo-3-nict-31b-base' \ --trust-remote-code \ --max-model-len \ 4096 ``` ## Guide ## Overview [PLaMo 3 NICT 31B Base](https://huggingface.co/pfnet/plamo-3-nict-31b-base) is the largest PLaMo 3 NICT base model from Preferred Networks and NICT. It is pretrained on Japanese and English data with a hybrid attention stack combining sliding-window and full-attention layers. This is a base checkpoint. Use completion-style prompts as-is, or fine-tune it before deploying instruction-following or chat-style workflows. ## Prerequisites - **Hardware:** H100/H200 80 GB for 4K context; use TP=2 for extra KV-cache headroom - **vLLM:** >= 0.12.0 - **License:** gated Hugging Face access under the PLaMo community license ### Install vLLM ```bash uv venv source .venv/bin/activate uv pip install -U vllm --torch-backend=auto hf auth login ``` ## Launching the Server ```bash vllm serve pfnet/plamo-3-nict-31b-base \ --trust-remote-code \ --max-model-len 4096 ``` With tensor parallelism for more KV-cache headroom: ```bash vllm serve pfnet/plamo-3-nict-31b-base \ --trust-remote-code \ --max-model-len 4096 \ --tensor-parallel-size 2 ``` ## Client Usage ```python from openai import OpenAI client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1") resp = client.completions.create( model="pfnet/plamo-3-nict-31b-base", prompt="The future of artificial intelligence technology is ", max_tokens=128, temperature=0.7, ) print(resp.choices[0].text) ``` ## Troubleshooting - **Out of memory on 80 GB GPUs:** add `--tensor-parallel-size 2` or lower `--max-model-len`. - **Gated repo errors:** accept the PLaMo community license on Hugging Face and run `hf auth login`. - **Repetitive or low-quality completions:** adjust sampling for free-form generation, or fine-tune the model for chat or assistant workloads. ## References - [Model card](https://huggingface.co/pfnet/plamo-3-nict-31b-base) - [vLLM PLaMo3 support](https://docs.vllm.ai/en/stable/models/supported_models.html) --- # Google/gemma-4-E2B-it on vLLM Source: https://recipes.vllm.ai/Google/gemma-4-E2B-it Provider: Google Released: 2026-03-02T19:58:09.000Z Updated: 2026-09-12 Architecture: dense Parameters: 5B Context length: 131,072 Min vLLM: 0.19.1 Tasks: multimodal, text Verified hardware: H100, MI300X, MI325X, MI355X, TPU v6e (Trillium), TPU v7 (Ironwood), Xeon 6 Summary: Google's compact Gemma 4 multimodal model (effective 2B) with native text, image, and audio, plus thinking mode and tool-use protocol. ## Variants - default: precision=bf16, vram_minimum_gb=13 Full BF16 — single 24 GB+ NVIDIA GPU or 1x MI300X/MI325X/MI350X/MI355X or 1x Xeon6/Xeon5 NUMA node - fp8: precision=fp8, vram_minimum_gb=6, model_id=RedHatAI/gemma-4-E2B-it-FP8-dynamic FP8 (E4M3) linear weights with dynamic per-token activation quantization (vision/audio encoders stay BF16) — Hopper or Blackwell - w4a16: precision=int4, vram_minimum_gb=8, model_id=google/gemma-4-E2B-it-qat-w4a16-ct W4A16 QAT (4-bit weights, 16-bit activations, group_size=32) via compressed-tensors — runs on any GPU - mobile: precision=int2/4/8, vram_minimum_gb=4, model_id=google/gemma-4-E2B-it-qat-mobile-ct Mobile QAT — mixed int2/4/8 compressed-tensors (int2 embeddings + lm_head + deep MLP, int4 attention + early MLP, int8 vision + gates). 3.6× weight compression vs BF16, up to 1.25× faster decode at low batch ## Guide ## Overview [Gemma 4 E2B](https://huggingface.co/google/gemma-4-E2B-it) is the smallest member of Google's Gemma 4 family — an effective-2B unified multimodal model that natively processes text, images, and audio, with structured thinking/reasoning, function calling, and dynamic vision resolution. It runs comfortably on a single 24 GB+ GPU. Intel Xeon 6 CPU support remains verified. No additional benchmark-specific CPU tuning is prescribed for general deployment. ### Key Features - **Multimodal**: Text + images + audio natively (video via custom frame-extraction pipeline). - **Dual Attention**: Alternating sliding-window (local) and global attention with different head dimensions. - **Thinking Mode**: Structured reasoning via `<|channel>thought\n...` delimiters. - **Function Calling**: Custom tool-call protocol with dedicated special tokens. - **Dynamic Vision Resolution**: Per-request configurable vision token budget (70, 140, 280, 560, 1120 tokens). TPU support is provided through [vLLM TPU](https://github.com/vllm-project/tpu-inference) with recipes for [Trillium](https://github.com/AI-Hypercomputer/tpu-recipes/tree/main/inference/trillium/vLLM/Gemma4) and [Ironwood](https://github.com/AI-Hypercomputer/tpu-recipes/blob/main/inference/ironwood/vLLM/Gemma4/). ## Prerequisites - Hardware: Intel Xeon 6/Xeon 5 CPUs - vLLM >= 0.19.1 ### pip (NVIDIA CUDA) ```bash uv venv source .venv/bin/activate uv pip install -U vllm --pre \ --extra-index-url https://wheels.vllm.ai/nightly/cu129 \ --extra-index-url https://download.pytorch.org/whl/cu129 \ --index-strategy unsafe-best-match ``` ### pip (AMD ROCm: MI300X, MI325X, MI350X, MI355X) Requires Python 3.12, ROCm 7.2.1, glibc >= 2.35 (Ubuntu 22.04+). ```bash uv venv --python 3.12 source .venv/bin/activate uv pip install vllm --pre \ --extra-index-url https://wheels.vllm.ai/rocm/nightly/rocm721 --upgrade ``` ### pip (Intel Xeon 6 CPUs) For Intel and AMD x86 CPUs, follow the [CPU pre-built wheels](https://docs.vllm.ai/en/latest/getting_started/installation/cpu/#pre-built-wheels) installation instructions. ### Docker ```bash docker pull vllm/vllm-openai:gemma4-0505-cu129 # NVIDIA Hopper (H100/H200, CUDA 12.9) docker pull vllm/vllm-openai:gemma4-0505-cu130 # NVIDIA Blackwell (B200/B300, CUDA 13.0) docker pull vllm/vllm-openai-rocm:latest # AMD docker pull vllm/vllm-openai-cpu:latest-x86_64 # For Intel Xeon 6 ``` ## Deployment Configurations ### Quick Start (Single GPU) ```bash vllm serve google/gemma-4-E2B-it \ --max-model-len 32768 ``` ### With Audio Support ```bash vllm serve google/gemma-4-E2B-it \ --max-model-len 8192 \ --limit-mm-per-prompt '{"image": 4, "audio": 1}' ``` ### Full-Featured Server Launch Enables text, image, audio, thinking, and tool calling: ```bash vllm serve google/gemma-4-E2B-it \ --max-model-len 16384 \ --gpu-memory-utilization 0.90 \ --enable-auto-tool-choice \ --reasoning-parser gemma4 \ --tool-call-parser gemma4 \ --chat-template examples/tool_chat_template_gemma4.jinja \ --limit-mm-per-prompt '{"image": 4, "audio": 1}' \ --async-scheduling \ --host 0.0.0.0 \ --port 8000 ``` ### Docker (NVIDIA) ```bash docker run -itd --name gemma4-e2b \ --ipc=host --network host --shm-size 16G --gpus all \ -v ~/.cache/huggingface:/root/.cache/huggingface \ vllm/vllm-openai:gemma4-0505-cu129 \ --model google/gemma-4-E2B-it \ --max-model-len 32768 \ --host 0.0.0.0 --port 8000 ``` Swap `vllm/vllm-openai:gemma4-0505-cu129` for `vllm/vllm-openai:gemma4-0505-cu130` on Blackwell (B200/B300). ### Docker (AMD MI300X/MI325X/MI350X/MI355X) ```bash docker run -itd --name gemma4-rocm \ --ipc=host --network=host --privileged \ --cap-add=CAP_SYS_ADMIN --device=/dev/kfd --device=/dev/dri \ --group-add=video --cap-add=SYS_PTRACE \ --security-opt=seccomp=unconfined --shm-size 16G \ -v ~/.cache/huggingface:/root/.cache/huggingface \ vllm/vllm-openai-rocm:latest \ --model google/gemma-4-E2B-it \ --host 0.0.0.0 --port 8000 ``` ### Docker (Cloud TPU — Trillium / Ironwood) TPU uses the separate `vllm/vllm-tpu` image (no pip wheel). Pull the tag specified by the upstream [Trillium](https://github.com/AI-Hypercomputer/tpu-recipes/tree/main/inference/trillium/vLLM/Gemma4) or [Ironwood](https://github.com/AI-Hypercomputer/tpu-recipes/blob/main/inference/ironwood/vLLM/Gemma4/) recipe, then run: ```bash docker run -itd --name gemma4-tpu \ --privileged --network host --shm-size 16G \ -v /dev/shm:/dev/shm -e HF_TOKEN=$HF_TOKEN \ vllm/vllm-tpu:latest \ --model google/gemma-4-E2B-it \ --max-model-len 16384 \ --disable_chunked_mm_input \ --host 0.0.0.0 --port 8000 ``` ### Intel Xeon 6 Deployment via Docker Launch the x86 CPU vLLM Docker container for `google/gemma-4-E2B-it`: ```bash docker run \ --privileged --ipc=host -p 8000:8000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ vllm/vllm-openai-cpu:latest-x86_64 google/gemma-4-E2B-it \ --tensor-parallel-size 1 \ --enable-auto-tool-choice \ --tool-call-parser gemma4 \ --chat-template examples/tool_chat_template_gemma4.jinja \ --reasoning-parser gemma4 ``` The following settings are intentionally not prescribed as portable CPU model defaults because they depend on the workload, hardware, or runtime environment: - `--max-num-batched-tokens`: scheduler/throughput tuning. - `--max-num-seqs`: concurrency and scheduler-capacity tuning. - `--gpu-memory-utilization`: platform memory-budget tuning. - `--no-enable-prefix-caching`: workload/benchmark cache-policy tuning. - `VLLM_ENGINE_ITERATION_TIMEOUT_S`: operational runtime timeout. These settings may still appear in validated hardware-specific overrides. Tune them at deployment time based on platform resources, workload shape, and latency/throughput goals. ## Client Usage ### Audio Transcription ```python from openai import OpenAI client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY") response = client.chat.completions.create( model="google/gemma-4-E2B-it", messages=[{"role": "user", "content": [ {"type": "audio_url", "audio_url": {"url": "https://upload.wikimedia.org/wikipedia/commons/2/22/Beatbox_by_Wikipedia_user_Wikipedia_Brown.ogg"}}, {"type": "text", "text": "Provide a verbatim, word-for-word transcription of the audio."}, ]}], max_tokens=512, ) print(response.choices[0].message.content) ``` ### Image Understanding ```python response = client.chat.completions.create( model="google/gemma-4-E2B-it", messages=[{"role": "user", "content": [ {"type": "image_url", "image_url": {"url": "https://upload.wikimedia.org/wikipedia/commons/thumb/3/3a/Cat03.jpg/1200px-Cat03.jpg"}}, {"type": "text", "text": "Describe this image in detail."}, ]}], max_tokens=1024, ) ``` ### Thinking Mode Launch with reasoning parser, then enable per-request: ```bash vllm serve google/gemma-4-E2B-it \ --max-model-len 16384 \ --reasoning-parser gemma4 \ --tool-call-parser gemma4 \ --enable-auto-tool-choice \ --chat-template examples/tool_chat_template_gemma4.jinja ``` Enable per-request via `extra_body={"chat_template_kwargs": {"enable_thinking": True}}`. ## Configuration Tips - Set `--max-model-len` to match your workload (max 131072). - Image-only workloads: `--limit-mm-per-prompt.audio 0`. - Text-only workloads: `--limit-mm-per-prompt '{"image": 0, "audio": 0}'` to skip MM profiling. - `--async-scheduling` improves throughput. - FP8 KV cache (`--kv-cache-dtype fp8`) saves ~50% KV memory. ## Quantized Variant [`RedHatAI/gemma-4-E2B-it-FP8-dynamic`](https://huggingface.co/RedHatAI/gemma-4-E2B-it-FP8-dynamic) is a pre-quantized FP8 (E4M3) checkpoint — linear weights with dynamic per-token activation quantization, vision/audio encoders kept in BF16. Runs on Hopper and Blackwell. Pick the **fp8** variant above, or pass the repo id directly to `vllm serve`. ## Speculative Decoding (MTP) Enable the **Spec Decoding** feature toggle (above) or add `--speculative-config` manually to use MTP drafting with the [assistant model](https://huggingface.co/google/gemma-4-E2B-it-assistant). Recommended `num_speculative_tokens`: 2 for this model. The E2B assistant uses centroids masking for efficient sparse logit computation. See the [Gemma 4 usage guide](../../Google/Gemma4) for details and benchmarks. > **Note:** MTP speculative decoding for Gemma 4 is only available on the vLLM nightly build — it has not yet landed in a stable release. Install via the nightly wheel (`uv pip install -U vllm --pre --extra-index-url https://wheels.vllm.ai/nightly/cu129 …`) or use the `vllm/vllm-openai:gemma4-0505-cu129` / `vllm/vllm-openai:gemma4-0505-cu130` images above; the standard `:latest` stable tag does not include this feature. ## References - [Model card](https://huggingface.co/google/gemma-4-E2B-it) - [FP8 variant](https://huggingface.co/RedHatAI/gemma-4-E2B-it-FP8-dynamic) - [Gemma docs](https://ai.google.dev/gemma/docs) - [vLLM Gemma 4 tool-call template](https://github.com/vllm-project/vllm/blob/main/examples/tool_chat_template_gemma4.jinja) - [TPU recipes: Trillium](https://github.com/AI-Hypercomputer/tpu-recipes/tree/main/inference/trillium/vLLM/Gemma4) - [TPU recipes: Ironwood](https://github.com/AI-Hypercomputer/tpu-recipes/blob/main/inference/ironwood/vLLM/Gemma4/) --- # Google/gemma-4-E4B-it on vLLM Source: https://recipes.vllm.ai/Google/gemma-4-E4B-it Provider: Google Released: 2026-03-02T19:57:40.000Z Updated: 2026-09-12 Architecture: dense Parameters: 8B Context length: 131,072 Min vLLM: 0.19.1 Tasks: multimodal, text Verified hardware: H100, MI300X, MI325X, MI355X, TPU v6e (Trillium), TPU v7 (Ironwood), Xeon 6 Summary: Google's compact Gemma 4 multimodal model (effective 4B) with native text, image, and audio, plus thinking mode and tool-use protocol. ## Variants - default: precision=bf16, vram_minimum_gb=20 Full BF16 — single 24 GB+ NVIDIA GPU or 1x MI300X/MI325X/MI350X/MI355Xi or 2x Xeon6/Xeon5 NUMA nodes - fp8: precision=fp8, vram_minimum_gb=10, model_id=RedHatAI/gemma-4-E4B-it-FP8-dynamic FP8 (E4M3) linear weights with dynamic per-token activation quantization (vision/audio encoders stay BF16) — Hopper or Blackwell - w4a16: precision=int4, vram_minimum_gb=10, model_id=google/gemma-4-E4B-it-qat-w4a16-ct W4A16 QAT (4-bit weights, 16-bit activations, group_size=32) via compressed-tensors — runs on any GPU - mobile: precision=int2/4/8, vram_minimum_gb=5, model_id=google/gemma-4-E4B-it-qat-mobile-ct Mobile QAT — mixed int2/4/8 compressed-tensors (int2 embeddings + lm_head + audio, int4 LLM attention + MLP, int8 vision + gates). 4.1× weight compression vs BF16, up to 1.5× faster decode at low batch ## Guide ## Overview [Gemma 4 E4B](https://huggingface.co/google/gemma-4-E4B-it) is Google's effective-4B unified multimodal model — text + images + audio in a single model, with structured thinking/reasoning, function calling, and dynamic vision resolution. It fits on a single 24 GB+ GPU. Intel Xeon 6 CPU support remains verified. No additional benchmark-specific CPU tuning is prescribed for general deployment. ### Key Features - **Multimodal**: Text + images + audio natively (video via custom frame-extraction pipeline). - **Dual Attention**: Alternating sliding-window (local) and global attention with different head dimensions. - **Thinking Mode**: Structured reasoning via `<|channel>thought\n...` delimiters. - **Function Calling**: Custom tool-call protocol with dedicated special tokens. - **Dynamic Vision Resolution**: Per-request configurable vision token budget (70, 140, 280, 560, 1120 tokens). TPU support is provided through [vLLM TPU](https://github.com/vllm-project/tpu-inference) with recipes for [Trillium](https://github.com/AI-Hypercomputer/tpu-recipes/tree/main/inference/trillium/vLLM/Gemma4) and [Ironwood](https://github.com/AI-Hypercomputer/tpu-recipes/blob/main/inference/ironwood/vLLM/Gemma4/). ## Prerequisites - Hardware: Intel Xeon 6/Xeon 5 CPUs - vLLM >= 0.19.1 ### pip (NVIDIA CUDA) ```bash uv venv source .venv/bin/activate uv pip install -U vllm --pre \ --extra-index-url https://wheels.vllm.ai/nightly/cu129 \ --extra-index-url https://download.pytorch.org/whl/cu129 \ --index-strategy unsafe-best-match ``` ### pip (AMD ROCm: MI300X, MI325X, MI350X, MI355X) Requires Python 3.12, ROCm 7.2.1, glibc >= 2.35 (Ubuntu 22.04+). ```bash uv venv --python 3.12 source .venv/bin/activate uv pip install vllm --pre \ --extra-index-url https://wheels.vllm.ai/rocm/nightly/rocm721 --upgrade ``` ### pip (Intel Xeon 6 CPUs) For Intel and AMD x86 CPUs, follow the [CPU pre-built wheels](https://docs.vllm.ai/en/latest/getting_started/installation/cpu/#pre-built-wheels) installation instructions. ### Docker ```bash docker pull vllm/vllm-openai:gemma4-0505-cu129 # NVIDIA Hopper (H100/H200, CUDA 12.9) docker pull vllm/vllm-openai:gemma4-0505-cu130 # NVIDIA Blackwell (B200/B300, CUDA 13.0) docker pull vllm/vllm-openai-rocm:latest # AMD docker pull vllm/vllm-openai-cpu:latest-x86_64 # For Intel Xeon 6 ``` ## Deployment Configurations ### Quick Start (Single GPU) ```bash vllm serve google/gemma-4-E4B-it \ --max-model-len 32768 ``` ### With Audio Support ```bash vllm serve google/gemma-4-E4B-it \ --max-model-len 8192 \ --limit-mm-per-prompt '{"image": 4, "audio": 1}' ``` ### Full-Featured Server Launch Enables text, image, audio, thinking, and tool calling: ```bash vllm serve google/gemma-4-E4B-it \ --max-model-len 16384 \ --gpu-memory-utilization 0.90 \ --enable-auto-tool-choice \ --reasoning-parser gemma4 \ --tool-call-parser gemma4 \ --chat-template examples/tool_chat_template_gemma4.jinja \ --limit-mm-per-prompt '{"image": 4, "audio": 1}' \ --async-scheduling \ --host 0.0.0.0 \ --port 8000 ``` ### Docker (NVIDIA) ```bash docker run -itd --name gemma4-e4b \ --ipc=host --network host --shm-size 16G --gpus all \ -v ~/.cache/huggingface:/root/.cache/huggingface \ vllm/vllm-openai:gemma4-0505-cu129 \ --model google/gemma-4-E4B-it \ --max-model-len 32768 \ --host 0.0.0.0 --port 8000 ``` Swap `vllm/vllm-openai:gemma4-0505-cu129` for `vllm/vllm-openai:gemma4-0505-cu130` on Blackwell (B200/B300). ### Docker (AMD MI300X/MI325X/MI350X/MI355X) ```bash docker run -itd --name gemma4-rocm \ --ipc=host --network=host --privileged \ --cap-add=CAP_SYS_ADMIN --device=/dev/kfd --device=/dev/dri \ --group-add=video --cap-add=SYS_PTRACE \ --security-opt=seccomp=unconfined --shm-size 16G \ -v ~/.cache/huggingface:/root/.cache/huggingface \ vllm/vllm-openai-rocm:latest \ --model google/gemma-4-E4B-it \ --host 0.0.0.0 --port 8000 ``` ### Docker (Cloud TPU — Trillium / Ironwood) TPU uses the separate `vllm/vllm-tpu` image (no pip wheel). Pull the tag specified by the upstream [Trillium](https://github.com/AI-Hypercomputer/tpu-recipes/tree/main/inference/trillium/vLLM/Gemma4) or [Ironwood](https://github.com/AI-Hypercomputer/tpu-recipes/blob/main/inference/ironwood/vLLM/Gemma4/) recipe, then run: ```bash docker run -itd --name gemma4-tpu \ --privileged --network host --shm-size 16G \ -v /dev/shm:/dev/shm -e HF_TOKEN=$HF_TOKEN \ vllm/vllm-tpu:latest \ --model google/gemma-4-E4B-it \ --max-model-len 16384 \ --disable_chunked_mm_input \ --host 0.0.0.0 --port 8000 ``` ### Intel Xeon 6 Deployment via Docker Launch the x86 CPU vLLM Docker container for `google/gemma-4-E4B-it`: ```bash docker run \ --privileged --ipc=host -p 8000:8000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ vllm/vllm-openai-cpu:latest-x86_64 google/gemma-4-E4B-it \ --tensor-parallel-size 1 \ --enable-auto-tool-choice \ --tool-call-parser gemma4 \ --chat-template examples/tool_chat_template_gemma4.jinja \ --reasoning-parser gemma4 ``` For additional Intel Xeon 6 deployment details, see the Intel Software Catalog entries for [Gemma 4 E4B IT](https://aiswcatalog.intel.com/models/google-gemma-4-e4b-it). The following settings are intentionally not prescribed as portable CPU model defaults because they depend on the workload, hardware, or runtime environment: - `--max-num-batched-tokens`: scheduler/throughput tuning. - `--max-num-seqs`: concurrency and scheduler-capacity tuning. - `--gpu-memory-utilization`: platform memory-budget tuning. - `--no-enable-prefix-caching`: workload/benchmark cache-policy tuning. - `VLLM_ENGINE_ITERATION_TIMEOUT_S`: operational runtime timeout. These settings may still appear in validated hardware-specific overrides. Tune them at deployment time based on platform resources, workload shape, and latency/throughput goals. ## Client Usage ### Audio Transcription ```python from openai import OpenAI client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY") response = client.chat.completions.create( model="google/gemma-4-E4B-it", messages=[{"role": "user", "content": [ {"type": "audio_url", "audio_url": {"url": "https://example.com/audio.wav"}}, {"type": "text", "text": "Transcribe this audio."}, ]}], max_tokens=512, ) print(response.choices[0].message.content) ``` ### Image Understanding ```python response = client.chat.completions.create( model="google/gemma-4-E4B-it", messages=[{"role": "user", "content": [ {"type": "image_url", "image_url": {"url": "https://upload.wikimedia.org/wikipedia/commons/thumb/3/3a/Cat03.jpg/1200px-Cat03.jpg"}}, {"type": "text", "text": "Describe this image in detail."}, ]}], max_tokens=1024, ) ``` ### Thinking Mode ```bash vllm serve google/gemma-4-E4B-it \ --max-model-len 16384 \ --reasoning-parser gemma4 \ --tool-call-parser gemma4 \ --enable-auto-tool-choice \ --chat-template examples/tool_chat_template_gemma4.jinja ``` Enable per-request via `extra_body={"chat_template_kwargs": {"enable_thinking": True}}`. ## Configuration Tips - Set `--max-model-len` to match your workload (max 131072). - Image-only workloads: `--limit-mm-per-prompt.audio 0`. - Text-only workloads: `--limit-mm-per-prompt '{"image": 0, "audio": 0}'` to skip MM profiling. - `--async-scheduling` improves throughput. - FP8 KV cache (`--kv-cache-dtype fp8`) saves ~50% KV memory. ## Quantized Variant [`RedHatAI/gemma-4-E4B-it-FP8-dynamic`](https://huggingface.co/RedHatAI/gemma-4-E4B-it-FP8-dynamic) is a pre-quantized FP8 (E4M3) checkpoint — linear weights with dynamic per-token activation quantization, vision/audio encoders kept in BF16. Runs on Hopper and Blackwell. Pick the **fp8** variant above, or pass the repo id directly to `vllm serve`. ## Speculative Decoding (MTP) Enable the **Spec Decoding** feature toggle (above) or add `--speculative-config` manually to use MTP drafting with the [assistant model](https://huggingface.co/google/gemma-4-E4B-it-assistant). Recommended `num_speculative_tokens`: 4 for this model. The E4B assistant uses centroids masking for efficient sparse logit computation. See the [Gemma 4 usage guide](../../Google/Gemma4) for details and benchmarks. > **Note:** MTP speculative decoding for Gemma 4 is only available on the vLLM nightly build — it has not yet landed in a stable release. Install via the nightly wheel (`uv pip install -U vllm --pre --extra-index-url https://wheels.vllm.ai/nightly/cu129 …`) or use the `vllm/vllm-openai:gemma4-0505-cu129` / `vllm/vllm-openai:gemma4-0505-cu130` images above; the standard `:latest` stable tag does not include this feature. ## References - [Model card](https://huggingface.co/google/gemma-4-E4B-it) - [FP8 variant](https://huggingface.co/RedHatAI/gemma-4-E4B-it-FP8-dynamic) - [Gemma docs](https://ai.google.dev/gemma/docs) - [vLLM Gemma 4 tool-call template](https://github.com/vllm-project/vllm/blob/main/examples/tool_chat_template_gemma4.jinja) - [TPU recipes: Trillium](https://github.com/AI-Hypercomputer/tpu-recipes/tree/main/inference/trillium/vLLM/Gemma4) - [TPU recipes: Ironwood](https://github.com/AI-Hypercomputer/tpu-recipes/blob/main/inference/ironwood/vLLM/Gemma4/) - [Xeon 6 recipe: Gemma 4 E4B IT](https://aiswcatalog.intel.com/models/google-gemma-4-e4b-it) --- # Qwen/QwQ-32B on vLLM Source: https://recipes.vllm.ai/Qwen/QwQ-32B Provider: Qwen Released: 2025-03-05T14:16:59.000Z Updated: 2026-09-12 Architecture: dense Parameters: 32B Context length: 40,960 Min vLLM: 0.7.3 Tasks: text Verified hardware: Xeon 6 Summary: Qwen's 32B dense reasoning model, validated for Intel Xeon 6 CPU execution. ## Variants - default: precision=bf16, vram_minimum_gb=77 Full precision BF16 - w8a8_int8: precision=int8, vram_minimum_gb=39, model_id=RedHatAI/QwQ-32B-quantized.w8a8 Red Hat AI INT8 W8A8 compressed-tensors checkpoint - awq: precision=int4, vram_minimum_gb=20, model_id=Qwen/QwQ-32B-AWQ Qwen official AWQ 4-bit checkpoint ## Guide ## Overview QwQ-32B is Qwen's 32B dense reasoning model. The CPU model validation table marks it supported on Intel Xeon 6. ## Prerequisites - Hardware: Intel Xeon 6 CPUs - vLLM CPU >= 0.8.5 for this validated Xeon 6 configuration ### pip (Intel Xeon 6 CPUs) For Intel and AMD x86 CPUs, follow the [CPU pre-built wheels](https://docs.vllm.ai/en/latest/getting_started/installation/cpu/#pre-built-wheels) installation instructions. ### Docker (Intel Xeon 6 CPUs) ```bash docker pull vllm/vllm-openai-cpu:latest-x86_64 ``` ## Intel Xeon 6 ```bash vllm serve Qwen/QwQ-32B \ --tensor-parallel-size 1 \ --enable-auto-tool-choice \ --tool-call-parser hermes \ --reasoning-parser deepseek_r1 ``` Docker (the image entrypoint is `vllm serve`): ```bash docker run \ --privileged --ipc=host -p 8000:8000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ vllm/vllm-openai-cpu:latest-x86_64 Qwen/QwQ-32B \ --tensor-parallel-size 1 \ --enable-auto-tool-choice \ --tool-call-parser hermes \ --reasoning-parser deepseek_r1 ``` W8A8 INT8: ```bash vllm serve RedHatAI/QwQ-32B-quantized.w8a8 ``` The example uses TP=1. Adjust tensor parallelism based on system topology; topology-specific CPU binding is intentionally not hard-coded. ## Runtime and Platform Tuning The following settings are intentionally not prescribed as portable CPU model defaults because they depend on the workload, hardware, or runtime environment: - `--max-num-batched-tokens`: scheduler/throughput tuning. - `--max-num-seqs`: concurrency and scheduler-capacity tuning. - `--gpu-memory-utilization`: platform memory-budget tuning. - `--no-enable-prefix-caching`: workload/benchmark cache-policy tuning. - `VLLM_ENGINE_ITERATION_TIMEOUT_S`: operational runtime timeout. These settings may still appear in validated hardware-specific overrides. Tune them at deployment time based on platform resources, workload shape, and latency/throughput goals. ## References - [Model card](https://huggingface.co/Qwen/QwQ-32B) - [W8A8 variant](https://huggingface.co/RedHatAI/QwQ-32B-quantized.w8a8) - [vLLM reasoning outputs](https://docs.vllm.ai/en/latest/features/reasoning_outputs.html) - [vLLM tool calling](https://docs.vllm.ai/en/latest/features/tool_calling.html) --- # Qwen/Qwen2.5-VL-7B-Instruct on vLLM Source: https://recipes.vllm.ai/Qwen/Qwen2.5-VL-7B-Instruct Provider: Qwen Released: 2025-01-26T09:26:37.000Z Updated: 2026-09-12 Architecture: dense Parameters: 7B Context length: 128,000 Min vLLM: 0.10.2 Tasks: multimodal, text Verified hardware: TPU v6e (Trillium), Xeon 6 Summary: Qwen2.5-VL dense vision-language model (7B) for image and video understanding — fits on a single TPU v6e chip, one GPU, or Intel Xeon 6 CPU. ## Variants - default: precision=bf16, vram_minimum_gb=17 Full precision BF16 — fits on a single TPU v6e chip or one 24GB+ GPU - awq: precision=int4, vram_minimum_gb=5, model_id=Qwen/Qwen2.5-VL-7B-Instruct-AWQ AWQ 4-bit quantized weights - fp8: precision=fp8, vram_minimum_gb=9, model_id=RedHatAI/Qwen2.5-VL-7B-Instruct-FP8-Dynamic RedHatAI FP8 (E4M3) with dynamic per-token activation quantization ## Guide ## Overview Qwen2.5-VL-7B-Instruct is the small dense vision-language model in the Qwen2.5-VL series. At 7B it fits comfortably on a single accelerator — one TPU v6e (Trillium) chip, or a single 24GB+ GPU — and is also validated for Intel Xeon 6 CPU serving. BF16 is the precision used in training, so BF16 inference gives the best accuracy. For the 72B sibling, see [Qwen2.5-VL-72B-Instruct](/Qwen/Qwen2.5-VL-72B-Instruct). ## Prerequisites ### NVIDIA ```bash uv venv source .venv/bin/activate uv pip install -U vllm --torch-backend auto ``` ### Intel Xeon 6 CPUs - Hardware: Intel Xeon 6 CPUs - vLLM CPU >= 0.8.5 for this validated Xeon 6 configuration For Intel and AMD x86 CPUs, follow the [CPU pre-built wheels](https://docs.vllm.ai/en/latest/getting_started/installation/cpu/#pre-built-wheels) installation instructions. ```bash docker pull vllm/vllm-openai-cpu:latest-x86_64 ``` ### TPU (Trillium / v6e) Use the official vLLM TPU image. The 7B model fits on a single v6e chip (`v6e-1`, `--topology 1x1`). ```bash export HF_HOME=/dev/shm export HF_TOKEN= docker run --rm --privileged --net=host \ --shm-size=16G \ -e HF_HOME -e HF_TOKEN \ vllm/vllm-tpu:latest \ vllm serve ... ``` ## Deployment Configurations ### Intel Xeon 6 Choose tensor parallelism based on the system topology; the recipe does not prescribe a fixed TP/DP layout or hard-code NUMA node IDs. ```bash vllm serve Qwen/Qwen2.5-VL-7B-Instruct ``` Docker: ```bash docker run \ --privileged --ipc=host -p 8000:8000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ vllm/vllm-openai-cpu:latest-x86_64 Qwen/Qwen2.5-VL-7B-Instruct \ --tensor-parallel-size 1 \ --mm-encoder-tp-mode data ``` #### Runtime and Platform Tuning The following settings are intentionally not prescribed as portable CPU model defaults because they depend on the workload, hardware, or runtime environment: - `--max-num-batched-tokens`: scheduler/throughput tuning. - `--max-num-seqs`: concurrency and scheduler-capacity tuning. - `--gpu-memory-utilization`: platform memory-budget tuning. - `--no-enable-prefix-caching`: workload/benchmark cache-policy tuning. - `VLLM_ENGINE_ITERATION_TIMEOUT_S`: operational runtime timeout. These settings may still appear in validated hardware-specific overrides. Tune them at deployment time based on platform resources, workload shape, and latency/throughput goals. ### TPU v6e (Trillium, single chip) Verified end-to-end on a single v6e chip. ```bash export HF_HOME=/dev/shm export HF_TOKEN= vllm serve Qwen/Qwen2.5-VL-7B-Instruct \ --tensor-parallel-size 1 \ --pipeline-parallel-size 1 \ --dtype bfloat16 \ --gpu-memory-utilization 0.98 \ --max-model-len 16384 \ --limit-mm-per-prompt '{"image":10,"video":0}' \ --mm-processor-kwargs '{"max_pixels":1003520}' \ --guided-decoding-backend xgrammar \ --disable-chunked-mm-input ``` ### Single GPU (BF16, TP=1) ```bash export CUDA_VISIBLE_DEVICES=0 vllm serve Qwen/Qwen2.5-VL-7B-Instruct \ --host 0.0.0.0 \ --port 8000 \ --limit-mm-per-prompt '{"image":2,"video":0}' ``` ### Multi-GPU (DP=4) For higher throughput across a node, data parallelism works better than TP for a model this size. ```bash export CUDA_VISIBLE_DEVICES=0,1,2,3 vllm serve Qwen/Qwen2.5-VL-7B-Instruct \ --host 0.0.0.0 \ --port 8000 \ --data-parallel-size 4 \ --limit-mm-per-prompt '{"image":2,"video":0}' ``` ## Configuration Tips - `--max-model-len 16384` is a safe default on a single v6e chip (native context is 128K); raise it if you have headroom. - `--limit-mm-per-prompt` caps incoming multimodal requests per prompt. - `--mm-processor-kwargs '{"max_pixels":1003520}'` bounds the per-image resolution to control encoder cost. - vLLM uses 90% of device memory by default; on TPU the recipe pushes `--gpu-memory-utilization 0.98` to maximize KV cache. - `--disable-chunked-mm-input` is recommended on TPU for stable multimodal batching. ## Benchmarking Launch the server with `--no-enable-prefix-caching` to get consistent measurements. ### VisionArena-Chat ```bash vllm bench serve \ --host 0.0.0.0 \ --port 8000 \ --backend openai-chat \ --endpoint /v1/chat/completions \ --model Qwen/Qwen2.5-VL-7B-Instruct \ --dataset-name hf \ --dataset-path lmarena-ai/VisionArena-Chat \ --num-prompts 128 ``` ### Random Synthetic ```bash vllm bench serve \ --host 0.0.0.0 \ --port 8000 \ --model Qwen/Qwen2.5-VL-7B-Instruct \ --dataset-name random \ --random-input-len 8000 \ --random-output-len 1000 \ --num-prompts 128 ``` ## References - [Model card](https://huggingface.co/Qwen/Qwen2.5-VL-7B-Instruct) - [Qwen2.5-VL on Trillium (v6e)](https://github.com/AI-Hypercomputer/tpu-recipes/tree/main/inference/trillium/vLLM/Qwen2.5-VL) - [vLLM multimodal inputs guide](https://docs.vllm.ai/en/latest/features/multimodal_inputs.html) --- # Qwen/Qwen3-1.7B on vLLM Source: https://recipes.vllm.ai/Qwen/Qwen3-1.7B Provider: Qwen Released: 2025-04-27T03:41:05.000Z Updated: 2026-09-12 Architecture: dense Parameters: 1.7B Context length: 40,960 Min vLLM: 0.8.5 Tasks: text Verified hardware: Xeon 6 Summary: Qwen3 1.7B dense model with hybrid thinking/non-thinking modes, validated for BF16 serving on Intel Xeon 6. ## Variants - default: precision=bf16, vram_minimum_gb=5 Full precision BF16 ## Guide ## Overview Qwen3-1.7B is a compact dense Qwen3 model with hybrid thinking and non-thinking modes. This recipe includes an Intel Xeon 6 CPU configuration validated with BF16 serving. ## Prerequisites - Hardware: 1x Xeon 6 CPUs - vLLM >= 0.8.5 ### pip (Intel Xeon 6 CPUs) For Intel and AMD x86 CPUs, follow the [CPU pre-built wheels](https://docs.vllm.ai/en/latest/getting_started/installation/cpu/#pre-built-wheels) installation instructions. ### Docker (Intel Xeon 6 CPUs) ```bash docker pull vllm/vllm-openai-cpu:latest-x86_64 ``` ## Intel Xeon 6 Choose the tensor parallel size based on the system topology and deployment requirements. Add `--tensor-parallel-size ` when needed; the recipe does not prescribe a Xeon 6 TP value or hard-code NUMA node IDs. ```bash vllm serve Qwen/Qwen3-1.7B \ --max-num-batched-tokens 16384 \ --gpu-memory-utilization 0.8 ``` Docker: ```bash docker run \ --privileged --ipc=host -p 8000:8000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ vllm/vllm-openai-cpu:latest-x86_64 Qwen/Qwen3-1.7B \ --tensor-parallel-size 1 \ --max-num-batched-tokens 16384 \ --gpu-memory-utilization 0.8 \ --enable-auto-tool-choice \ --tool-call-parser hermes \ --reasoning-parser qwen3 ``` ## Client Usage ```python from openai import OpenAI client = OpenAI(base_url="http://localhost:8000/v1", api_key="unused") response = client.chat.completions.create( model="Qwen/Qwen3-1.7B", messages=[{"role": "user", "content": "Explain tensor parallelism briefly."}], ) print(response.choices[0].message.content) ``` ## References - [Model card](https://huggingface.co/Qwen/Qwen3-1.7B) - [vLLM CPU installation](https://docs.vllm.ai/en/latest/getting_started/installation/cpu/) --- # Qwen/Qwen3-14B on vLLM Source: https://recipes.vllm.ai/Qwen/Qwen3-14B Provider: Qwen Released: 2025-04-27T03:42:45.000Z Updated: 2026-09-12 Architecture: dense Parameters: 14B Context length: 40,960 Min vLLM: 0.8.5 Tasks: text Verified hardware: Xeon 6 Summary: Qwen3 14B dense model with hybrid thinking/non-thinking modes, validated for Intel Xeon 6 CPU serving. ## Variants - default: precision=bf16, vram_minimum_gb=34 Full precision BF16 - fp8: precision=fp8, vram_minimum_gb=17, model_id=Qwen/Qwen3-14B-FP8 Qwen official FP8 checkpoint - awq: precision=int4, vram_minimum_gb=10, model_id=Qwen/Qwen3-14B-AWQ Qwen official AWQ 4-bit checkpoint ## Guide ## Overview Qwen3-14B is a Qwen3 dense model with hybrid thinking and non-thinking modes. Intel Xeon 6 CPU serving was validated with BF16. ## Prerequisites - Hardware: Intel Xeon 6 CPUs - vLLM >= 0.8.5 ### pip (Intel Xeon 6 CPUs) Follow the [CPU pre-built wheels](https://docs.vllm.ai/en/latest/getting_started/installation/cpu/#pre-built-wheels) installation instructions. ### Docker (Intel Xeon 6 CPUs) ```bash docker pull vllm/vllm-openai-cpu:latest-x86_64 ``` ## Intel Xeon 6 Choose tensor parallelism based on the system topology; the recipe does not prescribe a fixed TP/DP layout or hard-code NUMA node IDs. ```bash vllm serve Qwen/Qwen3-14B ``` Docker: ```bash docker run \ --privileged --ipc=host -p 8000:8000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ vllm/vllm-openai-cpu:latest-x86_64 Qwen/Qwen3-14B \ --tensor-parallel-size 1 \ --enable-auto-tool-choice \ --tool-call-parser hermes ``` ## Configuration Notes - TP/DP values and explicit CPU bindings are intentionally not prescribed. - The `qwen3` reasoning parser is available in vLLM 0.9.0 and later. ## Runtime and Platform Tuning The following settings are intentionally not prescribed as portable CPU model defaults because they depend on the workload, hardware, or runtime environment: - `--max-num-batched-tokens`: scheduler/throughput tuning. - `--max-num-seqs`: concurrency and scheduler-capacity tuning. - `--gpu-memory-utilization`: platform memory-budget tuning. - `--no-enable-prefix-caching`: workload/benchmark cache-policy tuning. - `VLLM_ENGINE_ITERATION_TIMEOUT_S`: operational runtime timeout. These settings may still appear in validated hardware-specific overrides. Tune them at deployment time based on platform resources, workload shape, and latency/throughput goals. ## References - [Model card](https://huggingface.co/Qwen/Qwen3-14B) - [vLLM CPU installation](https://docs.vllm.ai/en/latest/getting_started/installation/cpu/) --- # Qwen/Qwen3-4B on vLLM Source: https://recipes.vllm.ai/Qwen/Qwen3-4B Provider: Qwen Released: 2025-04-27T03:41:29.000Z Updated: 2026-09-12 Architecture: dense Parameters: 4B Context length: 40,960 Min vLLM: 0.8.5 Tasks: text Verified hardware: TPU v6e (Trillium), Xeon 6 Summary: Qwen3 4B dense model with hybrid thinking/non-thinking modes — fits on a single TPU v6e chip, one GPU or one Xeon 6 NUMA node. ## Variants - default: precision=bf16, vram_minimum_gb=10 Full precision BF16 — fits on a single TPU v6e chip or one 16GB+ GPU or one Xeon 6 NUMA node - fp8: precision=fp8, vram_minimum_gb=5, model_id=Qwen/Qwen3-4B-FP8 Qwen official FP8 checkpoint - redhat_fp8: precision=fp8, vram_minimum_gb=5, model_id=RedHatAI/Qwen3-4B-FP8-dynamic RedHatAI FP8 (E4M3) with dynamic per-token activation quantization ## Guide ## Overview Qwen3-4B is the small dense model in the Qwen3 series, with hybrid thinking / non-thinking modes. At 4B it fits on a single accelerator — one TPU v6e (Trillium) chip, a single 16GB+ GPU or one Xeon 6 NUMA node — making it the cheapest entry point in the family. BF16 is the training precision and gives the best accuracy. Toggle thinking mode per request with `enable_thinking` in `chat_template_kwargs`, or set the **Reasoning** feature to add the Qwen3 reasoning parser so `` content is split into a separate field. ## Prerequisites ### NVIDIA ```bash uv venv source .venv/bin/activate uv pip install -U vllm --torch-backend auto ``` ### pip (Intel Xeon 6 CPUs) For Intel and AMD x86 CPUs, follow the [CPU pre-built wheels](https://docs.vllm.ai/en/latest/getting_started/installation/cpu/#pre-built-wheels) installation instructions. ### TPU (Trillium / v6e) Use the official vLLM TPU image. The 4B model fits on a single v6e chip (`v6e-1`, `--topology 1x1`). ```bash export HF_HOME=/dev/shm export HF_TOKEN= ``` ## Deployment Configurations ### TPU v6e (Trillium, single chip, TP=1) Verified end-to-end on a single v6e chip. ```bash export HF_HOME=/dev/shm export HF_TOKEN= export TP=1 export MAX_MODEL_LEN=4096 vllm serve Qwen/Qwen3-4B \ --seed 42 \ --gpu-memory-utilization 0.98 \ --max-num-batched-tokens 1024 \ --max-num-seqs 128 \ --tensor-parallel-size $TP \ --max-model-len $MAX_MODEL_LEN ``` ### Single GPU (BF16) ```bash vllm serve Qwen/Qwen3-4B \ --host 0.0.0.0 \ --port 8000 ``` ### Intel Xeon 6 Deployment via Docker Launch the x86 CPU vLLM Docker container for `Qwen/Qwen3-4B`: ```bash docker run \ --privileged --ipc=host -p 8000:8000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ vllm/vllm-openai-cpu:latest-x86_64 Qwen/Qwen3-4B \ --tensor-parallel-size 1 \ --enable-auto-tool-choice \ --tool-call-parser hermes \ --reasoning-parser qwen3 ``` ## Configuration Tips - `--max-model-len 4096` is the TPU-recipe default; native context is 40K — raise it if you have headroom. - vLLM uses 90% of device memory by default; the TPU recipe pushes `--gpu-memory-utilization 0.98` to maximize KV cache. - For thinking mode Qwen recommends `temperature=0.6, top_p=0.95, top_k=20`; for non-thinking, `temperature=0.7, top_p=0.8`. ## Benchmarking Launch the server with `--no-enable-prefix-caching` for consistent measurements. ```bash vllm bench serve \ --host 0.0.0.0 \ --port 8000 \ --model Qwen/Qwen3-4B \ --dataset-name random \ --random-input-len 1024 \ --random-output-len 1024 \ --num-prompts 128 ``` ## References - [Model card](https://huggingface.co/Qwen/Qwen3-4B) - [Qwen3 on Trillium (v6e)](https://github.com/AI-Hypercomputer/tpu-recipes/tree/main/inference/trillium/vLLM/Qwen3) --- # Qwen/Qwen3-8B on vLLM Source: https://recipes.vllm.ai/Qwen/Qwen3-8B Provider: Qwen Released: 2025-04-27T03:42:21.000Z Updated: 2026-09-12 Architecture: dense Parameters: 8.2B Context length: 40,960 Min vLLM: 0.8.5 Tasks: text Verified hardware: Xeon 6 Summary: Qwen3 8.2B dense model with hybrid thinking/non-thinking modes, validated for BF16 serving on Intel Xeon 6. ## Variants - default: precision=bf16, vram_minimum_gb=20 Full precision BF16 ## Guide ## Overview Qwen3-8B is a dense Qwen3 model with hybrid thinking and non-thinking modes. This recipe includes an Intel Xeon 6 CPU configuration validated with BF16 serving. ## Prerequisites - Hardware: 2x Xeon 6 CPUs - vLLM >= 0.8.5 ### pip (Intel Xeon 6 CPUs) For Intel and AMD x86 CPUs, follow the [CPU pre-built wheels](https://docs.vllm.ai/en/latest/getting_started/installation/cpu/#pre-built-wheels) installation instructions. ### Docker (Intel Xeon 6 CPUs) ```bash docker pull vllm/vllm-openai-cpu:latest-x86_64 ``` ## Intel Xeon 6 Choose the tensor parallel size based on the system topology and deployment requirements. Add `--tensor-parallel-size ` when needed; the recipe does not prescribe a Xeon 6 TP value or hard-code NUMA node IDs. ```bash vllm serve Qwen/Qwen3-8B \ --max-num-batched-tokens 16384 \ --gpu-memory-utilization 0.8 ``` Docker: ```bash docker run \ --privileged --ipc=host -p 8000:8000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ vllm/vllm-openai-cpu:latest-x86_64 Qwen/Qwen3-8B \ --tensor-parallel-size 1 \ --max-num-batched-tokens 16384 \ --gpu-memory-utilization 0.8 \ --enable-auto-tool-choice \ --tool-call-parser hermes \ --reasoning-parser qwen3 ``` ## Client Usage ```python from openai import OpenAI client = OpenAI(base_url="http://localhost:8000/v1", api_key="unused") response = client.chat.completions.create( model="Qwen/Qwen3-8B", messages=[{"role": "user", "content": "Explain tensor parallelism briefly."}], ) print(response.choices[0].message.content) ``` ## References - [Model card](https://huggingface.co/Qwen/Qwen3-8B) - [vLLM CPU installation](https://docs.vllm.ai/en/latest/getting_started/installation/cpu/) --- # Qwen/Qwen3-VL-30B-A3B-Instruct on vLLM Source: https://recipes.vllm.ai/Qwen/Qwen3-VL-30B-A3B-Instruct Provider: Qwen Released: 2025-09-30T10:52:01.000Z Updated: 2026-09-12 Architecture: moe Parameters: 30B Context length: 262,144 Min vLLM: 0.11.0 Tasks: multimodal, text Verified hardware: Xeon 6 Summary: Qwen3-VL MoE vision-language model with 30B total / 3B active parameters, supporting image, video, and text workloads. ## Variants - default: precision=bf16, vram_minimum_gb=72 Full precision BF16 ## Guide ## Overview Qwen3-VL-30B-A3B-Instruct is a multimodal Qwen3-VL MoE model. Intel Xeon 6 CPU serving was validated with BF16. ## Prerequisites - Hardware: Intel Xeon 6 CPUs - vLLM >= 0.11.0 ### pip (Intel Xeon 6 CPUs) For Intel and AMD x86 CPUs, follow the [CPU pre-built wheels](https://docs.vllm.ai/en/latest/getting_started/installation/cpu/#pre-built-wheels) installation instructions. ### Docker (Intel Xeon 6 CPUs) ```bash docker pull vllm/vllm-openai-cpu:latest-x86_64 ``` ## Intel Xeon 6 ```bash vllm serve Qwen/Qwen3-VL-30B-A3B-Instruct \ --tensor-parallel-size 1 \ --mm-encoder-tp-mode data ``` Docker (the image entrypoint is `vllm serve`): ```bash docker run \ --privileged --ipc=host -p 8000:8000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ vllm/vllm-openai-cpu:latest-x86_64 Qwen/Qwen3-VL-30B-A3B-Instruct \ --tensor-parallel-size 1 \ --mm-encoder-tp-mode data ``` The example uses TP=1. Adjust TP/DP based on system topology; NUMA binding is intentionally not hard-coded in the recipe. ## Configuration Notes - TP/DP values and topology-specific CPU binding remain deployment choices. ## Runtime and Platform Tuning The following settings are intentionally not prescribed as portable CPU model defaults because they depend on the workload, hardware, or runtime environment: - `--max-num-batched-tokens`: scheduler/throughput tuning. - `--max-num-seqs`: concurrency and scheduler-capacity tuning. - `--gpu-memory-utilization`: platform memory-budget tuning. - `--no-enable-prefix-caching`: workload/benchmark cache-policy tuning. - `VLLM_ENGINE_ITERATION_TIMEOUT_S`: operational runtime timeout. These settings may still appear in validated hardware-specific overrides. Tune them at deployment time based on platform resources, workload shape, and latency/throughput goals. ## References - [Model card](https://huggingface.co/Qwen/Qwen3-VL-30B-A3B-Instruct) - [vLLM CPU installation](https://docs.vllm.ai/en/latest/getting_started/installation/cpu/) --- # meta-llama/Llama-3.1-8B-Instruct on vLLM Source: https://recipes.vllm.ai/meta-llama/Llama-3.1-8B-Instruct Provider: Meta Released: 2024-07-18T08:56:00.000Z Updated: 2026-09-12 Architecture: dense Parameters: 8B Context length: 131,072 Min vLLM: 0.6.0 Tasks: text Verified hardware: H100, H200, MI300X, MI325X, MI355X, TPU v6e (Trillium), Xeon 6, Arc Pro B60, Arc Pro B70 Summary: Meta's Llama 3.1 8B dense instruction-tuned language model with 128K context ## Variants - default: precision=bf16, vram_minimum_gb=20 Full precision BF16 - w8a8_int8: precision=int8, vram_minimum_gb=10, model_id=RedHatAI/Meta-Llama-3.1-8B-Instruct-quantized.w8a8 Red Hat AI INT8 W8A8 compressed-tensors checkpoint - awq_int4: precision=int4, vram_minimum_gb=5, model_id=hugging-quants/Meta-Llama-3.1-8B-Instruct-AWQ-INT4 AWQ INT4 checkpoint - nvfp4: precision=nvfp4, vram_minimum_gb=5, model_id=nvidia/Llama-3.1-8B-Instruct-NVFP4 NVIDIA NVFP4 quantized weights for Blackwell GPUs - nvidia_fp8: precision=fp8, vram_minimum_gb=10, model_id=nvidia/Llama-3.1-8B-Instruct-FP8 FP8 quantized weights for Hopper/Blackwell GPUs - fp8: precision=fp8, vram_minimum_gb=10, model_id=RedHatAI/Meta-Llama-3.1-8B-Instruct-FP8-dynamic RedHatAI FP8 (E4M3) with dynamic per-token activation quantization - redhat_fp8: precision=fp8, vram_minimum_gb=10, model_id=RedHatAI/Meta-Llama-3.1-8B-Instruct-FP8 RedHatAI FP8 quantization - redhat_fp8_block: precision=fp8, vram_minimum_gb=10, model_id=RedHatAI/Llama-3.1-8B-Instruct-FP8-block RedHatAI FP8 block quantization - redhat_nvfp4: precision=nvfp4, vram_minimum_gb=6, model_id=RedHatAI/Llama-3.1-8B-Instruct-NVFP4 RedHatAI NVFP4 quantization ## Guide ## Overview Llama 3.1 Instruct is Meta's instruction-tuned language model family. The 8B dense variant is lightweight and ideal for single-GPU deployment, with 128K context support. A 70B variant is also available (see related recipes). TPU support is provided through [vLLM TPU](https://github.com/vllm-project/tpu-inference) with a recipe for [Trillium](https://github.com/AI-Hypercomputer/tpu-recipes/tree/main/inference/trillium/vLLM/Llama3.x). ## Prerequisites - Hardware: 1x GPU with >=16 GB VRAM (e.g. A100, L40S, H100, H200, MI300X/MI325X/MI355X), Intel XPU B60/B70, or 2x Xeon6/Xeon5 NUMA node - vLLM >= 0.6.0 - CUDA Driver compatible with your vLLM version for 1x GPU - Docker with NVIDIA Container Toolkit (recommended) for 1x GPU ### Install vLLM ```bash uv venv source .venv/bin/activate uv pip install vllm --torch-backend auto ``` ### AMD ROCm The ROCm wheel requires Python 3.12, ROCm 7.0+, and glibc >= 2.35. Use the ROCm Docker image when your host environment does not meet those requirements. ```bash uv venv --python 3.12 source .venv/bin/activate uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/ ``` Launch on 1x MI300X / MI325X / MI355X with AITER enabled: ```bash export VLLM_ROCM_USE_AITER=1 vllm serve meta-llama/Llama-3.1-8B-Instruct \ --tensor-parallel-size 1 ``` The first AITER launch may take several minutes while ROCm kernels are compiled and cached. ### pip (Intel Xeon 6 CPUs) For Intel and AMD x86 CPUs, follow the [CPU pre-built wheels](https://docs.vllm.ai/en/latest/getting_started/installation/cpu/#pre-built-wheels) installation instructions. ### Docker (Intel Xeon 6 CPUs) ```bash docker pull vllm/vllm-openai-cpu:latest-x86_64 # For Intel Xeon 6 ``` ### Docker (Intel XPU B60 / B70) ```bash docker pull vllm/vllm-openai-xpu:latest ``` ### Docker (Cloud TPU — Trillium) TPU uses the separate `vllm/vllm-tpu` image (no pip wheel). Pull the tag specified by the upstream [Trillium recipe](https://github.com/AI-Hypercomputer/tpu-recipes/tree/main/inference/trillium/vLLM/Llama3.x), then run: ```bash docker run -itd --name llama31-tpu \ --privileged --network host --shm-size 16G \ -v /dev/shm:/dev/shm -e HF_TOKEN=$HF_TOKEN \ vllm/vllm-tpu:latest \ vllm serve meta-llama/Llama-3.1-8B-Instruct \ --tensor-parallel-size 1 \ --max-model-len 2048 \ --host 0.0.0.0 --port 8000 ``` The 8B fits on a single chip (`v6e-1`, TP=1) — unlike the 70B, which needs a v6e-8 slice. ### Intel Xeon 6 Deployment via Docker Launch the x86 CPU vLLM Docker container for `meta-llama/Llama-3.1-8B-Instruct`: ```bash docker run \ --privileged --ipc=host -p 8000:8000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ vllm/vllm-openai-cpu:latest-x86_64 meta-llama/Llama-3.1-8B-Instruct \ --tensor-parallel-size 2 ``` The validation TP/DP layout and host-specific CPU placement are intentionally not prescribed. The following validation settings are not model requirements and are intentionally not portable CPU recipe defaults: - `--max-num-batched-tokens` and `--max-num-seqs`: scheduler batch/concurrency tuning for workload shape, latency targets, and platform capacity. - `--gpu-memory-utilization`: platform memory-budget tuning. - `--no-enable-prefix-caching`: workload/benchmark cache-policy tuning. - `VLLM_ENGINE_ITERATION_TIMEOUT_S`: runtime operational timeout tuning. Hardware-specific overrides may still use these settings when they are part of a validated platform configuration. ### Intel XPU (B60, B70) ```bash docker run --device /dev/dri \ -v /dev/dri/by-path:/dev/dri/by-path --shm-size=16g \ --privileged --ipc=host -p 8000:8000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ vllm/vllm-openai-xpu:latest \ meta-llama/Llama-3.1-8B-Instruct \ --max-model-len 8192 ``` ## Client Usage ```python from openai import OpenAI client = OpenAI(base_url="http://localhost:8000/v1", api_key="unused") response = client.chat.completions.create( model="meta-llama/Llama-3.1-8B-Instruct", messages=[{"role": "user", "content": "Hello, how are you?"}], ) print(response.choices[0].message.content) ``` ## Speculative Decoding (EAGLE3) An EAGLE3 draft head is available at [`RedHatAI/Llama-3.1-8B-Instruct-speculator.eagle3`](https://huggingface.co/RedHatAI/Llama-3.1-8B-Instruct-speculator.eagle3). Enable the **Spec Decoding** toggle above, or add the `--speculative-config` manually: ```bash vllm serve meta-llama/Llama-3.1-8B-Instruct \ --speculative-config '{"model":"RedHatAI/Llama-3.1-8B-Instruct-speculator.eagle3","method":"eagle3","num_speculative_tokens":3}' ``` EAGLE3 verification requires vLLM >= 0.9.0. `num_speculative_tokens` of 3 is a reasonable starting point for chat workloads. ## Troubleshooting **OOM on small GPUs:** Lower `--max-model-len` or `--gpu-memory-utilization`. **EAGLE3 draft head not loading:** Upgrade vLLM to >= 0.9.0 — earlier releases don't support the EAGLE3 verification path. ## References - [Model card](https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct) - [EAGLE3 draft head](https://huggingface.co/RedHatAI/Llama-3.1-8B-Instruct-speculator.eagle3) - [Llama 3.1 70B model card](https://huggingface.co/meta-llama/Llama-3.1-70B-Instruct) - [TPU recipes: Trillium](https://github.com/AI-Hypercomputer/tpu-recipes/tree/main/inference/trillium/vLLM/Llama3.x) --- # meta-llama/Llama-3.1-8B on vLLM Source: https://recipes.vllm.ai/meta-llama/Llama-3.1-8B Provider: Meta Released: 2024-07-14T22:20:15.000Z Updated: 2026-09-12 Architecture: dense Parameters: 8B Context length: 131,072 Min vLLM: 0.6.0 Tasks: text Verified hardware: Xeon 6 Summary: Meta Llama 3.1 8B dense base language model with a CPU-validated Red Hat AI W8A8 variant. ## Variants - default: precision=bf16, vram_minimum_gb=20 Full precision BF16 parent checkpoint - w8a8_int8: precision=int8, vram_minimum_gb=10, model_id=RedHatAI/Meta-Llama-3.1-8B-quantized.w8a8 Red Hat AI INT8 W8A8 compressed-tensors checkpoint ## Guide ## Overview Llama 3.1 8B is Meta's dense pretrained base model with 128K context. This recipe is separate from Llama-3.1-8B-Instruct because the Red Hat AI `Meta-Llama-3.1-8B-quantized.w8a8` checkpoint is quantized from the base model, not the instruction-tuned checkpoint. The recipe retains portable model-level CPU settings while leaving TP/DP and host-specific CPU placement as deployment choices. ## Prerequisites - Hardware: 2x Xeon 6 NUMA nodes - vLLM >= 0.6.0 ### pip (Intel Xeon 6 CPUs) For Intel and AMD x86 CPUs, follow the [CPU pre-built wheels](https://docs.vllm.ai/en/latest/getting_started/installation/cpu/#pre-built-wheels) installation instructions. ### Docker (Intel Xeon 6 CPUs) ```bash docker pull vllm/vllm-openai-cpu:latest-x86_64 ``` ## Intel Xeon 6 ```bash vllm serve RedHatAI/Meta-Llama-3.1-8B-quantized.w8a8 ``` Docker: ```bash docker run \ --privileged --ipc=host -p 8000:8000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ vllm/vllm-openai-cpu:latest-x86_64 RedHatAI/Meta-Llama-3.1-8B-quantized.w8a8 \ --tensor-parallel-size 2 ``` ## Runtime and Platform Tuning The following validation settings are not model requirements and are intentionally not portable CPU recipe defaults: - `--max-num-batched-tokens` and `--max-num-seqs`: scheduler batch/concurrency tuning for workload shape, latency targets, and platform capacity. - `--gpu-memory-utilization`: platform memory-budget tuning. - `--no-enable-prefix-caching`: workload/benchmark cache-policy tuning. - `VLLM_ENGINE_ITERATION_TIMEOUT_S`: runtime operational timeout tuning. Hardware-specific overrides may still use these settings when they are part of a validated platform configuration. ## Configuration Notes - The validation TP/DP layout is benchmark topology and is not hard-coded. - Llama has native vLLM architecture support, so `--trust-remote-code` is not required by this recipe. ## References - [Parent model](https://huggingface.co/meta-llama/Llama-3.1-8B) - [Red Hat AI W8A8 variant](https://huggingface.co/RedHatAI/Meta-Llama-3.1-8B-quantized.w8a8) - [Llama-3.1-8B-Instruct recipe](https://recipes.vllm.ai/meta-llama/Llama-3.1-8B-Instruct) - [vLLM CPU installation](https://docs.vllm.ai/en/latest/getting_started/installation/cpu/) --- # meta-llama/Llama-3.2-1B-Instruct on vLLM Source: https://recipes.vllm.ai/meta-llama/Llama-3.2-1B-Instruct Provider: Meta Released: 2024-09-18T15:12:47.000Z Updated: 2026-09-12 Architecture: dense Parameters: 1.23B Context length: 131,072 Min vLLM: 0.6.2 Tasks: text Verified hardware: Xeon 6 Summary: Meta Llama 3.2 1B Instruct with CPU-validated Red Hat AI W8A8 and AMead10 AWQ variants. ## Variants - default: precision=bf16, vram_minimum_gb=3 Full precision BF16 parent checkpoint - w8a8_int8: precision=int8, vram_minimum_gb=2, model_id=RedHatAI/Llama-3.2-1B-Instruct-quantized.w8a8 Red Hat AI INT8 W8A8 compressed-tensors checkpoint - awq_int4: precision=int4, vram_minimum_gb=2, model_id=AMead10/Llama-3.2-1B-Instruct-AWQ AWQ INT4 checkpoint ## Guide ## Overview This recipe groups the Llama 3.2 1B Instruct family under the canonical Meta parent. The Xeon 6 validation workbook has PASS rows for the Red Hat AI W8A8 and AMead10 AWQ variants. The W8A8 checkpoint is detected from compressed-tensors metadata. The AWQ variant uses the recipe's `--quantization awq` selector. ## Prerequisites - Hardware: 1x Xeon6/Xeon5 NUMA node - vLLM >= 0.6.2 ### pip (Intel Xeon 6 CPUs) For Intel and AMD x86 CPUs, follow the [CPU pre-built wheels](https://docs.vllm.ai/en/latest/getting_started/installation/cpu/#pre-built-wheels) installation instructions. ### Docker (Intel Xeon 6 CPUs) ```bash docker pull vllm/vllm-openai-cpu:latest-x86_64 ``` ## Intel Xeon 6 ```bash vllm serve meta-llama/Llama-3.2-1B-Instruct \ --tensor-parallel-size 1 ``` Docker (the image entrypoint is `vllm serve`): ```bash docker run \ --privileged --ipc=host -p 8000:8000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ vllm/vllm-openai-cpu:latest-x86_64 meta-llama/Llama-3.2-1B-Instruct \ --tensor-parallel-size 1 ``` The example uses TP=1. Adjust TP/DP for the deployment topology; host-specific CPU binding is intentionally not hard-coded. ## Runtime and Platform Tuning The following validation settings are not model requirements and are intentionally not portable CPU recipe defaults: - `--max-num-batched-tokens` and `--max-num-seqs`: scheduler batch/concurrency tuning for workload shape, latency targets, and platform capacity. - `--gpu-memory-utilization`: platform memory-budget tuning. - `--no-enable-prefix-caching`: workload/benchmark cache-policy tuning. - `VLLM_ENGINE_ITERATION_TIMEOUT_S`: runtime operational timeout tuning. Hardware-specific overrides may still use these settings when they are part of a validated platform configuration. ## References - [Meta parent model](https://huggingface.co/meta-llama/Llama-3.2-1B-Instruct) - [Red Hat AI W8A8 checkpoint](https://huggingface.co/RedHatAI/Llama-3.2-1B-Instruct-quantized.w8a8) - [AMead10 AWQ checkpoint](https://huggingface.co/AMead10/Llama-3.2-1B-Instruct-AWQ) --- # meta-llama/Llama-3.2-1B on vLLM Source: https://recipes.vllm.ai/meta-llama/Llama-3.2-1B Provider: Meta Released: 2024-09-18T15:03:14.000Z Updated: 2026-09-12 Architecture: dense Parameters: 1.23B Context length: 131,072 Min vLLM: 0.6.2 Tasks: text Verified hardware: Xeon 6 Summary: Meta Llama 3.2 1.23B dense pretrained language model with 128K context, validated for BF16 serving on Intel Xeon 6. ## Variants - default: precision=bf16, vram_minimum_gb=3 Full precision BF16 ## Guide ## Overview Llama 3.2 1B is Meta's compact pretrained dense language model with 128K context. This recipe includes an Intel Xeon 6 CPU configuration validated with BF16 serving. ## Prerequisites - Hardware: 1x Xeon 6 NUMA node - vLLM >= 0.6.2 ### pip (Intel Xeon 6 CPUs) For Intel and AMD x86 CPUs, follow the [CPU pre-built wheels](https://docs.vllm.ai/en/latest/getting_started/installation/cpu/#pre-built-wheels) installation instructions. ### Docker (Intel Xeon 6 CPUs) ```bash docker pull vllm/vllm-openai-cpu:latest-x86_64 ``` ## Runtime and Platform Tuning The following validation settings are not model requirements and are intentionally not portable CPU recipe defaults: - `--max-num-batched-tokens` and `--max-num-seqs`: scheduler batch/concurrency tuning for workload shape, latency targets, and platform capacity. - `--gpu-memory-utilization`: platform memory-budget tuning. - `--no-enable-prefix-caching`: workload/benchmark cache-policy tuning. - `VLLM_ENGINE_ITERATION_TIMEOUT_S`: runtime operational timeout tuning. Hardware-specific overrides may still use these settings when they are part of a validated platform configuration. ## Intel Xeon 6 Choose the tensor parallel size based on the system topology and deployment requirements. Add `--tensor-parallel-size ` when needed; the recipe does not prescribe a Xeon 6 TP value or hard-code NUMA node IDs. ```bash vllm serve meta-llama/Llama-3.2-1B ``` Docker: ```bash docker run \ --privileged --ipc=host -p 8000:8000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ vllm/vllm-openai-cpu:latest-x86_64 meta-llama/Llama-3.2-1B \ --tensor-parallel-size 1 ``` ## Client Usage ```python from openai import OpenAI client = OpenAI(base_url="http://localhost:8000/v1", api_key="unused") response = client.completions.create( model="meta-llama/Llama-3.2-1B", prompt="Explain tensor parallelism briefly.", max_tokens=128, ) print(response.choices[0].text) ``` ## References - [Model card](https://huggingface.co/meta-llama/Llama-3.2-1B) - [vLLM CPU installation](https://docs.vllm.ai/en/latest/getting_started/installation/cpu/) --- # meta-llama/Llama-3.2-3B-Instruct on vLLM Source: https://recipes.vllm.ai/meta-llama/Llama-3.2-3B-Instruct Provider: Meta Released: 2024-09-18T15:19:20.000Z Updated: 2026-09-12 Architecture: dense Parameters: 3.21B Context length: 131,072 Min vLLM: 0.6.2 Tasks: text Verified hardware: Xeon 6 Summary: Meta Llama 3.2 3.21B dense instruction-tuned language model with 128K context, validated for BF16 serving on Intel Xeon 6. ## Variants - default: precision=bf16, vram_minimum_gb=8 Full precision BF16 - w8a8_int8: precision=int8, vram_minimum_gb=4, model_id=RedHatAI/Llama-3.2-3B-Instruct-quantized.w8a8 Red Hat AI INT8 W8A8 compressed-tensors checkpoint - awq_int4: precision=int4, vram_minimum_gb=3, model_id=AMead10/Llama-3.2-3B-Instruct-AWQ AWQ INT4 checkpoint ## Guide ## Overview Llama 3.2 3B Instruct is Meta's compact instruction-tuned dense language model with 128K context. This recipe includes an Intel Xeon 6 CPU configuration validated with BF16 serving. ## Prerequisites - Hardware: 1x Xeon6/Xeon5 NUMA node - vLLM >= 0.6.2 ### pip (Intel Xeon 6 CPUs) For Intel and AMD x86 CPUs, follow the [CPU pre-built wheels](https://docs.vllm.ai/en/latest/getting_started/installation/cpu/#pre-built-wheels) installation instructions. ### Docker (Intel Xeon 6 CPUs) ```bash docker pull vllm/vllm-openai-cpu:latest-x86_64 ``` ## Runtime and Platform Tuning The following validation settings are not model requirements and are intentionally not portable CPU recipe defaults: - `--max-num-batched-tokens` and `--max-num-seqs`: scheduler batch/concurrency tuning for workload shape, latency targets, and platform capacity. - `--gpu-memory-utilization`: platform memory-budget tuning. - `--no-enable-prefix-caching`: workload/benchmark cache-policy tuning. - `VLLM_ENGINE_ITERATION_TIMEOUT_S`: runtime operational timeout tuning. Hardware-specific overrides may still use these settings when they are part of a validated platform configuration. ## Intel Xeon 6 Choose the tensor parallel size based on the system topology and deployment requirements. Add `--tensor-parallel-size ` when needed; the recipe does not prescribe a Xeon 6 TP value or hard-code NUMA node IDs. ```bash vllm serve meta-llama/Llama-3.2-3B-Instruct ``` Docker: ```bash docker run \ --privileged --ipc=host -p 8000:8000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ vllm/vllm-openai-cpu:latest-x86_64 meta-llama/Llama-3.2-3B-Instruct \ --tensor-parallel-size 1 ``` ## Client Usage ```python from openai import OpenAI client = OpenAI(base_url="http://localhost:8000/v1", api_key="unused") response = client.chat.completions.create( model="meta-llama/Llama-3.2-3B-Instruct", messages=[{"role": "user", "content": "Explain tensor parallelism briefly."}], ) print(response.choices[0].message.content) ``` ## References - [Model card](https://huggingface.co/meta-llama/Llama-3.2-3B-Instruct) - [vLLM CPU installation](https://docs.vllm.ai/en/latest/getting_started/installation/cpu/) --- # meta-llama/Llama-4-Scout-17B-16E-Instruct on vLLM Source: https://recipes.vllm.ai/meta-llama/Llama-4-Scout-17B-16E-Instruct Provider: Meta Released: 2025-04-02T13:34:17.000Z Updated: 2026-09-12 Architecture: moe Parameters: 109B Context length: 10,485,760 Min vLLM: 0.12.0 Tasks: text Verified hardware: H100, B200, GB200 NVL4, MI300X, MI325X, MI355X, Xeon 6 Summary: Llama 4 Scout 17B-16E MoE model with NVIDIA FP8/FP4 variants, fits on a single GPU with quantization ## Variants - default: precision=bf16, vram_minimum_gb=262 Full precision BF16 - fp8: precision=fp8, vram_minimum_gb=131, model_id=nvidia/Llama-4-Scout-17B-16E-Instruct-FP8 NVIDIA FP8 quantization for Hopper and Blackwell, fits on 1x H100 - nvfp4: precision=nvfp4, vram_minimum_gb=65, model_id=nvidia/Llama-4-Scout-17B-16E-Instruct-NVFP4 NVIDIA NVFP4 quantized weights for Blackwell GPUs - redhat_fp8: precision=fp8, vram_minimum_gb=131, model_id=RedHatAI/Llama-4-Scout-17B-16E-Instruct-FP8-dynamic RedHatAI FP8 (E4M3) with dynamic per-token activation quantization - redhat_nvfp4: precision=nvfp4, vram_minimum_gb=65, model_id=RedHatAI/Llama-4-Scout-17B-16E-Instruct-NVFP4 RedHatAI compressed-tensors NVFP4 quantization for Blackwell GPUs - redhat_fp8_block: precision=fp8, vram_minimum_gb=131, model_id=RedHatAI/Llama-4-Scout-17B-16E-Instruct-FP8-block RedHatAI FP8 block quantization ## Guide ## Overview Llama 4 Scout is Meta's MoE model with 17B active parameters across 16 experts (109B total). NVIDIA provides FP8 and FP4 quantized variants. With FP4 quantization, the model fits on a single B200 GPU — making it one of the most accessible MoE models. ## Prerequisites - Hardware: 1x B200 (FP4), 1x H100 (FP8), 4x GPUs (BF16), 8x MI300X/MI325X/MI355X, or 4x Xeon6/Xeon5 NUMA nodes - vLLM >= 0.12.0 - CUDA Driver >= 575 for GPUs - Docker with NVIDIA Container Toolkit (recommended) for GPUs - License: Must agree to Meta's Llama 4 Scout Community License for GPUs ### AMD ROCm The ROCm wheel requires Python 3.12, ROCm 7.0+, and glibc >= 2.35. Use the ROCm Docker image when your host environment does not meet those requirements. ```bash uv venv --python 3.12 source .venv/bin/activate uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/ ``` Launch on 8x MI300X / MI325X / MI355X with AITER enabled: ```bash export VLLM_ROCM_USE_AITER=1 vllm serve meta-llama/Llama-4-Scout-17B-16E-Instruct \ --tensor-parallel-size 8 \ --max-num-seqs 64 \ --no-enable-prefix-caching \ --max-num-batched-tokens 16384 \ --max-model-len 32000 ``` The first AITER launch may take several minutes while ROCm kernels are compiled and cached. ### pip (Intel Xeon 6 CPUs) For Intel and AMD x86 CPUs, follow the [CPU pre-built wheels](https://docs.vllm.ai/en/latest/getting_started/installation/cpu/#pre-built-wheels) installation instructions. ### Docker (Intel Xeon 6 CPUs) ```bash docker pull vllm/vllm-openai-cpu:latest-x86_64 # For Intel Xeon 6 ``` ### Intel Xeon 6 Deployment via Docker Launch the x86 CPU vLLM Docker container for `meta-llama/Llama-4-Scout-17B-16E-Instruct`: ```bash docker run \ --privileged --ipc=host -p 8000:8000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ vllm/vllm-openai-cpu:latest-x86_64 meta-llama/Llama-4-Scout-17B-16E-Instruct \ --tensor-parallel-size 4 \ --max-model-len 8192 ``` `--max-model-len 8192` is retained because it is the model-specific row argument in the validation workbook. TP/DP and host-specific CPU placement are intentionally not prescribed. The following validation settings are not model requirements and are intentionally not portable CPU recipe defaults: - `--max-num-batched-tokens` and `--max-num-seqs`: scheduler batch/concurrency tuning for workload shape, latency targets, and platform capacity. - `--gpu-memory-utilization`: platform memory-budget tuning. - `--no-enable-prefix-caching`: workload/benchmark cache-policy tuning. - `VLLM_ENGINE_ITERATION_TIMEOUT_S`: runtime operational timeout tuning. Hardware-specific overrides may still use these settings when they are part of a validated platform configuration. ## Client Usage ```python from openai import OpenAI client = OpenAI(base_url="http://localhost:8000/v1", api_key="unused") response = client.chat.completions.create( model="nvidia/Llama-4-Scout-17B-16E-Instruct-FP8", messages=[{"role": "user", "content": "Explain MoE models briefly."}], ) print(response.choices[0].message.content) ``` ## Troubleshooting **FP4 only works on Blackwell:** FP4 quantization requires compute capability 10.0 (B200/GB200). Use FP8 on Hopper. **TP=1 recommended for best throughput:** For maximum throughput per GPU, keep TP=1. Increase TP to 2/4/8 for lower latency. ## References - [Llama-4-Scout-17B-16E-Instruct](https://huggingface.co/meta-llama/Llama-4-Scout-17B-16E-Instruct) - [NVIDIA FP8 variant](https://huggingface.co/nvidia/Llama-4-Scout-17B-16E-Instruct-FP8) - [NVIDIA FP4 variant](https://huggingface.co/nvidia/Llama-4-Scout-17B-16E-Instruct-FP4) --- # microsoft/Phi-4-multimodal-instruct on vLLM Source: https://recipes.vllm.ai/microsoft/Phi-4-multimodal-instruct Provider: Microsoft Released: 2025-02-24T22:33:32.000Z Updated: 2026-09-12 Architecture: dense Parameters: 5.6B Context length: 131,072 Min vLLM: 0.8.0 Tasks: multimodal Verified hardware: Xeon 6 Summary: Microsoft Phi-4 multimodal model supporting text, image, and audio inputs with text outputs. ## Variants - default: precision=bf16, vram_minimum_gb=14 Full precision BF16 ## Recommended base flags ```bash vllm serve 'microsoft/Phi-4-multimodal-instruct' \ --trust-remote-code ``` ## Guide ## Overview Phi-4-multimodal-instruct processes text, image, and audio inputs and generates text. `--trust-remote-code` is retained because the model's multimodal LoRA components require it. ## Prerequisites - Hardware: Intel Xeon 6/Xeon 5 CPUs - vLLM >= 0.8.0 ### Docker (Intel Xeon 6 CPUs) ```bash docker pull vllm/vllm-openai-cpu:latest-x86_64 ``` ## Intel Xeon 6 ```bash vllm serve microsoft/Phi-4-multimodal-instruct \ --trust-remote-code \ --tensor-parallel-size 1 ``` Docker (the image entrypoint is `vllm serve`): ```bash docker run \ --privileged --ipc=host -p 8000:8000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ vllm/vllm-openai-cpu:latest-x86_64 microsoft/Phi-4-multimodal-instruct \ --trust-remote-code \ --tensor-parallel-size 1 ``` The example uses TP=1. Adjust TP/DP for the deployment topology; host-specific CPU placement is intentionally not hard-coded. ## Runtime and Platform Tuning The following settings are intentionally not prescribed as portable CPU model defaults because they depend on the workload, hardware, or runtime environment: - `--max-num-batched-tokens`: scheduler/throughput tuning. - `--max-num-seqs`: concurrency and scheduler-capacity tuning. - `--gpu-memory-utilization`: platform memory-budget tuning. - `--no-enable-prefix-caching`: workload/benchmark cache-policy tuning. - `VLLM_ENGINE_ITERATION_TIMEOUT_S`: operational runtime timeout. These settings may still appear in validated hardware-specific overrides. Tune them at deployment time based on platform resources, workload shape, and latency/throughput goals. ## References - [Model card](https://huggingface.co/microsoft/Phi-4-multimodal-instruct) - [vLLM Phi-4 usage guide](https://docs.vllm.ai/projects/recipes/en/stable/Microsoft/Phi-4.html) - [vLLM CPU installation](https://docs.vllm.ai/en/latest/getting_started/installation/cpu/) --- # microsoft/Phi-4-reasoning on vLLM Source: https://recipes.vllm.ai/microsoft/Phi-4-reasoning Provider: Microsoft Released: 2025-04-09T21:11:28.000Z Updated: 2026-09-12 Architecture: dense Parameters: 14B Context length: 32,768 Min vLLM: 0.4.2 Tasks: text Verified hardware: Xeon 6 Summary: Microsoft Phi-4 14B dense reasoning model with 32K context. ## Variants - default: precision=bf16, vram_minimum_gb=34 Full precision BF16 ## Guide ## Overview Phi-4-reasoning is Microsoft's 14B dense reasoning model. Intel Xeon 6 CPU serving was validated with BF16. ## Prerequisites - Hardware: Intel Xeon 6/Xeon 5 CPUs - vLLM >= 0.4.2 ### Docker (Intel Xeon 6 CPUs) ```bash docker pull vllm/vllm-openai-cpu:latest-x86_64 ``` ## Intel Xeon 6 ```bash vllm serve microsoft/Phi-4-reasoning \ --tensor-parallel-size 1 ``` Docker (the image entrypoint is `vllm serve`): ```bash docker run \ --privileged --ipc=host -p 8000:8000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ vllm/vllm-openai-cpu:latest-x86_64 microsoft/Phi-4-reasoning \ --tensor-parallel-size 1 ``` The example uses TP=1. Adjust TP/DP for the deployment topology; host-specific CPU placement is intentionally not hard-coded. ## Runtime and Platform Tuning The following settings are intentionally not prescribed as portable CPU model defaults because they depend on the workload, hardware, or runtime environment: - `--max-num-batched-tokens`: scheduler/throughput tuning. - `--max-num-seqs`: concurrency and scheduler-capacity tuning. - `--gpu-memory-utilization`: platform memory-budget tuning. - `--no-enable-prefix-caching`: workload/benchmark cache-policy tuning. - `VLLM_ENGINE_ITERATION_TIMEOUT_S`: operational runtime timeout. These settings may still appear in validated hardware-specific overrides. Tune them at deployment time based on platform resources, workload shape, and latency/throughput goals. ## References - [Model card](https://huggingface.co/microsoft/Phi-4-reasoning) - [vLLM CPU installation](https://docs.vllm.ai/en/latest/getting_started/installation/cpu/) --- # openai/gpt-oss-20b on vLLM Source: https://recipes.vllm.ai/openai/gpt-oss-20b Provider: OpenAI Released: 2025-08-04T22:33:29.000Z Updated: 2026-09-12 Architecture: moe Parameters: 21B Context length: 131,072 Min vLLM: 0.10.1 Tasks: text Verified hardware: H100, H200, B200, MI300X, MI325X, MI355X, Arc Pro B60, Arc Pro B70, Xeon 6 Summary: OpenAI's gpt-oss-20b — 21B-total / 3.6B-active MoE reasoning model with native MXFP4 quant; supports GPU, XPU, and Xeon 6 CPU serving ## Variants - default: precision=mxfp4, vram_minimum_gb=16 MXFP4 MoE weights — fits in 16GB VRAM on a single consumer or datacenter GPU ## Guide ## Overview [`gpt-oss-20b`](https://huggingface.co/openai/gpt-oss-20b) is OpenAI's smaller open-weight reasoning model: 21B total parameters with 3.6B activated per token across 32 experts (top-4 routing), shipped with native MXFP4 quantization on the MoE weights. It targets lower-latency and on-device use cases — the model loads in ~16GB of VRAM, runs on a single H100/H200/B200 or AMD MI300X/MI325X/MI355X, and supports the same harmony chat format, configurable reasoning effort (low / medium / high), and built-in tools (browser, python, function calling) as its larger sibling [`gpt-oss-120b`](https://huggingface.co/openai/gpt-oss-120b). Architectural notes: - 24 layers alternating sliding-window (window=128) and full attention. - YaRN rope scaling (factor=32) extending 4K → 131K context. - MXFP4 quant on `model.layers.*.mlp` experts; attention, router, embeddings stay in BF16. ## Prerequisites - Hardware: NVIDIA H100/H200/B200, AMD MI300X/MI325X/MI355X, Intel XPU B60/B70, or 4x Xeon 6 CPUs. - vLLM >= 0.10.1. - CUDA >= 12.8 if building from source for NVIDIA GPUs (must match between install and serving). ### Install vLLM ```bash uv venv source .venv/bin/activate uv pip install vllm --torch-backend=auto ``` ### pip (Intel Xeon 6 CPUs) For Intel and AMD x86 CPUs, follow the [CPU pre-built wheels](https://docs.vllm.ai/en/latest/getting_started/installation/cpu/#pre-built-wheels) installation instructions. ### Docker (Intel Xeon 6 CPUs) ```bash docker pull vllm/vllm-openai-cpu:latest-x86_64 ``` Docker quickstart: ```bash docker run --gpus all -p 8000:8000 --ipc=host vllm/vllm-openai --model openai/gpt-oss-20b ``` AMD ROCm wheels: ```bash uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm ``` Intel XPU (B60 / B70): ```bash docker pull vllm/vllm-openai-xpu:latest ``` ## Launch commands Single GPU (default — works on any 16GB+ card): ```bash vllm serve openai/gpt-oss-20b ``` ### Intel Xeon 6 Choose the tensor parallel size based on the system topology and deployment requirements. Add `--tensor-parallel-size ` when needed; the recipe does not prescribe a Xeon 6 TP value or hard-code NUMA node IDs. ```bash vllm serve openai/gpt-oss-20b \ --max-num-batched-tokens 16384 \ --gpu-memory-utilization 0.8 ``` Docker: ```bash docker run \ --privileged --ipc=host -p 8000:8000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ vllm/vllm-openai-cpu:latest-x86_64 openai/gpt-oss-20b \ --tensor-parallel-size 1 \ --max-num-batched-tokens 16384 \ --gpu-memory-utilization 0.8 \ --tool-call-parser openai \ --enable-auto-tool-choice ``` Blackwell (B200) with FlashInfer MXFP4+MXFP8 MoE: ```bash vllm serve openai/gpt-oss-20b ``` On vLLM v0.28.0+ the FlashInfer TRTLLM MXFP4 MoE path is selected automatically on Blackwell; it can be pinned explicitly with `--moe-backend flashinfer_trtllm --quantization-config.moe.activation mxfp8`. Hopper (H100/H200): same as Blackwell minus `--kv-cache-dtype fp8` and the FlashInfer MoE flags. AMD MI300X/MI325X/MI355X: ```bash export HSA_NO_SCRATCH_RECLAIM=1 export AMDGCN_USE_BUFFER_OPS=0 export VLLM_ROCM_USE_AITER=1 export VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4 vllm serve openai/gpt-oss-20b \ --attention-backend ROCM_AITER_UNIFIED_ATTN \ -cc.pass_config.fuse_rope_kvcache=True \ -cc.use_inductor_graph_partition=True \ --gpu-memory-utilization 0.95 \ --block-size 64 ``` ### Intel XPU(B60, B70) ```bash docker run --device /dev/dri \ -v /dev/dri/by-path:/dev/dri/by-path --shm-size=16g \ --privileged --ipc=host -p 8000:8000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ vllm/vllm-openai-xpu:latest \ openai/gpt-oss-20b \ --max-model-len 8192 ``` ## Tool use The `/v1/responses` endpoint supports built-in tools (browsing, python, MCP). Setup requires `uv pip install gpt-oss` and either Docker (for the Python sandbox) or `PYTHON_EXECUTION_BACKEND=dangerously_use_uv`. For demo tools: ```bash vllm serve openai/gpt-oss-20b --tool-server demo ``` For user-defined function calling (toggle the **Tool Calling** feature above, or pass manually): ```bash vllm serve openai/gpt-oss-20b --tool-call-parser openai --enable-auto-tool-choice ``` ## Reasoning effort gpt-oss exposes three reasoning levels — low, medium, high — selected via the system prompt: ```python from openai import OpenAI client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1") response = client.chat.completions.create( model="openai/gpt-oss-20b", messages=[ {"role": "system", "content": "Reasoning: high"}, {"role": "user", "content": "Explain why eigenvalues matter."}, ], ) print(response.choices[0].message.content) ``` ## Troubleshooting - **Attention sinks dtype error on Blackwell:** pass `--quantization-config.moe.activation mxfp8` and `--kv-cache-dtype fp8`. - **`tl.language not defined`:** make sure no extra Triton (e.g., `pytorch-triton`) is installed alongside vLLM's bundled Triton. - **Harmony vocab download failure:** pre-download tiktoken files and set `TIKTOKEN_ENCODINGS_BASE`. ## References - [Model card — gpt-oss-20b](https://huggingface.co/openai/gpt-oss-20b) - [Sibling — gpt-oss-120b](https://huggingface.co/openai/gpt-oss-120b) - [OpenAI announcement](https://openai.com/index/introducing-gpt-oss/) - [gpt-oss model card paper (arXiv:2508.10925)](https://arxiv.org/abs/2508.10925) - [vLLM gpt-oss cookbook](https://cookbook.openai.com/articles/gpt-oss/run-vllm) --- # openai/whisper-large-v3 on vLLM Source: https://recipes.vllm.ai/openai/whisper-large-v3 Provider: OpenAI Released: 2023-11-07T18:41:14.000Z Updated: 2026-09-12 Architecture: dense Parameters: 1.55B Context length: 448 Min vLLM: 0.7.0 Tasks: multimodal Verified hardware: Xeon 6 Summary: Whisper Large v3 speech recognition model, validated for Intel Xeon 6 CPU serving. ## Variants - default: precision=bf16, vram_minimum_gb=4 BF16 serving configuration validated on Intel Xeon 6 ## Guide ## Overview Whisper Large v3 is OpenAI's speech recognition model. vLLM exposes transcription through the OpenAI-compatible audio transcription endpoint. ## Prerequisites - Hardware: Intel Xeon 6/Xeon 5 CPUs - vLLM >= 0.7.0 ### Docker (Intel Xeon 6 CPUs) ```bash docker pull vllm/vllm-openai-cpu:latest-x86_64 ``` ## Intel Xeon 6 ```bash vllm serve openai/whisper-large-v3 \ --tensor-parallel-size 1 ``` Docker (the image entrypoint is `vllm serve`): ```bash docker run \ --privileged --ipc=host -p 8000:8000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ vllm/vllm-openai-cpu:latest-x86_64 openai/whisper-large-v3 \ --tensor-parallel-size 1 ``` The example uses TP=1. Adjust TP/DP for the deployment topology. ## Runtime and Platform Tuning The following settings are intentionally not prescribed as portable CPU model defaults because they depend on the workload, hardware, or runtime environment: - `--max-num-batched-tokens`: scheduler/throughput tuning. - `--max-num-seqs`: concurrency and scheduler-capacity tuning. - `--gpu-memory-utilization`: platform memory-budget tuning. - `--no-enable-prefix-caching`: workload/benchmark cache-policy tuning. - `VLLM_ENGINE_ITERATION_TIMEOUT_S`: operational runtime timeout. These settings may still appear in validated hardware-specific overrides. Tune them at deployment time based on platform resources, workload shape, and latency/throughput goals. ## References - [Model card](https://huggingface.co/openai/whisper-large-v3) - [vLLM Whisper implementation](https://docs.vllm.ai/en/latest/api/vllm/model_executor/models/whisper/) - [vLLM CPU installation](https://docs.vllm.ai/en/latest/getting_started/installation/cpu/) --- # zai-org/glm-4-9b-hf on vLLM Source: https://recipes.vllm.ai/zai-org/glm-4-9b-hf Provider: Z.ai Released: 2025-01-16T06:02:41.000Z Updated: 2026-09-12 Architecture: dense Parameters: 9B Context length: 8,192 Min vLLM: 0.6.5 Tasks: text Verified hardware: Xeon 6 Summary: GLM-4 9B dense base language model with 8K context. ## Variants - default: precision=bf16, vram_minimum_gb=22 Full precision BF16 ## Guide ## Overview GLM-4-9B is the dense base model in the GLM-4 9B family. Intel Xeon 6 CPU serving was validated with BF16. ## Prerequisites - Hardware: Intel Xeon 6/Xeon 5 CPUs - vLLM >= 0.6.5 ### Docker (Intel Xeon 6 CPUs) ```bash docker pull vllm/vllm-openai-cpu:latest-x86_64 ``` ## Intel Xeon 6 ```bash vllm serve zai-org/glm-4-9b-hf \ --tensor-parallel-size 1 ``` Docker (the image entrypoint is `vllm serve`): ```bash docker run \ --privileged --ipc=host -p 8000:8000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ vllm/vllm-openai-cpu:latest-x86_64 zai-org/glm-4-9b-hf \ --tensor-parallel-size 1 ``` The example uses TP=1. Adjust TP/DP for the deployment topology; host-specific CPU placement is intentionally not hard-coded. ## Runtime and Platform Tuning The following settings are intentionally not prescribed as portable CPU model defaults because they depend on the workload, hardware, or runtime environment: - `--max-num-batched-tokens`: scheduler/throughput tuning. - `--max-num-seqs`: concurrency and scheduler-capacity tuning. - `--gpu-memory-utilization`: platform memory-budget tuning. - `--no-enable-prefix-caching`: workload/benchmark cache-policy tuning. - `VLLM_ENGINE_ITERATION_TIMEOUT_S`: operational runtime timeout. These settings may still appear in validated hardware-specific overrides. Tune them at deployment time based on platform resources, workload shape, and latency/throughput goals. ## References - [Model card](https://huggingface.co/zai-org/glm-4-9b-hf) - [vLLM CPU installation](https://docs.vllm.ai/en/latest/getting_started/installation/cpu/) --- # Qwen/Qwen3-32B on vLLM Source: https://recipes.vllm.ai/Qwen/Qwen3-32B Provider: Qwen Released: 2025-04-27T03:52:59.000Z Updated: 2026-09-10 Architecture: dense Parameters: 32B Context length: 40,960 Min vLLM: 0.8.5 Tasks: text Verified hardware: TPU v6e (Trillium), TPU v7 (Ironwood), Arc Pro B60, Arc Pro B70 Summary: Qwen3 32B dense model with hybrid thinking/non-thinking modes — verified on TPU v6e (Trillium). ## Variants - default: precision=bf16, vram_minimum_gb=77 Full precision BF16 — TPU v6e 2x2 (4 chips, TP=4) or a single 80GB+ GPU - fp8: precision=fp8, vram_minimum_gb=39, model_id=Qwen/Qwen3-32B-FP8 Qwen official FP8 checkpoint — single 40 GB GPU - awq: precision=int4, vram_minimum_gb=20, model_id=Qwen/Qwen3-32B-AWQ AWQ 4-bit quantized weights — fits on a single 24 GB GPU - redhat_fp8: precision=fp8, vram_minimum_gb=39, model_id=RedHatAI/Qwen3-32B-FP8-dynamic RedHatAI FP8 (E4M3) with dynamic per-token activation quantization - redhat_fp8_block: precision=fp8, vram_minimum_gb=39, model_id=RedHatAI/Qwen3-32B-FP8-block RedHatAI FP8 block quantization - nvfp4: precision=nvfp4, vram_minimum_gb=22, model_id=RedHatAI/Qwen3-32B-NVFP4 RedHatAI NVFP4 quantization - redhat_nvfp4: precision=nvfp4, vram_minimum_gb=22, model_id=RedHatAI/Qwen3-32B-NVFP4A16 RedHatAI NVFP4 with BF16 activations ## Guide ## Overview Qwen3-32B is the flagship dense model in the Qwen3 series, with hybrid thinking / non-thinking modes. BF16 is the training precision and gives the best accuracy. Toggle thinking mode per request with `enable_thinking` in `chat_template_kwargs`, or set the **Reasoning** feature to add the Qwen3 reasoning parser so `` content is split into a separate field. ## Prerequisites ### NVIDIA ```bash uv venv source .venv/bin/activate uv pip install -U vllm --torch-backend auto ``` ### TPU (Trillium / v6e) Use the official vLLM TPU image. The 32B model is served on a `v6e-4` slice (`--topology 2x2`, 4 chips, TP=4). ```bash export HF_HOME=/dev/shm export HF_TOKEN= ``` ### Intel XPU (B60 / B70) Use the official vLLM XPU image. ```bash docker pull vllm/vllm-openai-xpu:latest ``` ## Deployment Configurations ### TPU v6e (Trillium, 2x2 slice, TP=4) Verified end-to-end on a 4-chip v6e slice. ```bash export HF_HOME=/dev/shm export HF_TOKEN= export TP=4 export MAX_MODEL_LEN=4096 vllm serve Qwen/Qwen3-32B \ --seed 42 \ --gpu-memory-utilization 0.98 \ --max-num-batched-tokens 2048 \ --max-num-seqs 256 \ --tensor-parallel-size $TP \ --max-model-len $MAX_MODEL_LEN ``` ### TPU v7 (Ironwood, single host, TP=2) Verified end-to-end on a single Ironwood host (`tpu7x-standard-1t`, topology `1x1x1`). Served via the OpenAI API server entrypoint. ```bash export HF_HOME=/data export HUGGING_FACE_HUB_TOKEN= export LAYOUT_Q_PROJ_AS_NDH=1 export USE_BATCHED_RPA_KERNEL=1 vllm serve Qwen/Qwen3-32B \ --host 0.0.0.0 \ --port 8000 \ --seed 42 \ --tensor-parallel-size 2 \ --data-parallel-size 1 \ --max-model-len 9216 \ --max-num-batched-tokens 4096 \ --max-num-seqs 135 \ --gpu-memory-utilization 0.98 \ --block-size 256 \ --kv-cache-dtype fp8 \ --no-enable-prefix-caching \ --async-scheduling ``` > Ironwood support ships in the nightly TPU image (e.g. `vllm/vllm-tpu:nightly-20260330-2f76400-8c0b626`). ### Single node (BF16, TP=2) ```bash vllm serve Qwen/Qwen3-32B \ --host 0.0.0.0 \ --port 8000 \ --tensor-parallel-size 2 ``` ### Intel XPU (B60, B70) Validated on 4× Intel Arc Pro B60 / B70 (B60 24 GB, B70 32 GB per card) with the official vLLM XPU image `vllm/vllm-openai-xpu:latest`. ```bash docker run --device /dev/dri \ -v /dev/dri/by-path:/dev/dri/by-path --shm-size=16g \ --privileged --ipc=host -p 8000:8000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ vllm/vllm-openai-xpu:latest \ Qwen/Qwen3-32B \ --gpu-memory-utilization 0.90 \ --tensor-parallel-size 4 \ --max-model-len 8192 ``` ## Configuration Tips - `--max-model-len 4096` keeps memory in check on a 4-chip v6e slice; native context is 40K — raise it if you have headroom. - vLLM uses 90% of device memory by default; the TPU recipe pushes `--gpu-memory-utilization 0.98` to maximize KV cache. - `--max-num-batched-tokens` and `--max-num-seqs` tune batch throughput vs latency. - For thinking mode Qwen recommends `temperature=0.6, top_p=0.95, top_k=20`; for non-thinking, `temperature=0.7, top_p=0.8`. ## Benchmarking Launch the server with `--no-enable-prefix-caching` for consistent measurements. ```bash vllm bench serve \ --host 0.0.0.0 \ --port 8000 \ --model Qwen/Qwen3-32B \ --dataset-name random \ --random-input-len 1024 \ --random-output-len 1024 \ --num-prompts 128 ``` ## References - [Model card](https://huggingface.co/Qwen/Qwen3-32B) - [Qwen3 on Trillium (v6e)](https://github.com/AI-Hypercomputer/tpu-recipes/tree/main/inference/trillium/vLLM/Qwen3) - [Qwen3-32B on Ironwood (v7)](https://github.com/AI-Hypercomputer/tpu-recipes/tree/main/inference/ironwood/vLLM/Qwen3-32B) --- # Qwen/Qwen3-Next-80B-A3B-Instruct on vLLM Source: https://recipes.vllm.ai/Qwen/Qwen3-Next-80B-A3B-Instruct Provider: Qwen Released: 2025-09-09T15:40:56.000Z Updated: 2026-09-10 Architecture: moe Parameters: 80B Context length: 262,144 Min vLLM: 0.17.0 Tasks: text Verified hardware: H200, MI300X, MI325X, MI355X Summary: Advanced Qwen3-Next MoE model (80B total / 3B active) with hybrid attention, highly sparse experts, and multi-token prediction. ## Variants - default: precision=bf16, vram_minimum_gb=192 Full precision BF16 — fits on 4x H200/H20/A100/A800 - fp8: precision=fp8, vram_minimum_gb=96, model_id=Qwen/Qwen3-Next-80B-A3B-Instruct-FP8 Qwen official FP8 checkpoint — recommended on SM90/SM100 - nvfp4: precision=nvfp4, vram_minimum_gb=48, model_id=nvidia/Qwen3-Next-80B-A3B-Instruct-NVFP4 NVIDIA NVFP4 quantized weights for Blackwell GPUs - redhat_fp8: precision=fp8, vram_minimum_gb=96, model_id=RedHatAI/Qwen3-Next-80B-A3B-Instruct-FP8-dynamic RedHatAI FP8 (E4M3) with dynamic per-token activation quantization - redhat_fp8_plain: precision=fp8, vram_minimum_gb=96, model_id=RedHatAI/Qwen3-Next-80B-A3B-Instruct-FP8 RedHatAI FP8 quantization - redhat_fp8_block: precision=fp8, vram_minimum_gb=96, model_id=RedHatAI/Qwen3-Next-80B-A3B-Instruct-FP8-block RedHatAI FP8 block quantization - redhat_nvfp4: precision=nvfp4, vram_minimum_gb=48, model_id=RedHatAI/Qwen3-Next-80B-A3B-Instruct-NVFP4 RedHatAI NVFP4 quantization ## Guide ## Overview [Qwen3-Next](https://qwen.ai/blog?id=4074cca80393150c248e508aa62983f9cb7d27cd&from=research.latest-advancements-list) is an advanced LLM from the Qwen team featuring: - A hybrid attention mechanism - A highly sparse Mixture-of-Experts (MoE) structure - Training-stability-friendly optimizations - A multi-token prediction mechanism for faster inference ## Prerequisites ```bash uv venv source .venv/bin/activate uv pip install -U vllm --torch-backend auto ``` ## Deployment Configurations Launch on 4x H200/H20 or 4x A100/A800 GPUs. ### Basic Multi-GPU (BF16) ```bash vllm serve Qwen/Qwen3-Next-80B-A3B-Instruct \ --tensor-parallel-size 4 \ --served-model-name qwen3-next \ --enable-prefix-caching ``` If you hit `torch.AcceleratorError: CUDA error: an illegal memory access was encountered`, add `--compilation_config.cudagraph_mode=PIECEWISE`. ### FP8 (SM90/SM100) ```bash vllm serve Qwen/Qwen3-Next-80B-A3B-Instruct-FP8 \ --tensor-parallel-size 4 \ --enable-prefix-caching ``` On SM100, accelerate with the FlashInfer TRTLLM MoE kernel: ```bash VLLM_USE_DEEP_GEMM=0 \ vllm serve Qwen/Qwen3-Next-80B-A3B-Instruct-FP8 \ --tensor-parallel-size 4 \ --moe-backend flashinfer_trtllm \ --attention-backend FLASH_ATTN ``` ### MTP (Multi-Token Prediction) ```bash vllm serve Qwen/Qwen3-Next-80B-A3B-Instruct \ --tokenizer-mode auto --gpu-memory-utilization 0.8 \ --speculative-config '{"method": "qwen3_next_mtp", "num_speculative_tokens": 2}' \ --tensor-parallel-size 4 --no-enable-chunked-prefill ``` ### Tool / Function Calling ```bash vllm serve ... --tool-call-parser hermes --enable-auto-tool-choice ``` ### AMD (MI300X/MI325X/MI355X) ```bash uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/0.14.1/rocm700 ``` ```bash SAFETENSORS_FAST_GPU=1 \ VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 \ vllm serve Qwen/Qwen3-Next-80B-A3B-Instruct \ --tensor-parallel-size 4 \ --max-model-len 32768 \ --no-enable-prefix-caching \ --trust-remote-code ``` ## Client Usage Benchmark: ```bash vllm bench serve \ --backend vllm \ --model Qwen/Qwen3-Next-80B-A3B-Instruct \ --served-model-name qwen3-next \ --endpoint /v1/completions \ --dataset-name random \ --random-input 2048 \ --random-output 1024 \ --max-concurrency 10 \ --num-prompt 100 ``` ## Troubleshooting - **Sub-optimal MoE performance warning**: Tune the MoE Triton kernel with [benchmark_moe](https://github.com/vllm-project/vllm/blob/main/benchmarks/kernels/benchmark_moe.py), then set `VLLM_TUNED_CONFIG_FOLDER` to the directory containing the generated config. - **IMA error in DP mode**: add `--compilation_config.cudagraph_mode=PIECEWISE`. - For more parallel topologies, see the [Data Parallel Deployment docs](https://docs.vllm.ai/en/latest/serving/data_parallel_deployment.html). ## References - [Model card](https://huggingface.co/Qwen/Qwen3-Next-80B-A3B-Instruct) - [FP8 checkpoint](https://huggingface.co/Qwen/Qwen3-Next-80B-A3B-Instruct-FP8) - [Qwen3-Next blog](https://qwen.ai/blog?id=4074cca80393150c248e508aa62983f9cb7d27cd) --- # deepseek-ai/DeepSeek-V3 on vLLM Source: https://recipes.vllm.ai/deepseek-ai/DeepSeek-V3 Provider: DeepSeek Released: 2024-12-25T12:52:23.000Z Updated: 2026-09-10 Architecture: moe Parameters: 671B Context length: 163,840 Min vLLM: 0.12.0 Tasks: text Verified hardware: H200, B200, GB200 NVL4, MI300X, MI325X, MI350X, MI355X Summary: DeepSeek-V3 is a 671B-parameter Mixture-of-Experts model with native FP8 weights and strong reasoning, coding, and math capabilities. ## Variants - default: precision=fp8, vram_minimum_gb=805 Native FP8 weights on 8xH200 (recommended) - v3_0324: precision=fp8, vram_minimum_gb=805, model_id=deepseek-ai/DeepSeek-V3-0324 March 2025 DeepSeek-V3 refresh (DeepSeek-V3-0324) - fp4: precision=fp4, vram_minimum_gb=403, model_id=nvidia/DeepSeek-V3-FP4 NVIDIA FP4 quantized weights for Blackwell (e.g. 4xB200) ## Recommended base flags ```bash vllm serve 'deepseek-ai/DeepSeek-V3' \ --trust-remote-code \ --enable-expert-parallel ``` ## Guide ## Overview DeepSeek-V3 is a 671B-parameter Mixture-of-Experts model (37B activated per token) shipped with native FP8 weights. It shares its architecture with DeepSeek-R1, so the same launch recipes apply to both models. DeepSeek publishes a refreshed checkpoint as [`DeepSeek-V3-0324`](https://huggingface.co/deepseek-ai/DeepSeek-V3-0324), available here as the **V3 0324** variant. For Blackwell GPUs, NVIDIA publishes an FP4 quantized variant (`nvidia/DeepSeek-V3-FP4` / `nvidia/DeepSeek-R1-FP4`) that runs on fewer GPUs. ## Prerequisites - **Hardware (FP8)**: 8x H200 (CUDA) or 8x MI300X / MI325X / MI355X (ROCm) - **Hardware (FP4)**: 4x B200 GPUs - **vLLM**: Current stable release ### CUDA ```bash uv venv source .venv/bin/activate uv pip install -U vllm --torch-backend auto ``` ### ROCm (MI300X, MI325X, MI355X) Requires Python 3.12, ROCm 7.2.1, and glibc >= 2.35. ```bash uv venv --python 3.12 source .venv/bin/activate uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/ ``` ## Client Usage ### 8xH200 / 8xMI300X (FP8) Tensor Parallel + Expert Parallel (TP8+EP) — CUDA: ```bash vllm serve deepseek-ai/DeepSeek-V3 \ --trust-remote-code \ --tensor-parallel-size 8 \ --enable-expert-parallel ``` Tensor Parallel + Expert Parallel (TP8+EP) — ROCm: ```bash export VLLM_ROCM_USE_AITER=1 vllm serve deepseek-ai/DeepSeek-V3 \ --trust-remote-code \ --tensor-parallel-size 8 \ --enable-expert-parallel ``` Data Parallel + Expert Parallel (DP8+EP) — CUDA: ```bash vllm serve deepseek-ai/DeepSeek-V3 \ --trust-remote-code \ --data-parallel-size 8 \ --enable-expert-parallel ``` ### 8xMI350X / 8xMI355X (DPA+TP) Each GPU runs its own attention replica with a private KV cache, while the MoE experts stay tensor-sharded across all eight GPUs. Aggregate KV-cache capacity is roughly 8x that of a single TP8 replica, useful for workloads benefiting from larger KV capacities. Engine (all eight ranks in one process, listening on 8100): ```bash export VLLM_ROCM_USE_AITER=1 export VLLM_ENGINE_READY_TIMEOUT_S=1800 vllm serve deepseek-ai/DeepSeek-V3-0324 \ --trust-remote-code \ --port 8100 \ --data-parallel-size 8 \ --tensor-parallel-size 1 ``` Router (client-facing on 8000, spreads requests across the eight DP ranks): ```bash uv pip install vllm-router vllm-router \ --host 0.0.0.0 \ --port 8000 \ --policy consistent_hash \ --intra-node-data-parallel-size 8 \ --worker-urls http://localhost:8100 \ --worker-startup-timeout-secs 1800 ``` Notes: - `--intra-node-data-parallel-size` must match the engine's `--data-parallel-size`. A mismatch silently under-uses ranks. - `consistent_hash` is the policy that makes this worthwhile for chat: it pins a conversation to the rank that already holds its prefix, so the larger aggregate cache actually gets hit. `round_robin` scatters turns of the same conversation across ranks and throws the prefix away. - Start the engine first, then the router; `--worker-startup-timeout-secs 1800` covers the engine's weight load either way. - Send client traffic to the router on port 8000, never to 8100 directly. ### 4xB200 (FP4) On vLLM v0.28.0 and later, the FlashInfer MoE kernels are selected automatically for both FP4 and FP8 on Blackwell — no environment variables are needed. To pin the backend explicitly, pass `--moe-backend flashinfer_trtllm`. Tensor Parallel + Expert Parallel (TP4+EP): ```bash CUDA_VISIBLE_DEVICES=0,1,2,3 vllm serve nvidia/DeepSeek-V3-FP4 \ --trust-remote-code \ --tensor-parallel-size 4 \ --enable-expert-parallel ``` Data Parallel + Expert Parallel (DP4+EP): ```bash CUDA_VISIBLE_DEVICES=0,1,2,3 vllm serve nvidia/DeepSeek-V3-FP4 \ --trust-remote-code \ --data-parallel-size 4 \ --enable-expert-parallel ``` ## Benchmarking For benchmarking, prefix caching is disabled by default in vLLM — no extra server flag is needed. ```bash # Prompt-heavy benchmark (8k/1k) vllm bench serve \ --model deepseek-ai/DeepSeek-V3 \ --dataset-name random \ --random-input-len 8000 \ --random-output-len 1000 \ --request-rate 10000 \ --num-prompts 16 \ --ignore-eos ``` Test different workloads by adjusting input/output lengths: - Prompt-heavy: 8000 input / 1000 output - Decode-heavy: 1000 input / 8000 output - Balanced: 1000 input / 1000 output ## Troubleshooting - **Disaggregated Serving with Wide EP (Experimental GB200)**: See [vLLM issue #33583](https://github.com/vllm-project/vllm/issues/33583) and the [vLLM blog post](https://blog.vllm.ai/2026/02/03/dsr1-gb200-part1.html) for GB200 disaggregated serving recipes. ## References - [DeepSeek-V3 on Hugging Face](https://huggingface.co/deepseek-ai/DeepSeek-V3) - [DeepSeek-V3-0324 on Hugging Face](https://huggingface.co/deepseek-ai/DeepSeek-V3-0324) - [NVIDIA DeepSeek-V3-FP4](https://huggingface.co/nvidia/DeepSeek-V3-FP4) - [vLLM Expert Parallelism docs](https://docs.vllm.ai/en/latest/serving/expert_parallel_deployment.html) - [vLLM Data Parallel deployment docs](https://docs.vllm.ai/en/latest/serving/data_parallel_deployment.html) --- # nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16 on vLLM Source: https://recipes.vllm.ai/nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16 Provider: NVIDIA Released: 2025-12-04T03:37:11.000Z Updated: 2026-09-10 Architecture: moe Parameters: 30B Context length: 262,144 Min vLLM: 0.17.0 Tasks: text Verified hardware: H100, H200, Arc Pro B60, Arc Pro B70 Summary: NVIDIA Nemotron-3-Nano Mamba-hybrid MoE (30B total / ~3B active) with BF16 and FP8 variants ## Variants - default: precision=bf16, vram_minimum_gb=72 BF16 weights - fp8: precision=fp8, vram_minimum_gb=35, model_id=nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-FP8 FP8 weights + FP8 KV cache ## Recommended base flags ```bash vllm serve 'nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16' \ --trust-remote-code \ --async-scheduling ``` ## Guide ## Overview NVIDIA Nemotron-3-Nano-30B-A3B is a hybrid-Mamba MoE model (30B total, ~3B active) with FP8 and BF16 variants. It supports DGX Spark and Jetson Thor in addition to standard Hopper/Blackwell servers. ## Prerequisites - Hardware: 1x H100/H200 or comparable; DGX Spark, Jetson Thor, and Intel Arc Pro B60/B70 supported - vLLM >= 0.17.0 (0.28.0 recommended for full support) - Docker with NVIDIA Container Toolkit (recommended) ### Pull Docker Image ```bash docker pull --platform linux/amd64 vllm/vllm-openai:v0.28.0 docker tag vllm/vllm-openai:v0.28.0 vllm/vllm-openai:deploy ``` DGX Spark users can build from source (see README) or use the NGC image: ```bash docker pull nvcr.io/nvidia/vllm:25.12.post1-py3 ``` Jetson Thor: ```bash docker pull ghcr.io/nvidia-ai-iot/vllm:latest-jetson-thor ``` Intel XPU (B60 / B70): ```bash docker pull vllm/vllm-openai-xpu:latest ``` ## Launch commands FP8 with FlashInfer MoE backend (Blackwell/Hopper): ```bash vllm serve nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-FP8 \ --trust-remote-code \ --async-scheduling \ --kv-cache-dtype fp8 \ --tensor-parallel-size 1 \ --moe-backend flashinfer_cutlass ``` BF16 (with reasoning + tool parsers — typical for Spark/Thor): ```bash wget https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16/resolve/main/nano_v3_reasoning_parser.py vllm serve nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16 \ --max-num-seqs 8 \ --tensor-parallel-size 1 \ --max-model-len 262144 \ --trust-remote-code \ --enable-auto-tool-choice \ --tool-call-parser qwen3_coder \ --reasoning-parser-plugin nano_v3_reasoning_parser.py \ --reasoning-parser nano_v3 ``` ### Intel XPU (B60, B70) Validated on 4× Intel Arc Pro B60 / B70 (B60 24 GB, B70 32 GB per card) with the official vLLM XPU image `vllm/vllm-openai-xpu:latest`. FP8: ```bash docker run --device /dev/dri \ -v /dev/dri/by-path:/dev/dri/by-path --shm-size=16g \ --privileged --ipc=host -p 8000:8000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ vllm/vllm-openai-xpu:latest \ nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-FP8 \ --trust-remote-code \ --async-scheduling \ --kv-cache-dtype fp8 \ --tensor-parallel-size 4 \ --enforce-eager \ --max-model-len 8192 ``` BF16: ```bash docker run --device /dev/dri \ -v /dev/dri/by-path:/dev/dri/by-path --shm-size=16g \ --privileged --ipc=host -p 8000:8000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ vllm/vllm-openai-xpu:latest \ nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16 \ --trust-remote-code \ --async-scheduling \ --kv-cache-dtype auto \ --tensor-parallel-size 4 \ --enforce-eager \ --enable-auto-tool-choice \ --tool-call-parser qwen3_coder \ --reasoning-parser-plugin nano_v3_reasoning_parser.py \ --reasoning-parser nano_v3 \ --max-model-len 8192 ``` Key flags: - `kv-cache-dtype fp8` for FP8 variant, `auto` for BF16 - `async-scheduling` reduces host overhead between decode steps - `mamba-ssm-cache-dtype float32` for best accuracy, `float16` for speed - `max-num-seqs` cap to match client concurrency for lower per-user latency ## Benchmarking ```bash vllm bench serve \ --model nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-FP8 \ --trust-remote-code \ --dataset-name random \ --random-input-len 1024 --random-output-len 1024 \ --num-warmups 20 \ --ignore-eos \ --max-concurrency 1024 \ --num-prompts 2048 ``` ## Troubleshooting - Use `--kv-cache-dtype fp8` only with the FP8 checkpoint. - Balance TP and `--max-num-seqs` for throughput vs. per-user latency. ## References - [Nemotron-3-Nano-30B-A3B-BF16](https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16) - [Nemotron-3-Nano-30B-A3B-FP8](https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-FP8) --- # nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16 on vLLM Source: https://recipes.vllm.ai/nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16 Provider: NVIDIA Released: 2026-06-03T14:50:04.000Z Updated: 2026-09-10 Architecture: moe Parameters: 550B Context length: 262,144 Min vLLM: 0.22.0 Tasks: text Verified hardware: B200 Summary: NVIDIA Nemotron 3 Ultra hybrid Transformer-Mamba MoE model for long-context agentic reasoning, coding, and tool use. ## Variants - default: precision=bf16, vram_minimum_gb=1320 BF16 weights - fp4: precision=nvfp4, vram_minimum_gb=330, model_id=nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-NVFP4 NVFP4 weights ## Recommended base flags ```bash vllm serve 'nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16' \ --served-model-name \ nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B \ --trust-remote-code \ --kv-cache-dtype \ fp8 \ --max-num-seqs \ 16 \ --max-model-len \ 262144 \ --gpu-memory-utilization \ 0.90 \ --max-num-batched-tokens \ 32768 \ --enable-flashinfer-autotune \ --async-scheduling \ --mamba-backend \ triton \ --mamba-ssm-cache-dtype \ float32 ``` Environment: ```bash VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS='1' ``` ## Guide ## Overview NVIDIA Nemotron 3 Ultra is a 550B total / 55B active hybrid Transformer-Mamba MoE model built for long-running agentic reasoning, coding, research, and tool use. ## Launch command ```bash export VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=1 vllm serve nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-NVFP4 \ --served-model-name nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B \ --trust-remote-code \ --tensor-parallel-size 8 \ --kv-cache-dtype fp8 \ --max-num-seqs 16 \ --max-model-len 262144 \ --gpu-memory-utilization 0.90 \ --max-num-batched-tokens 32768 \ --enable-flashinfer-autotune \ --async-scheduling \ --speculative-config '{"method":"mtp","num_speculative_tokens":5}' \ --mamba-backend triton \ --mamba-ssm-cache-dtype float32 \ --reasoning-parser nemotron_v3 \ --enable-auto-tool-choice \ --tool-call-parser qwen3_coder ``` ## References - [BF16 model](https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16) - [NVFP4 model](https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-NVFP4) --- # tencent/Hunyuan-A13B-Instruct on vLLM Source: https://recipes.vllm.ai/tencent/Hunyuan-A13B-Instruct Provider: Hunyuan (Tencent) Released: 2025-06-25T12:39:52.000Z Updated: 2026-09-10 Architecture: moe Parameters: 80B Context length: 32,768 Min vLLM: 0.11.0 Tasks: text Verified hardware: MI300X, MI325X, MI355X Summary: Tencent Hunyuan A13B instruct-tuned MoE language model with AITER-accelerated AMD ROCm deployment ## Variants - default: precision=bf16, vram_minimum_gb=580 Full precision BF16 — 2x GPU (TP=2) on AMD MI300X/MI325X/MI355X - fp8: precision=fp8, vram_minimum_gb=96, model_id=tencent/Hunyuan-A13B-Instruct-FP8 FP8 quantized weights for Hopper/Blackwell GPUs ## Recommended base flags ```bash vllm serve 'tencent/Hunyuan-A13B-Instruct' \ --trust-remote-code ``` ## Guide ## Overview Hunyuan-A13B-Instruct is Tencent's instruct-tuned Hunyuan MoE model. This recipe covers deployment on AMD ROCm GPUs (MI300X / MI325X / MI355X) with [AITER](https://github.com/ROCm/aiter) acceleration enabled via `VLLM_ROCM_USE_AITER=1`. ## Prerequisites - **vLLM version:** ROCm build - **Python:** 3.12 - **Hardware:** AMD MI300X / MI325X / MI355X - **ROCm:** 7.0+, glibc >= 2.35 (or use Docker) ### Install vLLM (ROCm) ```bash uv venv source .venv/bin/activate uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/ ``` If the environment does not meet the Python/ROCm/glibc requirements, use the Docker-based setup from the vLLM install docs. ## Launching the Server ```bash export VLLM_ROCM_USE_AITER=1 vllm serve tencent/Hunyuan-A13B-Instruct \ --tensor-parallel-size 2 \ --trust-remote-code ``` ## Benchmarking ```bash vllm bench serve \ --model "tencent/Hunyuan-A13B-Instruct" \ --dataset-name random \ --random-input-len 8000 \ --random-output-len 1000 \ --request-rate 10000 \ --num-prompts 16 \ --ignore-eos ``` ## Troubleshooting - **First launch delay:** AITER JIT-compiles optimized kernels on first launch, which can take several minutes. Subsequent runs use cached kernels. - **Environment mismatch:** If wheel install fails, fall back to the vLLM ROCm Docker image. ## References - [Model card](https://huggingface.co/tencent/Hunyuan-A13B-Instruct) - [AITER](https://github.com/ROCm/aiter) - [vLLM install docs](https://docs.vllm.ai/en/latest/getting_started/installation/gpu/) --- # Google/gemma-4-12B-it on vLLM Source: https://recipes.vllm.ai/Google/gemma-4-12B-it Provider: Google Released: 2026-05-23T01:17:15.000Z Updated: 2026-09-09 Architecture: dense Parameters: 12B Context length: 131,072 Min vLLM: 0.23.0 Tasks: multimodal, text Verified hardware: H100, Arc Pro B60, Arc Pro B70 Summary: Google's encoder-free unified Gemma 4 dense model (12B) with native text, image, and audio, plus thinking mode and tool-use protocol. ## Variants - default: precision=bf16, vram_minimum_gb=29 Full BF16 — single 40 GB+ NVIDIA GPU - nvfp4: precision=nvfp4, vram_minimum_gb=8, model_id=RedHatAI/gemma-4-12B-it-NVFP4 RedHatAI compressed-tensors NVFP4 quantization for Blackwell GPUs - w4a16: precision=int4, vram_minimum_gb=9, model_id=google/gemma-4-12B-it-qat-w4a16-ct W4A16 QAT (4-bit weights, 16-bit activations, group_size=32) via compressed-tensors — runs on any GPU - fp8: precision=fp8, vram_minimum_gb=15, model_id=RedHatAI/gemma-4-12B-it-FP8-Dynamic RedHatAI FP8 (E4M3) with dynamic per-token activation quantization ## Guide ## Overview [Gemma 4](https://ai.google.dev/gemma/docs) is Google's most capable open model family, featuring a unified multimodal architecture that natively processes text, images, and audio. The **12B Unified** model is *encoder-free*: instead of dedicated vision/audio encoders, it projects raw image patches and audio waveforms directly into the decoder's embedding space through lightweight linear layers, so every modality flows into a single decoder-only transformer. Gemma 4 models support structured thinking/reasoning, function calling with a custom tool-use protocol, and dynamic vision resolution — all available through vLLM's OpenAI-compatible API. ### Key Features - **Encoder-free multimodality**: Text, image, and audio inputs handled by a single decoder-only transformer — no separate encoders to load, reducing multimodal latency. - **Dual Attention**: Alternating sliding-window (local, 1024 tokens) and global attention; the final layer is always global, with unified Keys/Values and Proportional RoPE on global layers for long-context efficiency. - **Thinking Mode**: Structured reasoning via `<|channel>thought\n...` delimiters. - **Function Calling**: Custom tool-call protocol with dedicated special tokens. - **Native System Prompt Support**: First-class `system` role. ### Supported Variants Dense: - `google/gemma-4-E2B-it` (effective 2B) - `google/gemma-4-E4B-it` (effective 4B) - `google/gemma-4-12B-it` (11.95B, encoder-free) - `google/gemma-4-31B-it` (31B) MoE: - `google/gemma-4-26B-A4B-it` (26B total / 4B active) > **Context length**: this recipe pins `context_length` to the value in `config.json` (`max_position_embeddings = 131072`, i.e. 128K). The Gemma 4 model card markets the 12B at up to 256K — raise `--max-model-len` only if your vLLM build accepts it. TPU support is provided through [vLLM TPU](https://github.com/vllm-project/tpu-inference) with recipes for [Trillium](https://github.com/AI-Hypercomputer/tpu-recipes/tree/main/inference/trillium/vLLM/Gemma4) and [Ironwood](https://github.com/AI-Hypercomputer/tpu-recipes/blob/main/inference/ironwood/vLLM/Gemma4/). > **Nightly required**: support for the encoder-free 12B Unified model landed in [vllm-project/vllm#44429](https://github.com/vllm-project/vllm/pull/44429) and has not yet shipped in a stable release. Use the pinned Docker image (recommended) or a nightly pip wheel. ## Prerequisites ### Docker (recommended) ```bash docker pull vllm/vllm-openai:gemma4-unified # NVIDIA (CUDA 13; append -cu129 for CUDA 12.9 hosts) docker pull vllm/vllm-openai-xpu:latest # Intel XPU (B60 / B70) ``` TPU images are published separately by [vllm-project/tpu-inference](https://github.com/vllm-project/tpu-inference); see the Trillium / Ironwood tpu-recipes below for the pinned tag. ### pip (nightly, NVIDIA CUDA) ```bash uv venv source .venv/bin/activate uv pip install -U vllm --pre \ --extra-index-url https://wheels.vllm.ai/nightly/cu129 \ --extra-index-url https://download.pytorch.org/whl/cu129 \ --index-strategy unsafe-best-match ``` ## Deployment Configurations ### Quick Start (Single GPU, BF16) The 12B fits comfortably on a single 40 GB+ GPU. ```bash vllm serve google/gemma-4-12B-it \ --max-model-len 32768 \ --gpu-memory-utilization 0.90 ``` ### Full-Featured Server Launch Enables text, image, audio, thinking, and tool calling: ```bash vllm serve google/gemma-4-12B-it \ --max-model-len 16384 \ --gpu-memory-utilization 0.90 \ --enable-auto-tool-choice \ --reasoning-parser gemma4 \ --tool-call-parser gemma4 \ --chat-template examples/tool_chat_template_gemma4.jinja \ --limit-mm-per-prompt '{"image": 4, "audio": 1}' \ --async-scheduling \ --host 0.0.0.0 \ --port 8000 ``` ### Docker (NVIDIA) ```bash docker run -itd --name gemma4 \ --ipc=host --network host --shm-size 16G --gpus all \ -v ~/.cache/huggingface:/root/.cache/huggingface \ vllm/vllm-openai:gemma4-unified \ --model google/gemma-4-12B-it \ --max-model-len 32768 \ --gpu-memory-utilization 0.90 \ --host 0.0.0.0 --port 8000 ``` On CUDA 12.9 hosts, use the `vllm/vllm-openai:gemma4-unified-cu129` tag instead. ### Docker (Intel XPU B60 / B70, TP=2, Full-Featured) Validated on 2× Intel Arc Pro B60 / B70 with the official vLLM XPU image. ```bash docker run --device /dev/dri \ -v /dev/dri/by-path:/dev/dri/by-path --shm-size=16g \ --privileged --ipc=host -p 8000:8000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ vllm/vllm-openai-xpu:latest \ google/gemma-4-12B-it \ --enforce-eager \ --attention-backend flash_attn \ --host 0.0.0.0 \ --port 8000 \ --trust-remote-code \ --enable-auto-tool-choice \ --reasoning-parser gemma4 \ --tool-call-parser gemma4 \ --chat-template examples/tool_chat_template_gemma4.jinja \ --limit-mm-per-prompt '{"image": 4, "audio": 1}' \ --async-scheduling \ --gpu-memory-utilization 0.9 \ --no-enable-prefix-caching \ --max-num-batched-tokens 8192 \ --max-model-len 5120 \ --tensor-parallel-size 2 ``` ### Docker (Cloud TPU — Trillium / Ironwood) TPU uses the separate `vllm/vllm-tpu` image (no pip wheel). Pull the tag specified by the upstream [Trillium](https://github.com/AI-Hypercomputer/tpu-recipes/tree/main/inference/trillium/vLLM/Gemma4) or [Ironwood](https://github.com/AI-Hypercomputer/tpu-recipes/blob/main/inference/ironwood/vLLM/Gemma4/) recipe, then run: ```bash docker run -itd --name gemma4-tpu \ --privileged --network host --shm-size 16G \ -v /dev/shm:/dev/shm -e HF_TOKEN=$HF_TOKEN \ vllm/vllm-tpu:latest \ --model google/gemma-4-12B-it \ --max-model-len 16384 \ --disable_chunked_mm_input \ --host 0.0.0.0 --port 8000 ``` ## Client Usage ### Text Generation ```python from openai import OpenAI client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY") response = client.chat.completions.create( model="google/gemma-4-12B-it", messages=[{"role": "user", "content": "Write a poem about the ocean."}], max_tokens=512, temperature=0.7, ) print(response.choices[0].message.content) ``` ### Image Understanding ```python response = client.chat.completions.create( model="google/gemma-4-12B-it", messages=[{"role": "user", "content": [ {"type": "image_url", "image_url": {"url": "https://upload.wikimedia.org/wikipedia/commons/thumb/3/3a/Cat03.jpg/1200px-Cat03.jpg"}}, {"type": "text", "text": "Describe this image in detail."}, ]}], max_tokens=1024, ) ``` ### Audio Requires `uv pip install "vllm[audio]"`. ```bash vllm serve google/gemma-4-12B-it \ --max-model-len 8192 \ --limit-mm-per-prompt '{"image": 4, "audio": 1}' ``` ### Thinking Mode ```bash vllm serve google/gemma-4-12B-it \ --max-model-len 16384 \ --enable-auto-tool-choice \ --reasoning-parser gemma4 \ --tool-call-parser gemma4 \ --chat-template examples/tool_chat_template_gemma4.jinja ``` Enable thinking per-request via `extra_body={"chat_template_kwargs": {"enable_thinking": True}}`, or default-on with `--default-chat-template-kwargs '{"enable_thinking": true}'`. ### Structured Outputs vLLM guided decoding constrains output to a JSON schema. Include semantic instructions in the system prompt — the model does not see schema descriptions. ## Configuration Tips - Set `--max-model-len` to match your workload. - `--gpu-memory-utilization 0.90–0.95` maximizes KV cache. - Image-only workloads: pass `--limit-mm-per-prompt.audio 0`. - Text-only workloads: pass `--limit-mm-per-prompt '{"image": 0, "audio": 0}'` to skip MM profiling. - `--async-scheduling` improves throughput. - FP8 KV cache (`--kv-cache-dtype fp8`) saves ~50% KV memory. ## Speculative Decoding (MTP) Enable the **Spec Decoding** feature toggle (above) or add `--speculative-config` manually to use MTP drafting with the [assistant model](https://huggingface.co/google/gemma-4-12B-it-assistant). Recommended `num_speculative_tokens`: 4–8 for this model. > **Note:** MTP speculative decoding for Gemma 4 requires the nightly build — the pinned `vllm/vllm-openai:gemma4-unified` image or a nightly pip wheel. The standard `:latest` stable tag does not include this feature. ## References - [Model card](https://huggingface.co/google/gemma-4-12B-it) - [vLLM support PR (#44429)](https://github.com/vllm-project/vllm/pull/44429) - [Assistant (MTP draft) model](https://huggingface.co/google/gemma-4-12B-it-assistant) - [Gemma docs](https://ai.google.dev/gemma/docs) - [vLLM Gemma 4 tool-call template](https://github.com/vllm-project/vllm/blob/main/examples/tool_chat_template_gemma4.jinja) - [TPU recipes: Trillium](https://github.com/AI-Hypercomputer/tpu-recipes/tree/main/inference/trillium/vLLM/Gemma4) - [TPU recipes: Ironwood](https://github.com/AI-Hypercomputer/tpu-recipes/blob/main/inference/ironwood/vLLM/Gemma4/) --- # Google/gemma-4-31B-it on vLLM Source: https://recipes.vllm.ai/Google/gemma-4-31B-it Provider: Google Released: 2026-03-11T18:22:36.000Z Updated: 2026-09-09 Architecture: dense Parameters: 31B Context length: 262,144 Min vLLM: 0.19.1 Tasks: multimodal, text Verified hardware: H100, MI300X, MI325X, MI355X, TPU v6e (Trillium), TPU v7 (Ironwood), Arc Pro B60, Arc Pro B70 Summary: Google's unified multimodal Gemma 4 dense model (31B) with native text, image, and audio, plus thinking mode and tool-use protocol. ## Variants - default: precision=bf16, vram_minimum_gb=75 Full BF16 — single 80 GB NVIDIA GPU or 1x MI300X/MI325X/MI350X/MI355X - fp8: precision=fp8, vram_minimum_gb=38, model_id=RedHatAI/gemma-4-31B-it-FP8-dynamic FP8 (E4M3) linear weights with dynamic per-token activation quantization (vision tower stays BF16) — Hopper or Blackwell - nvfp4: precision=nvfp4, vram_minimum_gb=19, model_id=nvidia/gemma-4-31B-it-NVFP4 NVIDIA NVFP4 quantized weights for Blackwell GPUs - redhat_nvfp4: precision=nvfp4, vram_minimum_gb=19, model_id=RedHatAI/gemma-4-31B-it-NVFP4 RedHatAI compressed-tensors NVFP4 quantization for Blackwell GPUs - w4a16: precision=int4, vram_minimum_gb=20, model_id=google/gemma-4-31B-it-qat-w4a16-ct W4A16 QAT (4-bit weights, 16-bit activations, group_size=32) via compressed-tensors — runs on any GPU - redhat_fp8: precision=fp8, vram_minimum_gb=38, model_id=RedHatAI/gemma-4-31B-it-FP8-block RedHatAI FP8 block quantization ## Guide ## Overview [Gemma 4](https://ai.google.dev/gemma/docs) is Google's most capable open model family, featuring a unified multimodal architecture that natively processes text, images, and audio. Gemma 4 models support structured thinking/reasoning, function calling with a custom tool-use protocol, and dynamic vision resolution — all available through vLLM's OpenAI-compatible API. ### Key Features - **Multimodal**: Text + images natively (video via custom frame-extraction pipeline). The smaller E2B and E4B models also support audio. - **MoE variant**: 128 fine-grained experts with top-8 routing and custom GELU-activated FFN (Gemma 4 26B-A4B). - **Dual Attention**: Alternating sliding-window (local) and global attention with different head dimensions. - **Thinking Mode**: Structured reasoning via `<|channel>thought\n...` delimiters. - **Function Calling**: Custom tool-call protocol with dedicated special tokens. - **Dynamic Vision Resolution**: Per-request configurable vision token budget (70, 140, 280, 560, 1120 tokens). ### Supported Variants Dense: - `google/gemma-4-E2B-it` (effective 2B) - `google/gemma-4-E4B-it` (effective 4B) - `google/gemma-4-31B-it` (31B) MoE: - `google/gemma-4-26B-A4B-it` (26B total / 4B active) TPU support is provided through [vLLM TPU](https://github.com/vllm-project/tpu-inference) with recipes for [Trillium](https://github.com/AI-Hypercomputer/tpu-recipes/tree/main/inference/trillium/vLLM/Gemma4) and [Ironwood](https://github.com/AI-Hypercomputer/tpu-recipes/blob/main/inference/ironwood/vLLM/Gemma4/). ## Prerequisites ### pip (NVIDIA CUDA) ```bash uv venv source .venv/bin/activate uv pip install -U vllm --pre \ --extra-index-url https://wheels.vllm.ai/nightly/cu129 \ --extra-index-url https://download.pytorch.org/whl/cu129 \ --index-strategy unsafe-best-match ``` ### pip (AMD ROCm: MI300X, MI325X, MI350X, MI355X) Requires Python 3.12, ROCm 7.2.1, glibc >= 2.35 (Ubuntu 22.04+). ```bash uv venv --python 3.12 source .venv/bin/activate uv pip install vllm --pre \ --extra-index-url https://wheels.vllm.ai/rocm/nightly/rocm721 --upgrade ``` ### Docker ```bash docker pull vllm/vllm-openai:gemma4-0505-cu129 # NVIDIA Hopper (H100/H200, CUDA 12.9) docker pull vllm/vllm-openai:gemma4-0505-cu130 # NVIDIA Blackwell (B200/B300, CUDA 13.0) docker pull vllm/vllm-openai-rocm:latest # AMD docker pull vllm/vllm-openai-xpu:latest # Intel XPU (B60 / B70) ``` TPU images are published separately by [vllm-project/tpu-inference](https://github.com/vllm-project/tpu-inference); see the Trillium / Ironwood tpu-recipes below for the pinned tag. ## Deployment Configurations ### Quick Start (Single GPU) ```bash vllm serve google/gemma-4-E4B-it \ --max-model-len # up to 131072 ``` ### 31B Dense on 2xA100/H100 (TP=2, BF16) ```bash vllm serve google/gemma-4-31B-it \ --tensor-parallel-size 2 \ --max-model-len 32768 \ --gpu-memory-utilization 0.90 ``` ### 26B MoE on 1xA100/H100 (BF16) ```bash vllm serve google/gemma-4-26B-A4B-it \ --max-model-len 32768 \ --gpu-memory-utilization 0.90 ``` ### Full-Featured Server Launch Enables text, image, audio, thinking, and tool calling: ```bash vllm serve google/gemma-4-31B-it \ --tensor-parallel-size 2 \ --max-model-len 16384 \ --gpu-memory-utilization 0.90 \ --enable-auto-tool-choice \ --reasoning-parser gemma4 \ --tool-call-parser gemma4 \ --chat-template examples/tool_chat_template_gemma4.jinja \ --limit-mm-per-prompt '{"image": 4, "audio": 1}' \ --async-scheduling \ --host 0.0.0.0 \ --port 8000 ``` ### Docker (NVIDIA) ```bash docker run -itd --name gemma4 \ --ipc=host --network host --shm-size 16G --gpus all \ -v ~/.cache/huggingface:/root/.cache/huggingface \ vllm/vllm-openai:gemma4-0505-cu129 \ --model google/gemma-4-31B-it \ --tensor-parallel-size 2 \ --max-model-len 32768 \ --gpu-memory-utilization 0.90 \ --host 0.0.0.0 --port 8000 ``` Swap `vllm/vllm-openai:gemma4-0505-cu129` for `vllm/vllm-openai:gemma4-0505-cu130` on Blackwell (B200/B300). ### Docker (AMD MI300X/MI325X/MI350X/MI355X) ```bash docker run -itd --name gemma4-rocm \ --ipc=host --network=host --privileged \ --cap-add=CAP_SYS_ADMIN --device=/dev/kfd --device=/dev/dri \ --group-add=video --cap-add=SYS_PTRACE \ --security-opt=seccomp=unconfined --shm-size 16G \ -v ~/.cache/huggingface:/root/.cache/huggingface \ vllm/vllm-openai-rocm:latest \ --model google/gemma-4-31B-it \ --host 0.0.0.0 --port 8000 ``` ### Docker (Intel XPU B60 / B70, TP=4, Full-Featured) Validated on 4× Intel Arc Pro B60 / B70 with the official vLLM XPU image. ```bash docker run --device /dev/dri \ -v /dev/dri/by-path:/dev/dri/by-path --shm-size=16g \ --privileged --ipc=host -p 8000:8000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ vllm/vllm-openai-xpu:latest \ google/gemma-4-31B-it \ --enforce-eager \ --attention-backend flash_attn \ --host 0.0.0.0 \ --port 8000 \ --trust-remote-code \ --enable-auto-tool-choice \ --reasoning-parser gemma4 \ --tool-call-parser gemma4 \ --chat-template examples/tool_chat_template_gemma4.jinja \ --limit-mm-per-prompt '{"image": 4, "audio": 1}' \ --async-scheduling \ --gpu-memory-utilization 0.9 \ --no-enable-prefix-caching \ --max-num-batched-tokens 8192 \ --max-model-len 5120 \ --tensor-parallel-size 4 ``` ### Docker (Cloud TPU — Trillium / Ironwood) TPU uses the separate `vllm/vllm-tpu` image (no pip wheel). Pull the tag specified by the upstream [Trillium](https://github.com/AI-Hypercomputer/tpu-recipes/tree/main/inference/trillium/vLLM/Gemma4) or [Ironwood](https://github.com/AI-Hypercomputer/tpu-recipes/blob/main/inference/ironwood/vLLM/Gemma4/) recipe, then run: ```bash docker run -itd --name gemma4-tpu \ --privileged --network host --shm-size 16G \ -v /dev/shm:/dev/shm -e HF_TOKEN=$HF_TOKEN \ vllm/vllm-tpu:latest \ --model google/gemma-4-31B-it \ --tensor-parallel-size 8 \ --max-model-len 16384 \ --disable_chunked_mm_input \ --host 0.0.0.0 --port 8000 ``` ## Client Usage ### Text Generation ```python from openai import OpenAI client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY") response = client.chat.completions.create( model="google/gemma-4-31B-it", messages=[{"role": "user", "content": "Write a poem about the ocean."}], max_tokens=512, temperature=0.7, ) print(response.choices[0].message.content) ``` ### Image Understanding ```python response = client.chat.completions.create( model="google/gemma-4-31B-it", messages=[{"role": "user", "content": [ {"type": "image_url", "image_url": {"url": "https://upload.wikimedia.org/wikipedia/commons/thumb/3/3a/Cat03.jpg/1200px-Cat03.jpg"}}, {"type": "text", "text": "Describe this image in detail."}, ]}], max_tokens=1024, ) ``` ### Dynamic Vision Resolution Supported values: 70, 140, 280 (default), 560, 1120 tokens/image. ```bash vllm serve google/gemma-4-31B-it \ --mm-processor-kwargs '{"max_soft_tokens": 560}' ``` ### Audio (E2B / E4B) Requires `uv pip install "vllm[audio]"`. ```bash vllm serve google/gemma-4-E2B-it \ --max-model-len 8192 \ --limit-mm-per-prompt '{"image": 4, "audio": 1}' ``` ### Thinking Mode ```bash vllm serve google/gemma-4-31B-it \ --max-model-len 16384 \ --enable-auto-tool-choice \ --reasoning-parser gemma4 \ --tool-call-parser gemma4 \ --chat-template examples/tool_chat_template_gemma4.jinja ``` Enable thinking per-request via `extra_body={"chat_template_kwargs": {"enable_thinking": True}}`, or default-on with `--default-chat-template-kwargs '{"enable_thinking": true}'`. ### Structured Outputs vLLM guided decoding constrains output to a JSON schema. Include semantic instructions in the system prompt — the model does not see schema descriptions. ## Configuration Tips - Set `--max-model-len` to match your workload. - `--gpu-memory-utilization 0.90–0.95` maximizes KV cache. - Image-only workloads: pass `--limit-mm-per-prompt.audio 0`. - Text-only workloads: pass `--limit-mm-per-prompt '{"image": 0, "audio": 0}'` to skip MM profiling. - `--async-scheduling` improves throughput. - FP8 KV cache (`--kv-cache-dtype fp8`) saves ~50% KV memory. ## Quantized Variants Two pre-quantized checkpoints are available: - [`RedHatAI/gemma-4-31B-it-FP8-dynamic`](https://huggingface.co/RedHatAI/gemma-4-31B-it-FP8-dynamic) — FP8 (E4M3) linear weights with dynamic per-token activation quantization (vision tower stays BF16); runs on Hopper and Blackwell. - [`nvidia/gemma-4-31B-it-NVFP4`](https://huggingface.co/nvidia/gemma-4-31B-it-NVFP4) — NVFP4 (4-bit) weights; requires Blackwell (B200/B300). Pick them from the **Variant** dropdown above, or pass the repo id directly to `vllm serve`. ## Throughput vs Latency | Goal | TP | `--max-num-seqs` | Notes | |------|----|------------------|-------| | Max throughput | 1-2 | 256-512 | Best tok/s per GPU | | Min latency | 4-8 | 8-16 | Best TTFT/TPOT | | Balanced | 2 | 128 | Mixed workloads | ## Speculative Decoding (MTP) Enable the **Spec Decoding** feature toggle (above) or add `--speculative-config` manually to use MTP drafting with the [assistant model](https://huggingface.co/google/gemma-4-31B-it-assistant). Recommended `num_speculative_tokens`: 4–8 for this model. See the [Gemma 4 usage guide](../../Google/Gemma4) for details and benchmarks. > **Note:** MTP speculative decoding for Gemma 4 is only available on the vLLM nightly build — it has not yet landed in a stable release. Install via the nightly wheel (`uv pip install -U vllm --pre --extra-index-url https://wheels.vllm.ai/nightly/cu129 …`) or use the `vllm/vllm-openai:gemma4-0505-cu129` / `vllm/vllm-openai:gemma4-0505-cu130` images above; the standard `:latest` stable tag does not include this feature. ## References - [Model card](https://huggingface.co/google/gemma-4-31B-it) - [FP8 variant](https://huggingface.co/RedHatAI/gemma-4-31B-it-FP8-dynamic) - [NVFP4 variant](https://huggingface.co/nvidia/gemma-4-31B-it-NVFP4) - [Gemma docs](https://ai.google.dev/gemma/docs) - [vLLM Gemma 4 tool-call template](https://github.com/vllm-project/vllm/blob/main/examples/tool_chat_template_gemma4.jinja) - [TPU recipes: Trillium](https://github.com/AI-Hypercomputer/tpu-recipes/tree/main/inference/trillium/vLLM/Gemma4) - [TPU recipes: Ironwood](https://github.com/AI-Hypercomputer/tpu-recipes/blob/main/inference/ironwood/vLLM/Gemma4/) --- # dots-studio/dots3-note-prev on vLLM Source: https://recipes.vllm.ai/dots-studio/dots3-note-prev Provider: Dots Released: 2026-08-09T07:33:17.000Z Updated: 2026-09-09 Architecture: moe Parameters: 288B Context length: 524,288 Min vLLM: 0.28.0 Tasks: multimodal, text Verified hardware: H100 Summary: Multimodal MoE model available in BF16 and native FP8, with hybrid DSA and SWA, 512K context, and MTP speculative decoding. ## Variants - default: precision=fp8, vram_minimum_gb=359, model_id=dots-studio/dots3-note-prev-fp8 Native block-scaled FP8 language-model weights with BF16 multimodal encoders; verified on 8x 80GB NVIDIA GPUs. - bf16: precision=bf16, vram_minimum_gb=692, model_id=dots-studio/dots3-note-prev Full BF16 checkpoint using the Triton unquantized-MoE backend; its indexed weights total about 576.9 GB before runtime and KV-cache memory. ## Recommended base flags ```bash vllm serve 'dots-studio/dots3-note-prev' \ --served-model-name \ dots3-note-prev \ --gpu-memory-utilization \ 0.9 \ --max-model-len \ 262144 \ --max-num-batched-tokens \ 8192 \ --block-size \ 64 \ --mm-processor-cache-gb \ 32 ``` ## Guide ## Overview dots3-note-prev is a native multimodal Mixture-of-Experts model supporting text, image, video, and audio inputs through one model architecture. It is published as a full BF16 checkpoint (`dots-studio/dots3-note-prev`) and a native FP8 checkpoint (`dots-studio/dots3-note-prev-fp8`). Its language backbone has 256 routed experts (top-8) plus a shared expert and mixes DSA and SWA layers. Both releases expose MTP speculative decoding. The model configuration supports up to 524,288 tokens. The commands below use a validated 262,144-token serving limit and 8,192-token chunked-prefill budget so the remaining GPU memory can be used for KV cache. Increase `--max-model-len` only after checking the KV-cache budget for the target workload. ## Prerequisites - 8 NVIDIA GPUs with at least 80 GB each for the native FP8 checkpoint. - The BF16 checkpoint contains about 576.9 GB of indexed weights and needs additional memory for the runtime and KV cache; the recipe budgets 692 GB aggregate VRAM. - A recent CUDA environment. Native FP8 uses DeepGEMM, while unquantized BF16 uses the Triton MoE backend. - vLLM 0.28.0 or newer, which includes native dots3-note-prev support. ## Launch commands The examples expose an OpenAI-compatible endpoint on all network interfaces. Remove `--host 0.0.0.0` if the service should only be reachable locally. The general strategy examples use the FP8 checkpoint. BF16 needs a different MoE backend and a larger memory budget, so use the explicit BF16 command below instead of only replacing the model ID. ### BF16 tensor parallel + expert parallel (TP=8, EP=8) This command serves the full multimodal BF16 checkpoint. Triton is required because DeepGEMM does not support unquantized MoE weights. A 0.95 memory-utilization target leaves enough KV-cache capacity for the 262,144-token serving limit after profiling the multimodal encoders. If the target hardware cannot reserve that much memory, lower `--max-model-len` or add `--language-model-only` for text workloads. ```bash vllm serve dots-studio/dots3-note-prev \ --served-model-name dots3-note-prev \ --host 0.0.0.0 \ --tensor-parallel-size 8 \ --enable-expert-parallel \ --gpu-memory-utilization 0.95 \ --max-model-len 262144 \ --max-num-batched-tokens 8192 \ --block-size 64 \ --mm-processor-cache-gb 32 \ --moe-backend triton ``` ### Tensor parallel (TP=8) TP shards both dense and expert weights across all eight GPUs. It is the simplest deployment, but it does not enable expert parallelism. ```bash vllm serve dots-studio/dots3-note-prev-fp8 \ --served-model-name dots3-note-prev \ --host 0.0.0.0 \ --tensor-parallel-size 8 \ --gpu-memory-utilization 0.9 \ --max-model-len 262144 \ --max-num-batched-tokens 8192 \ --block-size 64 \ --mm-processor-cache-gb 32 \ --moe-backend deep_gemm ``` ### Tensor parallel + expert parallel (TP=8, EP=8) This is the recommended single-node configuration. Dense layers use TP while MoE experts are distributed across the eight expert-parallel ranks. ```bash vllm serve dots-studio/dots3-note-prev-fp8 \ --served-model-name dots3-note-prev \ --host 0.0.0.0 \ --tensor-parallel-size 8 \ --enable-expert-parallel \ --gpu-memory-utilization 0.9 \ --max-model-len 262144 \ --max-num-batched-tokens 8192 \ --block-size 64 \ --mm-processor-cache-gb 32 \ --moe-backend deep_gemm ``` ### Data parallel + expert parallel (DP=8, EP=8) DP+EP is intended for throughput-oriented serving. Each DP rank owns a replica of the non-expert weights, while routed experts are distributed across all eight ranks. It therefore needs more GPU memory than TP+EP. ```bash vllm serve dots-studio/dots3-note-prev-fp8 \ --served-model-name dots3-note-prev \ --host 0.0.0.0 \ --data-parallel-size 8 \ --enable-expert-parallel \ --gpu-memory-utilization 0.9 \ --max-model-len 262144 \ --max-num-batched-tokens 8192 \ --block-size 64 \ --mm-processor-cache-gb 32 \ --moe-backend deep_gemm ``` ### Language-model-only serving The checkpoint keeps a single multimodal architecture and model type. For pure-text serving, add `--language-model-only`; a separate text checkpoint is not required. The option can be combined with TP, TP+EP, or DP+EP. ```bash vllm serve dots-studio/dots3-note-prev-fp8 \ --served-model-name dots3-note-prev \ --host 0.0.0.0 \ --tensor-parallel-size 8 \ --enable-expert-parallel \ --language-model-only \ --gpu-memory-utilization 0.9 \ --max-model-len 262144 \ --max-num-batched-tokens 8192 \ --block-size 64 \ --mm-processor-cache-gb 32 \ --moe-backend deep_gemm ``` ### MTP speculative decoding Add the speculative configuration to any of the deployments above. The current checkpoints expose an MTP prediction head that vLLM reuses for drafting without loading a separate draft model. The example uses TP+EP with the native FP8 checkpoint. ```bash vllm serve dots-studio/dots3-note-prev-fp8 \ --served-model-name dots3-note-prev \ --host 0.0.0.0 \ --tensor-parallel-size 8 \ --enable-expert-parallel \ --speculative-config '{"method":"mtp","num_speculative_tokens":3}' \ --gpu-memory-utilization 0.9 \ --max-model-len 262144 \ --max-num-batched-tokens 8192 \ --block-size 64 \ --mm-processor-cache-gb 32 \ --moe-backend deep_gemm ``` ### Tool calling dots3-note-prev emits tool calls inside `` XML wrappers. Enable automatic tool choice with the native `dots` parser. These options can be combined with any parallel strategy, `--language-model-only`, and MTP speculative decoding. ```bash vllm serve dots-studio/dots3-note-prev-fp8 \ --served-model-name dots3-note-prev \ --host 0.0.0.0 \ --tensor-parallel-size 8 \ --enable-expert-parallel \ --enable-auto-tool-choice \ --tool-call-parser dots \ --gpu-memory-utilization 0.9 \ --max-model-len 262144 \ --max-num-batched-tokens 8192 \ --block-size 64 \ --mm-processor-cache-gb 32 \ --moe-backend deep_gemm ``` ## Client usage ```bash curl http://localhost:8000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "dots3-note-prev", "messages": [{"role": "user", "content": "Explain sparse attention briefly."}], "temperature": 0.7 }' ``` ## Tuning notes - `--max-num-batched-tokens 8192` is the chunked-prefill budget, not the maximum request length. Increase it only when prefill throughput is more important than inter-token latency and enough temporary workspace is available. - DP+EP replicates the non-expert portion on every DP rank. Prefer TP+EP when memory capacity or long-context KV cache is the priority. - `--language-model-only` skips multimodal encoder loading but does not change the checkpoint's model type or architecture. - Full multimodal BF16 serving at 262,144 tokens may not have enough KV-cache memory with `--gpu-memory-utilization 0.9`; the BF16 variant uses `0.95`. Lower `--max-model-len` instead when the extra memory cannot be reserved. - DeepGEMM is specific to the native FP8 checkpoint in this recipe. BF16 is unquantized and must use `--moe-backend triton`. - MTP changes decode scheduling and memory use. Benchmark it against non-speculative decoding with the same prompt/output distribution before production deployment. - `--tool-call-parser dots` supports the model's XML invoke/parameter format and its JSON fallback format, including multiple tool calls in one response. ## References - [dots3-note-prev FP8 on Hugging Face](https://huggingface.co/dots-studio/dots3-note-prev-fp8) - [dots3-note-prev BF16 on Hugging Face](https://huggingface.co/dots-studio/dots3-note-prev) - [vLLM dots3-note-prev support PR](https://github.com/vllm-project/vllm/pull/51255) - [vLLM parallelism and scaling guide](https://docs.vllm.ai/en/latest/serving/parallelism_scaling.html) --- # inclusionAI/Ling-3.0-flash on vLLM Source: https://recipes.vllm.ai/inclusionAI/Ling-3.0-flash Provider: inclusionAI Released: 2026-08-02T16:14:41.000Z Updated: 2026-09-09 Architecture: moe Parameters: 124B Context length: 262,144 Min vLLM: 0.28.0 Tasks: text Verified hardware: H200, DGX Spark (GB10), MI300X, MI355X Summary: Ling-3.0-flash MoE model with BF16, FP8, FP4, and INT4 checkpoints, native MTP, and an external DSpark draft model ## Variants - default: precision=bf16, vram_minimum_gb=300 BF16 weights, the reference precision. - fp8: precision=fp8, vram_minimum_gb=150, model_id=inclusionAI/Ling-3.0-flash-fp8 Block-FP8 (E4M3) weights, 128x128 block scaling. - fp4: precision=fp4, vram_minimum_gb=85, model_id=inclusionAI/Ling-3.0-flash-fp4 Block-FP8 dense with MXFP4 routed experts. Emulated on gfx942 — prefer INT4 there. - int4: precision=int4, vram_minimum_gb=95, model_id=inclusionAI/Ling-3.0-flash-int4 compressed-tensors W4 (group 32) on the routed experts. ## Recommended base flags ```bash vllm serve 'inclusionAI/Ling-3.0-flash' \ --trust-remote-code \ --dtype \ bfloat16 \ --gpu-memory-utilization \ 0.9 \ --enable-chunked-prefill \ --compilation-config \ {"cudagraph_mode":"FULL_AND_PIECEWISE"} \ --enable-prefix-caching ``` ## Guide ## Overview `inclusionAI/Ling-3.0-flash` uses the `BailingMoeV3ForCausalLM` architecture with a hybrid MLA/KDA attention stack, 512 routed experts (8 active per token), one shared expert, and a native multi-token prediction head. The 42-layer base model has 124.4B total and 5.5B active parameters. The checkpoint also contains a 3.1B MTP layer, bringing the complete checkpoint to 127.5B parameters. A serialized block-FP8 checkpoint is available as `inclusionAI/Ling-3.0-flash-fp8`. Lower-memory checkpoints are also available as `inclusionAI/Ling-3.0-flash-fp4` and `inclusionAI/Ling-3.0-flash-int4`. The external `inclusionAI/Ling-3.0-flash-dspark` draft can accelerate any target variant without changing which target checkpoint is served. ## Prerequisites - **vLLM:** 0.28.0 or newer, which includes native Bailing V3 support; - **Validated hardware:** NVIDIA H20, H20-3e, H200, DGX Spark (FP4/INT4), and AMD MI300X / MI355X - **Precision:** BF16, serialized block FP8, mixed block-FP8/MXFP4, or compressed-tensors INT4 weights with BF16 compute - **Context length:** 262,144 tokens ## Launching the Server ```bash NCCL_DEBUG=WARN vllm serve inclusionAI/Ling-3.0-flash \ --trust-remote-code \ --dtype bfloat16 \ --tensor-parallel-size 4 \ --gpu-memory-utilization 0.9 \ --enable-chunked-prefill \ --compilation-config '{"cudagraph_mode":"FULL_AND_PIECEWISE"}' \ --enable-prefix-caching \ --enable-auto-tool-choice \ --tool-call-parser ling3 \ --reasoning-parser ling3 \ --speculative-config '{"method":"mtp","num_speculative_tokens":3}' ``` For the serialized FP8 variant on 2x H200, use: ```bash vllm serve inclusionAI/Ling-3.0-flash-fp8 \ --trust-remote-code \ --dtype bfloat16 \ --tensor-parallel-size 2 \ --gpu-memory-utilization 0.9 \ --enable-chunked-prefill \ --compilation-config '{"cudagraph_mode":"FULL_AND_PIECEWISE"}' \ --enable-prefix-caching \ --enable-auto-tool-choice \ --tool-call-parser ling3 \ --reasoning-parser ling3 ``` For the validated TP4+EP4 alternative on 4x H200, use: ```bash vllm serve inclusionAI/Ling-3.0-flash-fp8 \ --trust-remote-code \ --dtype bfloat16 \ --tensor-parallel-size 4 \ --enable-expert-parallel \ --gpu-memory-utilization 0.9 \ --enable-chunked-prefill \ --compilation-config '{"cudagraph_mode":"FULL_AND_PIECEWISE"}' \ --enable-prefix-caching \ --enable-auto-tool-choice \ --tool-call-parser ling3 \ --reasoning-parser ling3 ``` For the FP4 variant on one GPU, use: ```bash vllm serve inclusionAI/Ling-3.0-flash-fp4 \ --trust-remote-code \ --dtype bfloat16 \ --tensor-parallel-size 1 \ --gpu-memory-utilization 0.9 \ --enable-chunked-prefill \ --compilation-config '{"cudagraph_mode":"FULL_AND_PIECEWISE"}' \ --enable-prefix-caching \ --enable-auto-tool-choice \ --tool-call-parser ling3 \ --reasoning-parser ling3 ``` For the INT4 variant on one GPU, use: ```bash vllm serve inclusionAI/Ling-3.0-flash-int4 \ --trust-remote-code \ --dtype bfloat16 \ --tensor-parallel-size 1 \ --gpu-memory-utilization 0.9 \ --enable-chunked-prefill \ --compilation-config '{"cudagraph_mode":"FULL_AND_PIECEWISE"}' \ --enable-prefix-caching \ --enable-auto-tool-choice \ --tool-call-parser ling3 \ --reasoning-parser ling3 ``` ## Speculative Decoding The **Spec decoding** control exposes MTP and DSpark in the same place. They are mutually exclusive modes, so the generated command contains exactly one `--speculative-config`. ### MTP Use the target checkpoint's native MTP head with three draft tokens: ```bash --speculative-config '{"method":"mtp","num_speculative_tokens":3}' ``` ### DSpark `inclusionAI/Ling-3.0-flash-dspark` is an external draft model. It does not replace the served target checkpoint. It can be paired with any Ling-3.0-flash target variant; pass the DSpark repository through `--speculative-config.model` while keeping the selected target as the model served by `vllm serve`. The checkpoint was trained with query block size `Q=8`. Runtime `num_speculative_tokens` (`K`) may be any integer from 1 through 8. The recipe emits `K=7` as the recommended throughput/KV-capacity trade-off; change the value to select another supported width. `K=8` remains valid. ```bash --speculative-config '{"method":"dspark","model":"inclusionAI/Ling-3.0-flash-dspark","num_speculative_tokens":7,"draft_sample_method":"probabilistic","attention_backend":"FLASH_ATTN","enable_adaptive_verification":false}' ``` `enable_adaptive_verification` is disabled because the current Ling hybrid MLA/KDA path is validated with fixed-width DSpark verification. The `attention_backend` setting applies only to the external draft model; the target model keeps its automatically selected attention backends. ## Single-GPU INT4 `inclusionAI/Ling-3.0-flash-int4` is a `compressed-tensors` / `pack-quantized` checkpoint: symmetric W4, `group_size: 32`, applied to the **routed experts only** — attention, `lm_head`, the shared expert and the dense projections are all in the config's `ignore` list. It is not covered by the model-specific quantization plumbing in `bailing_moe_v3.py` (which handles block FP8 and MXFP4); it loads through vLLM's generic `compressed-tensors` path and runs at TP1 on H200 or DGX Spark. The engine names the mechanism at startup: ``` quantization=compressed-tensors ... trust_remote_code=True, dtype=torch.bfloat16 INFO [int_wna16.py:297] Using 'MARLIN' WNA16 MoE backend. INFO [cuda.py:492] Using FLASH_ATTN_MLA attention backend INFO [int_wna16.py:409] Using MarlinExperts ``` The vendor model card documents SGLang only, and there is no merged vLLM PR specific to Ling INT4. This variant works through the generic quantization path; tool calling, reasoning parsing, chunked prefill and CUDA graph capture behave as they do on the other variants. ## Running on MI300X/MI355X ```bash export VLLM_ROCM_USE_AITER=1 vllm serve inclusionAI/Ling-3.0-flash \ --trust-remote-code \ --dtype bfloat16 \ --tensor-parallel-size 4 \ --gpu-memory-utilization 0.9 \ --enable-chunked-prefill \ --compilation-config '{"cudagraph_mode":"FULL_AND_PIECEWISE"}' \ --enable-prefix-caching \ --enable-auto-tool-choice \ --tool-call-parser ling3 \ --reasoning-parser ling3 ``` On MI300X the BF16 run served the full 262,144-token context with a 105.15 GiB KV pool and 52.38x max concurrency. **MTP** needs a backend pin on gfx950: `ROCM_AITER_MLA` lets the speculator capture a graph the backend cannot serve and the workers abort with a memory access fault, so the mode pins `--attention-backend TRITON_MLA` on `mi355x`. gfx942 is unaffected and keeps AITER MLA. ## Validation The BF16, FP8, FP4, and INT4 checkpoints support NVIDIA H20 and H200. For FP8, TP2 remains the recommended multi-GPU default; TP4+EP4 is a validated H200 alternative for deployments that prefer expert parallelism. DGX Spark is limited to the TP1 FP4 and INT4 variants because the BF16 and FP8 checkpoints are too large for its 128 GB unified memory. DSpark is external and works with any target variant that fits the selected hardware. The `int4` variant was verified on **1x H200 (SM90), TP=1**, on a `main` build reporting `0.26.1rc1.dev1133+gf94666b60`, serving at `--max-model-len 32768`: 77.0 GB of weights on disk, 70.27 GiB resident after load (165 s), 5 min 56 s from container start to ready, `FULL_AND_PIECEWISE` capture (68 PIECEWISE + 68 FULL, 2.15 GiB), and a 4,072,594-token KV pool. Single-stream decode measured **159.5 tok/s** at TTFT p50 0.17 s; 973 output tok/s aggregate at concurrency 16, with 0 failed requests and 0 preemptions over a 1..24 concurrency sweep. MI355X was verified on MI350X silicon, the same gfx950 class. ## Thinking Mode Thinking is selected per request through the chat template rather than by a server flag: ```python from openai import OpenAI client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY") response = client.chat.completions.create( model="inclusionAI/Ling-3.0-flash", messages=[{"role": "user", "content": "Solve the problem step by step."}], temperature=0.0, max_tokens=200000, extra_body={"chat_template_kwargs": {"enable_thinking": True}}, ) msg = response.choices[0].message print(msg.reasoning) print(msg.content) ``` When serving a quantized variant, set `model` in the client request to the corresponding FP8, FP4, or INT4 checkpoint ID. ## References - [Ling-3.0-flash on Hugging Face](https://huggingface.co/inclusionAI/Ling-3.0-flash) - [Ling-3.0-flash-fp8 on Hugging Face](https://huggingface.co/inclusionAI/Ling-3.0-flash-fp8) - [Ling-3.0-flash-dspark on Hugging Face](https://huggingface.co/inclusionAI/Ling-3.0-flash-dspark) - [Ling-3.0-flash-fp4 on Hugging Face](https://huggingface.co/inclusionAI/Ling-3.0-flash-fp4) - [Ling-3.0-flash-int4 on Hugging Face](https://huggingface.co/inclusionAI/Ling-3.0-flash-int4) --- # inclusionAI/Ling-3.0-tiny on vLLM Source: https://recipes.vllm.ai/inclusionAI/Ling-3.0-tiny Provider: inclusionAI Released: 2026-08-10T02:44:06.000Z Updated: 2026-09-09 Architecture: moe Parameters: 7.9B Context length: 131,072 Min vLLM: 0.28.0 Tasks: text Verified hardware: H200 Summary: Ling-3.0-tiny lightweight hybrid-reasoning MoE with BF16, block-FP8, and compressed-tensors INT4 checkpoints ## Variants - default: precision=bf16, vram_minimum_gb=18 Official BF16 checkpoint; approximately 15.8 GB of repository weights and served with TP1 - fp8: precision=fp8, vram_minimum_gb=11, model_id=inclusionAI/Ling-3.0-tiny-fp8 Official serialized block-FP8 checkpoint with dynamic activations and 128x128 weight blocks; served with TP1 - int4: precision=int4, vram_minimum_gb=8, model_id=inclusionAI/Ling-3.0-tiny-int4 Official compressed-tensors group-wise INT4 checkpoint; served with TP1 ## Recommended base flags ```bash vllm serve 'inclusionAI/Ling-3.0-tiny' \ --trust-remote-code \ --dtype \ bfloat16 \ --gpu-memory-utilization \ 0.85 \ --enable-prefix-caching ``` ## Guide ## Overview `inclusionAI/Ling-3.0-tiny` is the lightweight member of the Ling 3.0 family. It has 7.9B total parameters and activates 1.3B parameters per token. Its 24-layer hybrid stack repeats three Kimi Delta Attention (KDA) layers and one Multi-Head Latent Attention (MLA) layer, while each sparse MoE block selects 8 of 128 routed experts plus one shared expert. The official release provides three served checkpoints: - `inclusionAI/Ling-3.0-tiny`: BF16; - `inclusionAI/Ling-3.0-tiny-fp8`: serialized block FP8; - `inclusionAI/Ling-3.0-tiny-int4`: compressed-tensors INT4. All three use the same `BailingMoeV3ForCausalLM` architecture and a native 131,072-token context window. Select the corresponding Variant above; the served model ID changes automatically. ## Prerequisites - **vLLM:** 0.28.0 or newer, which includes Ling 3.0 / Bailing V3 hybrid-attention support; - **Parallelism:** TP1 is recommended for all three checkpoints; - **Thinking mode:** enabled per request by the chat template. ## Launching the Server BF16 on one GPU: ```bash vllm serve inclusionAI/Ling-3.0-tiny \ --trust-remote-code \ --dtype bfloat16 \ --tensor-parallel-size 1 \ --gpu-memory-utilization 0.85 \ --enable-prefix-caching \ --enable-auto-tool-choice \ --tool-call-parser ling3 \ --reasoning-parser ling3 ``` For FP8 or INT4, keep the same arguments and replace the served checkpoint: ```bash # Serialized block FP8 vllm serve inclusionAI/Ling-3.0-tiny-fp8 \ --trust-remote-code --dtype bfloat16 --tensor-parallel-size 1 \ --gpu-memory-utilization 0.85 --enable-prefix-caching --enable-auto-tool-choice \ --tool-call-parser ling3 --reasoning-parser ling3 # Compressed-tensors INT4 vllm serve inclusionAI/Ling-3.0-tiny-int4 \ --trust-remote-code --dtype bfloat16 --tensor-parallel-size 1 \ --gpu-memory-utilization 0.85 --enable-prefix-caching --enable-auto-tool-choice \ --tool-call-parser ling3 --reasoning-parser ling3 ``` ## Thinking Mode Thinking is controlled per request. The model card recommends `temperature=1.0`, `top_p=0.95`, and `top_k=20` when thinking is enabled: ```python from openai import OpenAI client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY") response = client.chat.completions.create( model="inclusionAI/Ling-3.0-tiny", messages=[{"role": "user", "content": "Solve the problem step by step."}], temperature=1.0, top_p=0.95, extra_body={ "top_k": 20, "chat_template_kwargs": {"enable_thinking": True}, }, ) msg = response.choices[0].message print(msg.reasoning) print(msg.content) ``` Set `enable_thinking` to `false` for direct answers. When using a quantized variant, set the client `model` to its FP8 or INT4 checkpoint ID. ## References - [Ling 3.0 collection](https://huggingface.co/collections/inclusionAI/ling-30) - [Ling-3.0-tiny on Hugging Face](https://huggingface.co/inclusionAI/Ling-3.0-tiny) - [Ling-3.0-tiny-fp8 on Hugging Face](https://huggingface.co/inclusionAI/Ling-3.0-tiny-fp8) - [Ling-3.0-tiny-int4 on Hugging Face](https://huggingface.co/inclusionAI/Ling-3.0-tiny-int4) --- # meta-models/Muse-Glimmer-30B on vLLM Source: https://recipes.vllm.ai/meta-models/Muse-Glimmer-30B Provider: Muse (Meta) Released: 2026-08-09T17:51:35.000Z Updated: 2026-09-09 Architecture: dense Parameters: 29.6B Context length: 131,072 Min vLLM: 0.28.0 Tasks: text, multimodal Verified hardware: GB300 NVL4, DGX Spark (GB10), MI300X, MI325X, MI355X, Arc Pro B60, Arc Pro B70, RTX 5090, RTX 5090 Summary: Dense 29.6B vision-language model with a ViT-G/14 perception encoder and 128K context, distilled from Muse Spark for local agentic use. Emits channel-scoped reasoning and XML-style ATEM tool calls rather than JSON, so it needs the dedicated `muse_glimmer` tool-call and reasoning parsers. ## Variants - default: precision=bf16, vram_minimum_gb=72 BF16 weights, the reference precision. - fp8_block: precision=fp8, vram_minimum_gb=40, model_id=RedHatAI/Muse-Glimmer-30B-FP8-block FP8_BLOCK (E4M3) weights with 128x128 block scaling, dynamic activations. 33 GB on disk, fits one GPU. - nvfp4: precision=nvfp4, vram_minimum_gb=31, model_id=Inferact/Muse-Glimmer-30B-NVFP4-W4A4 ModelOpt NVFP4 W4A4 — weights and activations at 4 bits, group size 16. 25.42 GB on disk, fits one GPU. ## Recommended base flags ```bash vllm serve 'meta-models/Muse-Glimmer-30B' \ --generation-config \ auto ``` ## Guide ## Overview Muse Glimmer 30B is a dense vision-language model built for agentic work on consumer hardware: a 52-layer text decoder (hidden 6656) plus a ~1.8B ViT-G/14 perception encoder, 128K trained context, BF16. Apache 2.0, knowledge cutoff January 4 2026, trained on 100+ languages. What makes it unlike most recipes here is its output format. It does not emit JSON tool calls and it does not wrap reasoning in `` tags. Every turn is written as a sequence of channel-scoped messages: ``` to=self<|message|>...chain of thought...<|eom|> <|start|>assistant to=<|message|> value <|eom|> <|start|>assistant to=user<|message|>...final answer...<|eot|> ``` Both `--tool-call-parser muse_glimmer` and `--reasoning-parser muse_glimmer` key off that framing, so run them together. The reasoning parser also forces `skip_special_tokens=False` — without it the markers are stripped before parsing and both channels collapse into content. ## Prerequisites - **Hardware**: DGX Spark. FP4 weights might also run on 5090. - **Image**: `vllm/vllm-openai:v0.28.0`. ## Checkpoints | variant | repo | size | notes | |---|---|---|---| | BF16 | [`meta-models/Muse-Glimmer-30B`](https://huggingface.co/meta-models/Muse-Glimmer-30B) | 59.58 GB | reference precision | | FP8 | [`RedHatAI/Muse-Glimmer-30B-FP8-block`](https://huggingface.co/RedHatAI/Muse-Glimmer-30B-FP8-block) | 32.78 GB | FP8 block-scaled weights, dynamic activations (vision tower + embeddings BF16) | | NVFP4 | [`Inferact/Muse-Glimmer-30B-NVFP4-W4A4`](https://huggingface.co/Inferact/Muse-Glimmer-30B-NVFP4-W4A4) | 25.42 GB | ModelOpt NVFP4, **W4A4**, group size 16 | | draft | [`meta-models/Muse-Glimmer-30B-assistant`](https://huggingface.co/meta-models/Muse-Glimmer-30B-assistant) | 5.11 GB | DFlash draft head for speculative decoding | The NVFP4 build quantizes **both weights and activations** to 4 bits across 364 language-model projections (52 layers x 7); embeddings, `lm_head` and the vision tower stay unquantized, which is why it is 25 GB rather than the ~15 GB a uniform 4-bit quant would give. vLLM detects it from `config.json` (`quant_method: modelopt`, `quant_algo: NVFP4`) and selects `modelopt_fp4` with fused activation quantization — no extra flags, the launch command below is unchanged apart from the model path. NVFP4 kernels are Blackwell-only. ## Launching the Server ```bash vllm serve /model \ --served-model-name muse-glimmer \ --tensor-parallel-size 4 \ --gpu-memory-utilization 0.92 \ --max-model-len 131072 \ --max-num-seqs 64 \ --enable-auto-tool-choice --tool-call-parser muse_glimmer \ --reasoning-parser muse_glimmer \ --generation-config auto ``` At TP=4 on GB300 this reports a 26.8M-token KV pool and ~204x max concurrency at the full 128K context. ## Running on DGX Spark (GB10) Muse Glimmer was built for local deployment and runs on a DGX Spark as an ordinary single-card deployment — **no special configuration**. GB10 is aarch64 like GB300 so the same image applies, and the launch command above is unchanged apart from having one GPU to run on: ```bash vllm serve /model \ --served-model-name muse-glimmer \ --tensor-parallel-size 1 \ --enable-auto-tool-choice --tool-call-parser muse_glimmer \ --reasoning-parser muse_glimmer \ --generation-config auto ``` There is no tensor parallelism to configure: GB10 is a single GPU, so TP is 1. BF16 weights are ~55 GiB against 128 GB of unified memory, leaving the rest for the KV cache and the perception encoder. The **NVFP4** checkpoint is 25.42 GB, which leaves considerably more of that shared pool for KV and the host — the better fit of the two on this hardware. ## Running on MI300X/MI325X/MI355X Use the official `vllm/vllm-openai-rocm:nightly` image. ```bash export VLLM_ROCM_USE_AITER=1 vllm serve /model \ --served-model-name muse-glimmer \ --tensor-parallel-size 4 \ --max-model-len 131072 \ --enable-auto-tool-choice \ --tool-call-parser muse_glimmer \ --reasoning-parser muse_glimmer \ --generation-config auto \ --attention-backend ROCM_AITER_FA ``` At TP=1 on MI300X this reports an 6.66M-token KV pool and ~50.82x max concurrency at the full 128K context. At TP=4 on MI300X this reports an 17.79M-token KV pool and ~135.73x max concurrency at the full 128K context. At TP=1 on MI355X this reports an 11.64M-token KV pool and ~88.86x max concurrency at the full 128K context. At TP=4 on MI355X this reports an 27.68M-token KV pool and ~211.22x max concurrency at the full 128K context. ## Running on Intel Arc Pro (XPU) and Xeon (CPU) validated on Intel Arc Pro B70, Arc Pro B60, and Xeon 6. **Intel GPU (XPU):** ```bash docker run -it --rm --network=host --ipc=host --privileged \ --device /dev/dri:/dev/dri \ -v /dev/dri/by-path:/dev/dri/by-path \ --entrypoint bash \ vllm/vllm-openai-xpu:latest ``` Serve inside the container — `--enforce-eager` is required on the XPU backend: ```bash vllm serve meta-models/Muse-Glimmer-30B \ --tensor-parallel-size 4 \ --reasoning-parser muse_glimmer \ --enforce-eager ``` ## Sampling Use the published settings: ``` temperature = 1.0 top_p = 0.95 top_k = 64 ``` Do not run it greedy — it is a reasoning model, and greedy is not reproducible here either: identical requests at `temperature: 0` with a fixed `seed` returned 70, 80, and 86 completion tokens across runs. **Reasoning strength.** Effort is set with a `Reasoning strength: ` line in the system prompt, one of `low` / `medium` / `high` / `xhigh`. Use `high` or `xhigh` for coding and agentic tasks. ## RTX 5090 (32 GB, consumer Blackwell) The `nvfp4` variant runs on a **single** RTX 5090 with no overrides: 131,072 context, a 179,647-token KV cache (1.37x concurrency at full length), 28.8 GB of 32.6 GB used, and 68.8 tok/s on a 600-token greedy generation. Tool calling and vision both work. `vram_minimum_gb: 31` is accurate — the margin is real but thin. **Speculative decoding needs a second card.** The draft head is 5.11 GB and the nvfp4 weights are 25.42 GB, so on one 32 GB card vLLM OOMs during init with roughly 400 MiB free. On 2x 5090 with `--tensor-parallel-size 2` it is well worth the second card — decode roughly triples. ### 2x RTX 5090 Verified with the drafter enabled, and worth the second card: decode goes from 68.8 tok/s to ~240 tok/s. `--gpu-memory-utilization` only sizes the KV cache; it does not gate initialisation. Measured across the range on the base recipe (131,072 ctx, bf16 KV) plus DFlash: | `--gpu-memory-utilization` | KV cache | concurrency @131K | |---|---|---| | 0.92 | 1,202,076 | 9.17x | | 0.88 | 1,054,407 | 8.04x | | 0.84 | 906,720 | 6.92x | | 0.80 | 759,051 | 5.79x | All seven steps from 0.92 down to 0.80 initialised cleanly, including the default 0.92. Decode was 212-261 tok/s with no trend attributable to utilisation — the spread is run-to-run noise (another model shared the host during the early steps) — so treat ~240 tok/s as the figure and pick utilisation purely for the cache size you want. If you *extend* beyond the recipe — a longer `--max-model-len` via rope scaling, or `--kv-cache-dtype fp8` — the arithmetic changes and 0.92 can OOM: at 262,144 with fp8 KV, vLLM sized a 2,594,229-token cache and then failed to allocate 396 MiB for the speculator. Lowering to 0.80 fixed it. That is a consequence of the extension, not of the recipe as written. Not tested at 2x: vision, and concurrent load at `--max-num-seqs 64`. ### fp8_block on sm120 needs DeepGEMM off The `fp8_block` variant does not load on RTX 5090 with DeepGEMM enabled. It fails in `process_weights_after_loading`, before the engine starts: ``` RuntimeError: Assertion error (deepgemm/csrc/apis/layout.hpp:60): Unknown SF transformation ``` vLLM treats capability family 120 as a supported DeepGEMM target, so the path is selected and then DeepGEMM rejects the scale-factor layout. Filed as vllm-project/vllm#51884. Set both to work around it: ```bash VLLM_USE_DEEP_GEMM=0 VLLM_MOE_USE_DEEP_GEMM=0 ``` With that, fp8_block serves fine on 2x 5090 and is the better choice than nvfp4 if you have the second card — 8-bit weights instead of W4A4, and faster: 177.6 tok/s with the drafter at 262,144 context (rope-extended, so outside the recipe as written). nvfp4 is unaffected and loads with DeepGEMM at its default. A note on `max_tokens`: the channel-scoped output means a tight budget can truncate before the final channel closes, returning empty `content` with `finish_reason: stop`. Give it room. ## Speculative decoding Muse Glimmer ships a DFlash block-diffusion draft head. Rather than proposing one token at a time, it predicts a whole block in a single forward — the slots attend to each other bidirectionally — and the target verifies them in parallel. DFlash uses 16 query positions per active request during warmup. To keep the expanded batch within vLLM's token budget, replace the earlier sequence limit with: ```bash --max-num-seqs 32 \ --speculative-config '{"method": "dflash", "model": "meta-models/Muse-Glimmer-30B-assistant", "num_speculative_tokens": 15}' ``` **`num_speculative_tokens: 15` is fixed, not tuned.** The head has `block_size: 16` and slot 0 re-presents the last accepted token, so 15 slots remain to predict. The head is 5 layers / 5.11 GB, reads the target's residual stream at layers `[1, 13, 25, 37, 49]`, and ties its embedding and output projection to the target — so it stores neither and is useless on its own. It must be paired with the specific target it was distilled against, not merely a compatible one. --- # IFM/K2-Horizon-375B-A23B on vLLM Source: https://recipes.vllm.ai/IFM/K2-Horizon-375B-A23B Provider: IFM Released: 2026-09-01T23:25:09.000Z Updated: 2026-09-08 Architecture: moe Parameters: 379.17B Context length: 524,288 Tasks: text Verified hardware: H200, MI355X Summary: Frontier-scale sparse MoE model for long-context research and high-capacity serving ## Variants - default: precision=bf16, vram_minimum_gb=911 BF16 serving runtime with an eight-GPU tensor-parallel baseline ## Recommended base flags ```bash vllm serve 'IFM/K2-Horizon-375B-A23B' \ --trust-remote-code \ --chat-template-content-format \ string \ --dtype \ bfloat16 ``` ## Guide ## Overview K2-Horizon-375B-A23B is a frontier-scale mixture-of-expert (MoE) model in the IFM K2-Horizon family, with 379.17B stored parameters including embeddings, 26.67B active parameters per token, and support for up to 524,288 tokens of context. K2-Horizon is an open-weight IFM model family built for transparent foundation-model research, staged checkpoint analysis, and practical deployment. The series spans compact dense models for local experimentation, dense mid-size and large models for high-quality research workloads, and mixture-of-expert (MoE) models such as K2-Horizon-375B-A23B for frontier-scale research and higher-capacity serving. ## Launch Commands ```bash vllm serve IFM/K2-Horizon-375B-A23B \ --tensor-parallel-size 8 \ --trust-remote-code \ --chat-template-content-format string \ --dtype bfloat16 \ --enable-expert-parallel \ --reasoning-parser k2_horizon \ --enable-auto-tool-choice \ --tool-call-parser k2_horizon ``` ### AMD MI355X Verified on an 8-GPU MI355X (gfx950) node. AITER supplies the fused MoE and the unified attention backend: ```bash export VLLM_ROCM_USE_AITER=1 export NCCL_MIN_NCHANNELS=112 vllm serve IFM/K2-Horizon-375B-A23B \ --tensor-parallel-size 8 \ --trust-remote-code \ --chat-template-content-format string \ --dtype bfloat16 \ --max-model-len 131072 \ --gpu-memory-utilization 0.9 \ --attention-backend ROCM_AITER_UNIFIED_ATTN \ --generation-config auto \ --reasoning-parser k2_horizon \ --enable-auto-tool-choice \ --tool-call-parser k2_horizon ``` ## Client usage ```python from openai import OpenAI client = OpenAI( api_key="EMPTY", base_url="http://localhost:8000/v1", timeout=3600 ) resp = client.chat.completions.create( model="IFM/K2-Horizon-375B-A23B", messages=[{"role": "user", "content": "Give me three primes above 100."}], temperature=1.0, top_p=0.95, max_tokens=2048, ) print(resp.choices[0].message.content) ``` ### Thinking modes The model supports selectable thinking through `chat_template_kwargs`, per request or server-wide via `--default-chat-template-kwargs`: - `{"reasoning_effort": "high"}` — full thinking (default). - `{"reasoning_effort": "medium"}` — faster thinking. - `{"reasoning_effort": "low"}` — fastest thinking. ## References - [IFM/K2-Horizon-375B-A23B on Hugging Face](https://huggingface.co/IFM/K2-Horizon-375B-A23B) - [K2 Horizon Collection](https://huggingface.co/collections/IFM/k2-horizon) --- # inclusionAI/Ling-3.0-flash-VL on vLLM Source: https://recipes.vllm.ai/inclusionAI/Ling-3.0-flash-VL Provider: inclusionAI Released: 2026-09-04T15:24:37.000Z Updated: 2026-09-08 Architecture: moe Parameters: 124.85B Context length: 131,072 Min vLLM: nightly Tasks: text, multimodal Summary: Ling-3.0-flash vision-language MoE model with BF16, FP8, FP4, and INT4 checkpoints ## Variants - default: precision=bf16, vram_minimum_gb=300 BF16 checkpoint; defaults to TP4 - fp8: precision=fp8, vram_minimum_gb=150, model_id=inclusionAI/Ling-3.0-flash-VL-fp8 Serialized block-FP8 checkpoint; defaults to TP2 - fp4: precision=fp4, vram_minimum_gb=85, model_id=inclusionAI/Ling-3.0-flash-VL-fp4 Mixed block-FP8 and MXFP4 checkpoint; defaults to TP1 - int4: precision=int4, vram_minimum_gb=95, model_id=inclusionAI/Ling-3.0-flash-VL-int4 INT4 checkpoint; defaults to TP1 ## Recommended base flags ```bash vllm serve 'inclusionAI/Ling-3.0-flash-VL' \ --trust-remote-code \ --dtype \ bfloat16 \ --gpu-memory-utilization \ 0.9 \ --enable-chunked-prefill \ --compilation-config \ {"cudagraph_mode":"FULL_AND_PIECEWISE"} \ --enable-prefix-caching ``` ## Guide ## Overview `inclusionAI/Ling-3.0-flash-VL` uses the `BailingMoeV3VLForConditionalGeneration` architecture. It combines the Ling-3.0-flash hybrid MLA/KDA language backbone with a vision encoder and projector for image understanding. The language backbone has 512 routed experts (8 active per token) and one shared expert. BF16, FP8, mixed block-FP8/MXFP4, and INT4 checkpoints are available. The vision tower and projector remain BF16 in the mixed FP4 checkpoint. ## Prerequisites - **vLLM:** an official nightly build containing Ling-3.0-flash-VL support - **Precision:** BF16, FP8, FP4, or INT4 weights with BF16 compute - **Context length:** 131,072 tokens ## Launching the Server ```bash NCCL_DEBUG=WARN vllm serve inclusionAI/Ling-3.0-flash-VL \ --trust-remote-code \ --dtype bfloat16 \ --tensor-parallel-size 4 \ --gpu-memory-utilization 0.9 \ --enable-chunked-prefill \ --compilation-config '{"cudagraph_mode":"FULL_AND_PIECEWISE"}' \ --enable-prefix-caching \ --enable-auto-tool-choice \ --tool-call-parser ling3 \ --reasoning-parser ling3 ``` For a quantized variant, use the same flags with the corresponding model ID and tensor parallel size: | Variant | Model ID | TP | |---|---|---:| | FP8 | `inclusionAI/Ling-3.0-flash-VL-fp8` | 2 | | FP4 | `inclusionAI/Ling-3.0-flash-VL-fp4` | 1 | | INT4 | `inclusionAI/Ling-3.0-flash-VL-int4` | 1 | ## Image Requests ```python import base64 from pathlib import Path from openai import OpenAI client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY") image = base64.b64encode(Path("image.png").read_bytes()).decode("ascii") response = client.chat.completions.create( model="inclusionAI/Ling-3.0-flash-VL", messages=[{ "role": "user", "content": [ {"type": "image_url", "image_url": {"url": f"data:image/png;base64,{image}"}}, {"type": "text", "text": "Describe the image."}, ], }], temperature=0.0, max_tokens=1024, extra_body={"chat_template_kwargs": {"enable_thinking": False}}, ) print(response.choices[0].message.content) ``` For multiple images, add more `image_url` entries to `content`. For text-only requests, use ordinary string content. When serving a quantized variant, use its checkpoint ID in the client request. ## Thinking Mode As with Ling-3.0-flash, thinking is selected per request through `chat_template_kwargs`. Set `enable_thinking` to `True` to enable it; with `--reasoning-parser ling3`, the parsed trace is in `message.reasoning` and the final answer is in `message.content`. ## References - [Ling-3.0-flash-VL](https://huggingface.co/inclusionAI/Ling-3.0-flash-VL) - [Ling-3.0-flash-VL-fp8](https://huggingface.co/inclusionAI/Ling-3.0-flash-VL-fp8) - [Ling-3.0-flash-VL-fp4](https://huggingface.co/inclusionAI/Ling-3.0-flash-VL-fp4) - [Ling-3.0-flash-VL-int4](https://huggingface.co/inclusionAI/Ling-3.0-flash-VL-int4) --- # openbmb/MiniCPM5-2B on vLLM Source: https://recipes.vllm.ai/openbmb/MiniCPM5-2B Provider: MiniCPM (OpenBMB) Released: 2026-09-06T11:12:19.000Z Updated: 2026-09-07 Architecture: dense Parameters: 2B Context length: 131,072 Min vLLM: 0.21.0 Tasks: text Summary: MiniCPM5-2B — dense 2B LLM with hybrid Think/No-Think reasoning, native 128K context, and native tool calling support, built on the standard Llama architecture ## Variants - default: precision=bf16, vram_minimum_gb=5 BF16 weights (~2B params) — fits on a single consumer GPU ## Guide ## Overview **MiniCPM5-2B** is the 2B checkpoint in OpenBMB's MiniCPM5 series — a dense model built for on-device and resource-constrained deployment, with strong performance on agentic tool use, code generation, and reasoning tasks. It uses the **standard `LlamaForCausalLM` architecture**, so vLLM loads it natively with no custom kernels or model-code fork. Its headline feature is **hybrid reasoning**: a single checkpoint serves as both a fast assistant (No-Think) and a deliberate reasoner (Think), toggled by the chat template's `enable_thinking` flag. ## Prerequisites - **vLLM ≥ 0.21.0** — MiniCPM5-2B is supported natively as of the v0.21.0 release. ## Launch command Use the command builder above. The baseline is simply: ```bash vllm serve openbmb/MiniCPM5-2B --port 8000 ``` At ~2B params the model fits on a single GPU (TP=1). It supports the full native 128K context; drop `--max-model-len` to `8192` / `32768` to free KV cache on small GPUs. ## Reasoning modes Toggle the **Reasoning** feature to serve with deep-thinking on by default (`--default-chat-template-kwargs '{"enable_thinking": true}'`). You can also flip it per request via `chat_template_kwargs`: ```bash curl http://localhost:8000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "openbmb/MiniCPM5-2B", "messages": [{"role": "user", "content": "Explain GQA in one sentence."}], "temperature": 0.9, "top_p": 0.95, "max_tokens": 1024, "chat_template_kwargs": {"enable_thinking": true} }' ``` | Mode | `enable_thinking` | `temperature` | `top_p` | | --- | --- | --- | --- | | Think | `true` | 0.9 | 0.95 | | No-Think | `false` | 1.0 | 0.95 | ## Tool calling MiniCPM5-2B emits **XML-style** tool calls. The `minicpm5` parser ([PR #43175](https://github.com/vllm-project/vllm/pull/43175)) is natively supported in vLLM. Enable it with: ```bash vllm serve openbmb/MiniCPM5-2B --port 8000 \ --enable-auto-tool-choice \ --tool-call-parser minicpm5 ``` ## References - [Model card](https://huggingface.co/openbmb/MiniCPM5-2B) - [vLLM deployment cookbook](https://github.com/OpenBMB/MiniCPM/blob/main/docs/deployment/vllm.md) - [MiniCPM Tech Report](https://arxiv.org/pdf/2506.07900) --- # IFM/K2-Horizon-MoVA-36B-A4B on vLLM Source: https://recipes.vllm.ai/IFM/K2-Horizon-MoVA-36B-A4B Provider: IFM Released: 2026-09-01T23:01:25.000Z Updated: 2026-09-03 Architecture: moe Parameters: 37.44B Context length: 524,288 Tasks: text Verified hardware: H200 Summary: Sparse MoVA and MoE long-context model for research and production-style serving experiments ## Variants - default: precision=bf16, vram_minimum_gb=90 BF16 serving runtime with native vLLM execution ## Recommended base flags ```bash vllm serve 'IFM/K2-Horizon-MoVA-36B-A4B' \ --trust-remote-code \ --chat-template-content-format \ string \ --dtype \ bfloat16 \ --enable-expert-parallel ``` ## Guide ## Overview K2-Horizon-MoVA-36B-A4B is a mixture-of-expert (MoE) model in the IFM K2-Horizon family. It combines MoVA attention and MoE layers, with 37.44B stored parameters including embeddings, 5.95B active parameters per token, and support for up to 524,288 tokens of context. K2-Horizon is an open-weight IFM model family built for transparent foundation-model research, staged checkpoint analysis, and practical deployment. The series spans compact dense models for local experimentation, dense mid-size and large models for high-quality research workloads, and mixture-of-expert (MoE) models such as K2-Horizon-MoVA-36B-A4B for higher-capacity serving. ## Launch Commands ```bash vllm serve IFM/K2-Horizon-MoVA-36B-A4B \ --tensor-parallel-size 2 \ --trust-remote-code \ --chat-template-content-format string \ --dtype bfloat16 \ --enable-expert-parallel \ --reasoning-parser k2_horizon \ --enable-auto-tool-choice \ --tool-call-parser k2_horizon ``` ## Client usage ```python from openai import OpenAI client = OpenAI( api_key="EMPTY", base_url="http://localhost:8000/v1", timeout=3600 ) resp = client.chat.completions.create( model="IFM/K2-Horizon-MoVA-36B-A4B", messages=[{"role": "user", "content": "Give me three primes above 100."}], temperature=1.0, top_p=0.95, max_tokens=2048, ) print(resp.choices[0].message.content) ``` ### Thinking modes The model supports selectable thinking through `chat_template_kwargs`, per request or server-wide via `--default-chat-template-kwargs`: - `{"reasoning_effort": "high"}` — full thinking (default). - `{"reasoning_effort": "medium"}` — faster thinking. - `{"reasoning_effort": "low"}` — fastest thinking. ## References - [IFM/K2-Horizon-MoVA-36B-A4B on Hugging Face](https://huggingface.co/IFM/K2-Horizon-MoVA-36B-A4B) - [K2 Horizon Collection](https://huggingface.co/collections/IFM/k2-horizon) --- # IFM/K2-Horizon-3.7B on vLLM Source: https://recipes.vllm.ai/IFM/K2-Horizon-3.7B Provider: IFM Released: 2026-09-01T23:14:18.000Z Updated: 2026-09-02 Architecture: dense Parameters: 5.06B Context length: 524,288 Tasks: text Verified hardware: H200 Summary: Dense model for long-context research, serving, and downstream adaptation ## Variants - default: precision=bf16, vram_minimum_gb=13 BF16 serving runtime with a 131K context baseline ## Recommended base flags ```bash vllm serve 'IFM/K2-Horizon-3.7B' \ --trust-remote-code \ --chat-template-content-format \ string \ --dtype \ bfloat16 ``` ## Guide ## Overview K2-Horizon-3.7B is a small dense model in the IFM K2-Horizon family, with 5.06B stored parameters including embeddings and support for up to 524,288 tokens of context. K2-Horizon is an open-weight IFM model family built for transparent foundation-model research, staged checkpoint analysis, and practical deployment. The series spans compact dense models for local experimentation, dense mid-size and large models for high-quality research workloads, and mixture-of-expert (MoE) models for higher-capacity serving. ## Launch Commands ```bash vllm serve IFM/K2-Horizon-3.7B \ --trust-remote-code \ --chat-template-content-format string \ --dtype bfloat16 \ --reasoning-parser k2_horizon \ --enable-auto-tool-choice \ --tool-call-parser k2_horizon ``` ## Client usage ```python from openai import OpenAI client = OpenAI( api_key="EMPTY", base_url="http://localhost:8000/v1", timeout=3600 ) resp = client.chat.completions.create( model="IFM/K2-Horizon-3.7B", messages=[{"role": "user", "content": "Give me three primes above 100."}], temperature=1.0, top_p=0.95, max_tokens=2048, ) print(resp.choices[0].message.content) ``` ### Thinking modes The model supports selectable thinking through `chat_template_kwargs`, per request or server-wide via `--default-chat-template-kwargs`: - `{"reasoning_effort": "high"}` — full thinking (default). - `{"reasoning_effort": "medium"}` — faster thinking. - `{"reasoning_effort": "low"}` — fastest thinking. ## References - [IFM/K2-Horizon-3.7B on Hugging Face](https://huggingface.co/IFM/K2-Horizon-3.7B) - [K2 Horizon Collection](https://huggingface.co/collections/IFM/k2-horizon) --- # IFM/K2-Horizon-32B on vLLM Source: https://recipes.vllm.ai/IFM/K2-Horizon-32B Provider: IFM Released: 2026-09-01T23:14:02.000Z Updated: 2026-09-02 Architecture: dense Parameters: 34.78B Context length: 524,288 Tasks: text Verified hardware: H200 Summary: Dense long-context model for reasoning research and high-capacity serving ## Variants - default: precision=bf16, vram_minimum_gb=84 BF16 serving runtime with native vLLM execution ## Recommended base flags ```bash vllm serve 'IFM/K2-Horizon-32B' \ --trust-remote-code \ --chat-template-content-format \ string \ --dtype \ bfloat16 ``` ## Guide ## Overview K2-Horizon-32B is a large dense model in the IFM K2-Horizon family, with 34.78B stored parameters including embeddings and support for up to 524,288 tokens of context. K2-Horizon is an open-weight IFM model family built for transparent foundation-model research, staged checkpoint analysis, and practical deployment. The series spans compact dense models for local experimentation, dense mid-size and large models such as K2-Horizon-32B for high-quality research workloads, and mixture-of-expert (MoE) models for higher-capacity serving. ## Launch Commands ```bash vllm serve IFM/K2-Horizon-32B \ --tensor-parallel-size 2 \ --trust-remote-code \ --chat-template-content-format string \ --dtype bfloat16 \ --reasoning-parser k2_horizon \ --enable-auto-tool-choice \ --tool-call-parser k2_horizon ``` ## Client usage ```python from openai import OpenAI client = OpenAI( api_key="EMPTY", base_url="http://localhost:8000/v1", timeout=3600 ) resp = client.chat.completions.create( model="IFM/K2-Horizon-32B", messages=[{"role": "user", "content": "Give me three primes above 100."}], temperature=1.0, top_p=0.95, max_tokens=2048, ) print(resp.choices[0].message.content) ``` ### Thinking modes The model supports selectable thinking through `chat_template_kwargs`, per request or server-wide via `--default-chat-template-kwargs`: - `{"reasoning_effort": "high"}` — full thinking (default). - `{"reasoning_effort": "medium"}` — faster thinking. - `{"reasoning_effort": "low"}` — fastest thinking. ## References - [IFM/K2-Horizon-32B on Hugging Face](https://huggingface.co/IFM/K2-Horizon-32B) - [K2 Horizon Collection](https://huggingface.co/collections/IFM/k2-horizon) --- # IFM/K2-Horizon-7B on vLLM Source: https://recipes.vllm.ai/IFM/K2-Horizon-7B Provider: IFM Released: 2026-09-01T23:14:30.000Z Updated: 2026-09-02 Architecture: dense Parameters: 9B Context length: 524,288 Tasks: text Verified hardware: H200 Summary: Dense model for long-context research, fine-tuning, and cost-conscious deployment ## Variants - default: precision=bf16, vram_minimum_gb=22 BF16 serving runtime with a 131K context baseline ## Recommended base flags ```bash vllm serve 'IFM/K2-Horizon-7B' \ --trust-remote-code \ --chat-template-content-format \ string \ --dtype \ bfloat16 ``` ## Guide ## Overview K2-Horizon-7B is a compact dense model in the IFM K2-Horizon family, with 9B stored parameters including embeddings and support for up to 524,288 tokens of context. K2-Horizon is an open-weight IFM model family built for transparent foundation-model research, staged checkpoint analysis, and practical deployment. The series spans compact dense models such as K2-Horizon-7B for local experimentation, dense mid-size and large models for high-quality research workloads, and mixture-of-expert (MoE) models for higher-capacity serving. ## Launch Commands ```bash vllm serve IFM/K2-Horizon-7B \ --trust-remote-code \ --chat-template-content-format string \ --dtype bfloat16 \ --reasoning-parser k2_horizon \ --enable-auto-tool-choice \ --tool-call-parser k2_horizon ``` ## Client usage ```python from openai import OpenAI client = OpenAI( api_key="EMPTY", base_url="http://localhost:8000/v1", timeout=3600 ) resp = client.chat.completions.create( model="IFM/K2-Horizon-7B", messages=[{"role": "user", "content": "Give me three primes above 100."}], temperature=1.0, top_p=0.95, max_tokens=2048, ) print(resp.choices[0].message.content) ``` ### Thinking modes The model supports selectable thinking through `chat_template_kwargs`, per request or server-wide via `--default-chat-template-kwargs`: - `{"reasoning_effort": "high"}` — full thinking (default). - `{"reasoning_effort": "medium"}` — faster thinking. - `{"reasoning_effort": "low"}` — fastest thinking. ## References - [IFM/K2-Horizon-7B on Hugging Face](https://huggingface.co/IFM/K2-Horizon-7B) - [K2 Horizon Collection](https://huggingface.co/collections/IFM/k2-horizon) --- # IFM/K2-Horizon-0.9B on vLLM Source: https://recipes.vllm.ai/IFM/K2-Horizon-0.9B Provider: IFM Released: 2026-09-01T22:55:52.000Z Updated: 2026-08-31 Architecture: dense Parameters: 1.08B Context length: 131,072 Tasks: text Verified hardware: H200 Summary: Compact dense model for mathematics, code, instruction following, and STEM tasks ## Variants - default: precision=bf16, vram_minimum_gb=3 Released BF16 checkpoint; the command starts at an 8K runtime context ## Recommended base flags ```bash vllm serve 'IFM/K2-Horizon-0.9B' \ --trust-remote-code \ --chat-template-content-format \ string \ --dtype \ bfloat16 \ --max-model-len \ 131072 \ --gpu-memory-utilization \ 0.85 ``` ## Guide ## Overview K2-Horizon-0.9B is a compact dense model in the IFM K2-Horizon family, with 1.08B stored parameters including embeddings and a 131,072-token context window. K2-Horizon is an open-weight IFM model family built for transparent foundation-model research, staged checkpoint analysis, and practical deployment. The series spans compact dense models such as K2-Horizon-0.9B for local experimentation, dense mid-size and large models for high-quality research workloads, and mixture-of-expert (MoE) models for higher-capacity serving. ## Launch Commands K2-Horizon-0.9B's `config.json` already enables YaRN, so the full 131,072-token context needs no RoPE override. ```bash vllm serve IFM/K2-Horizon-0.9B \ --trust-remote-code \ --chat-template-content-format string \ --dtype bfloat16 \ --max-model-len 131072 \ --gpu-memory-utilization 0.85 \ --reasoning-parser k2_horizon \ --enable-auto-tool-choice \ --tool-call-parser k2_horizon ``` Lower `--max-model-len` (e.g. 8192) to reduce KV-cache memory. ## Client usage ```python from openai import OpenAI client = OpenAI( api_key="EMPTY", base_url="http://localhost:8000/v1", timeout=3600 ) resp = client.chat.completions.create( model="IFM/K2-Horizon-0.9B", messages=[{"role": "user", "content": "Give me three primes above 100."}], temperature=1.0, top_p=0.95, max_tokens=2048, ) print(resp.choices[0].message.content) ``` ### Thinking modes The model supports selectable thinking through `chat_template_kwargs`, per request or server-wide via `--default-chat-template-kwargs`: - `{"reasoning_effort": "high"}` — full thinking (default). - `{"reasoning_effort": "medium"}` — faster thinking. - `{"reasoning_effort": "low"}` — fastest thinking. ## References - [IFM/K2-Horizon-0.9B on Hugging Face](https://huggingface.co/IFM/K2-Horizon-0.9B) - [K2 Horizon Collection](https://huggingface.co/collections/IFM/k2-horizon) --- # ibm-granite/granite-3.2-2b-instruct on vLLM Source: https://recipes.vllm.ai/ibm-granite/granite-3.2-2b-instruct Provider: IBM Granite Released: 2025-02-17T22:52:35.000Z Updated: 2026-08-19 Architecture: dense Parameters: 2B Context length: 131,072 Min vLLM: 0.6.1 Tasks: text Verified hardware: Xeon 6 Summary: IBM Granite 3.2 2B instruction-tuned dense language model with long-context support. ## Variants - default: precision=bf16, vram_minimum_gb=6 Full precision BF16 ## Guide ## Overview Granite-3.2-2B-Instruct is IBM's compact instruction-tuned dense model with long-context support. ## Prerequisites - Hardware: Intel Xeon 6/Xeon 5 CPUs - vLLM >= 0.6.1 ### Docker (Intel Xeon 6 CPUs) ```bash docker pull vllm/vllm-openai-cpu:latest-x86_64 ``` ## Intel Xeon 6 ```bash vllm serve ibm-granite/granite-3.2-2b-instruct ``` TP/DP values and host-specific CPU placement remain deployment choices and are not hard-coded into the portable recipe. ## Runtime and Platform Tuning The following settings are intentionally not prescribed as portable CPU model defaults because they depend on the workload, hardware, or runtime environment: - `--max-num-batched-tokens`: scheduler/throughput tuning. - `--max-num-seqs`: concurrency and scheduler-capacity tuning. - `--gpu-memory-utilization`: platform memory-budget tuning. - `--no-enable-prefix-caching`: workload/benchmark cache-policy tuning. - `VLLM_ENGINE_ITERATION_TIMEOUT_S`: operational runtime timeout. These settings may still appear in validated hardware-specific overrides. Tune them at deployment time based on platform resources, workload shape, and latency/throughput goals. ## References - [Model card](https://huggingface.co/ibm-granite/granite-3.2-2b-instruct) - [vLLM CPU installation](https://docs.vllm.ai/en/latest/getting_started/installation/cpu/) --- # Lightricks/LTX-2.5-Diffusers on vLLM Source: https://recipes.vllm.ai/Lightricks/LTX-2.5-Diffusers Provider: Lightricks Released: 2026-07-26T06:42:34.000Z Updated: 2026-08-15 Architecture: dense Parameters: 19B Min vLLM: 0.27.0 Tasks: omni Verified hardware: H200, B300 Summary: 19B diffusion transformer for joint video and synchronized audio generation, served via vLLM-Omni ## Variants - default: precision=bf16, vram_minimum_gb=80 Original BF16 weights with Full/SFT and distilled one-stage/two-stage generation paths - fp8: precision=fp8, vram_minimum_gb=61 Online FP8 quantization for lower memory use ## Guide ## Overview [LTX-2.5-Diffusers](https://huggingface.co/Lightricks/LTX-2.5-Diffusers) is a dense 19B diffusion transformer for video with synchronized 48 kHz stereo audio. vLLM-Omni supports T2V and first-frame I2V through four pipeline classes: | Pipeline | Mode | Default size | Schedule | |----------|------|------------|----------| | `LTX2Pipeline` | Full/SFT one-stage | 960x544 | 30 steps | | `LTX2TwoStagePipeline` | Full/SFT two-stage | 1920x1088 | 30 + 3 steps | | `LTX2DistilledOneStagePipeline` | Distilled one-stage | 960x544 | 8 steps | | `LTX2DistilledTwoStagePipeline` | Distilled two-stage | 1920x1088 | 8 + 3 steps | Select the class with `--model-class-name`; no `--task-type` flag is needed. The supported model ID is `Lightricks/LTX-2.5-Diffusers`. The gated raw `Lightricks/LTX-2.5` repository only supplies official two-stage sidecars and is not directly loadable with `--model`. ## Quantization Choose **BF16** to keep the original weights, or **FP8** to add online `--quantization fp8` and reduce memory use. The command builder disables a quantization option when it is not available on the selected hardware. FP8 is available on the listed NVIDIA Hopper and Blackwell profiles; AMD uses BF16. B200, GB200, B300, and GB300 use cuDNN attention for both options. ## Prerequisites - Accept both Lightricks model licenses and run `hf auth login`. - Use current vLLM-Omni main, targeting vLLM 0.27+. - Ensure `ffmpeg` and `ffprobe` are on `PATH`. - I2V requires PyAV with the `libx264` encoder. ## Installation ```bash uv venv && source .venv/bin/activate uv pip install git+https://github.com/vllm-project/vllm-omni.git hf auth login ``` ## Offline inference Choose `PIPELINE`, `WIDTH`, `HEIGHT`, and `STEPS` from the table. This example selects the distilled two-stage path: ```bash export MODEL=Lightricks/LTX-2.5-Diffusers export PIPELINE=LTX2DistilledTwoStagePipeline export WIDTH=1920 HEIGHT=1088 STEPS=8 python examples/offline_inference/text_to_video/text_to_video.py \ --model "${MODEL}" --model-class-name "${PIPELINE}" \ --prompt "A cinematic red fox walking through a snowy forest at dawn." \ --width "${WIDTH}" --height "${HEIGHT}" --num-frames 121 \ --num-inference-steps "${STEPS}" --frame-rate 24 --fps 24 --seed 42 \ --output ltx25-t2v.mp4 ``` For first-frame I2V, use the generic image-to-video example with the same pipeline values and one image: ```bash python examples/offline_inference/image_to_video/image_to_video.py \ --model "${MODEL}" --model-class-name "${PIPELINE}" \ --image /absolute/path/to/first-frame.png \ --prompt "The red fox walks forward while the camera tracks alongside." \ --width "${WIDTH}" --height "${HEIGHT}" --num-frames 121 \ --num-inference-steps "${STEPS}" --frame-rate 24 --fps 24 --seed 42 \ --output ltx25-i2v.mp4 ``` ## Online serving ```bash vllm serve "${MODEL}" \ --omni \ --model-class-name "${PIPELINE}" ``` Use the task presets on this page for complete T2V and I2V requests. Restart the server after changing `PIPELINE`. ## Constraints - Online `width`, `height`, `num_frames`, `fps`, and `num_inference_steps` are optional and use the selected pipeline defaults when omitted. `seed` is optional and random when omitted. - `num_frames` must be `8k+1` when overridden. - One-stage dimensions must be divisible by 32; two-stage final dimensions must be divisible by 64. - Distilled pipelines are positive-only and reject negative prompts. - I2V accepts exactly one initial RGB image per prompt and uses the official CRF-18 conditioning path by default. ## References - [LTX-2.5 Diffusers model card](https://huggingface.co/Lightricks/LTX-2.5-Diffusers) - [Official LTX-2 repository](https://github.com/Lightricks/LTX-2) - [vLLM-Omni LTX-2.5 recipe](https://github.com/vllm-project/vllm-omni/blob/main/recipes/LTX/LTX-2.5.md) --- # stabilityai/stable-audio-open-1.0 on vLLM Source: https://recipes.vllm.ai/stabilityai/stable-audio-open-1.0 Provider: Stability AI Released: 2024-05-24T05:11:20.000Z Updated: 2026-08-15 Architecture: dense Parameters: 1.2B Min vLLM: 0.22.0 Tasks: omni Summary: Text-to-audio generation model (1.2B params) producing up to ~47 s stereo audio at 44.1 kHz, served via vLLM-Omni ## Variants - default: precision=bf16, vram_minimum_gb=19 BF16 weights for text-to-audio generation (via vLLM-Omni) ## Recommended base flags ```bash vllm serve 'stabilityai/stable-audio-open-1.0' \ --trust-remote-code \ --enforce-eager \ --gpu-memory-utilization \ 0.9 ``` ## Guide ## Overview [Stable Audio Open 1.0](https://huggingface.co/stabilityai/stable-audio-open-1.0) is Stability AI's text-to-audio generation model (~1.2B parameters). It produces stereo audio at 44.1 kHz, up to ~47 seconds. Served via **vLLM-Omni** (not standard vLLM). Limitations: - No realistic vocals (no singing or speech). - English-only training data. - Better at sound effects than complex music. ## Prerequisites - vLLM-Omni on top of vLLM 0.22.0 or newer - `soundfile` or `scipy` for saving audio ## Installation ```bash uv venv source .venv/bin/activate uv pip install git+https://github.com/vllm-project/vllm-omni.git # Audio saving uv pip install soundfile ``` ## Python Usage ```python import torch import soundfile as sf from vllm_omni.entrypoints.omni import Omni omni = Omni(model="stabilityai/stable-audio-open-1.0") generator = torch.Generator(device="cuda").manual_seed(42) audio = omni.generate( "The sound of a dog barking", negative_prompt="Low quality.", generator=generator, guidance_scale=7.0, num_inference_steps=100, extra={"audio_start_in_s": 0.0, "audio_end_in_s": 10.0}, ) audio_data = audio[0].cpu().float().numpy().T # [samples, channels] sf.write("output.wav", audio_data, 44100) ``` ## CLI Usage (from vLLM-Omni repo) ```bash python examples/offline_inference/text_to_audio/text_to_audio.py \ --model stabilityai/stable-audio-open-1.0 \ --prompt "The sound of a dog barking" \ --audio-length 10.0 \ --num-inference-steps 100 \ --guidance-scale 7.0 \ --output dog_barking.wav ``` ## Key Parameters | Parameter | Default | Description | |-----------|---------|-------------| | `audio_start_in_s` | 0.0 | Start time in seconds | | `audio_end_in_s` | 10.0 | End time in seconds | | `num_inference_steps` | 100 | Denoising steps (higher = better quality, slower) | | `guidance_scale` | 7.0 | Classifier-free guidance scale | | `negative_prompt` | "Low quality." | Text to avoid | | `num_waveforms` | 1 | Samples per prompt | | `sample_rate` | 44100 | Output sample rate (Hz) | ## License Released under the Stability AI Community License. Commercial use requires a separate license. ## References - [Stable Audio Open on Hugging Face](https://huggingface.co/stabilityai/stable-audio-open-1.0) - [vLLM-Omni text-to-audio example](https://github.com/vllm-project/vllm-omni/blob/main/examples/offline_inference/text_to_audio/text_to_audio.py) --- # IndexTeam/IndexTTS-2.5 on vLLM Source: https://recipes.vllm.ai/IndexTeam/IndexTTS-2.5 Provider: IndexTeam Released: 2026-08-10T07:49:03.000Z Updated: 2026-08-12 Architecture: dense Parameters: Undisclosed Min vLLM: 0.27.0 Tasks: omni Summary: Multilingual zero-shot voice-cloning TTS with native speed, emotion, and text-normalization controls, served through vLLM-Omni's OpenAI-compatible speech API. ## Variants - default: precision=bf16, vram_minimum_gb=96 Native BF16 bundle using the standard two-stage single-GPU deploy profile; lower-memory configurations have not been characterized ## Recommended base flags ```bash vllm serve 'IndexTeam/IndexTTS-2.5' \ --trust-remote-code ``` ## Guide ## Overview IndexTTS-2.5 is a two-stage text-to-speech system: an autoregressive talker produces semantic codes, then EnhancedCodec, S2Mel CFM/DiT, and BigVGAN synthesize **22.05 kHz mono WAV** audio. The vLLM-Omni integration exposes zero-shot voice cloning, reusable named voices, multilingual text processing, text normalization, model-native speed control, and three emotion-conditioning modes through `/v1/audio/speech`. The model is available from [IndexTeam/IndexTTS-2.5 on Hugging Face](https://huggingface.co/IndexTeam/IndexTTS-2.5). ## Prerequisites - Download access to [IndexTeam/IndexTTS-2.5 on Hugging Face](https://huggingface.co/IndexTeam/IndexTTS-2.5). vLLM-Omni also accepts an already-downloaded native bundle and its nested upstream `checkpoints/` layout. - Use a CUDA GPU with enough memory for both stages. The standard deploy config reserves up to `0.4` GPU-memory utilization per stage on one GPU; lower-memory profiles have not been validated. - Follow the merged [vLLM-Omni IndexTTS-2.5 recipe](https://github.com/vllm-project/vllm-omni/blob/main/recipes/IndexTeam/IndexTTS-2_5.md) for the upstream serving requirements. ## Installation ```bash uv venv && source .venv/bin/activate uv pip install "vllm-omni[indextts2] @ git+https://github.com/vllm-project/vllm-omni.git" ``` ## Launch the server IndexTTS-2.5's standard two-stage deploy config is selected automatically: ```bash vllm serve IndexTeam/IndexTTS-2.5 \ --omni \ --trust-remote-code \ --port 8092 ``` ## Quick start: upload and reuse a voice Upload a reference recording once. The voice-storage API requires a consent identifier: ```bash curl -X POST http://localhost:8092/v1/audio/voices \ -F "audio_sample=@/path/to/reference.wav" \ -F "consent=user-consent-id" \ -F "name=demo_voice" \ -F "speaker_description=IndexTTS-2.5 demo voice" ``` Reuse that speaker by name: ```bash curl -X POST http://localhost:8092/v1/audio/speech \ -H "Content-Type: application/json" \ -d '{ "model": "IndexTeam/IndexTTS-2.5", "input": "你好，这是复用已上传音色的语音合成测试。", "voice": "demo_voice", "response_format": "wav", "extra_params": {"lang": "zh", "text_normalization": true} }' --output named_voice.wav ``` Use `ref_audio` instead of `voice` to clone directly from a URL, data URL, or permitted `file://` URI. There are no built-in text-only preset speakers. ## Language and text controls IndexTTS-2.5 officially supports `zh` (Chinese), `en` (English), `ja` (Japanese), `es` (Spanish), and `ar` (Arabic). Pass the corresponding code through `extra_params.lang`. vLLM-Omni also provides `zhen` as an additional Chinese-English mixed-text preprocessing mode; it is not a separate model language. Text normalization is enabled by default: ```json { "input": "Hello，欢迎使用 IndexTTS 二点五。", "voice": "demo_voice", "extra_params": { "lang": "zhen", "text_normalization": true } } ``` Spanish and Arabic requests use `"lang": "es"` and `"lang": "ar"`, respectively. Cantonese (`yue`) is not listed as an officially supported IndexTTS-2.5 language. For Japanese, write numbers, dates, and percentages as readable Japanese text because the Japanese tokenizer does not automatically expand them. ## Native speed control `speed` accepts values from `0.5` to `2.0`. Values above `1.0` produce shorter, faster speech. vLLM-Omni maps the public API convention to the model-native `duration_factor = 1 / speed`, so the audio waveform is not post-resampled: ```bash curl -X POST http://localhost:8092/v1/audio/speech \ -H "Content-Type: application/json" \ -d '{ "model": "IndexTeam/IndexTTS-2.5", "input": "这是使用模型原生语速控制的测试。", "voice": "demo_voice", "speed": 1.5, "extra_params": {"lang": "zh"} }' --output speed_1_5.wav ``` ## Emotion control IndexTTS-2.5 supports an eight-value emotion vector, an emotion description, or a separate emotion-reference recording. `emo_alpha` controls conditioning strength from `0` to `1`. Explicit vector order: `happy`, `angry`, `sad`, `afraid`, `disgusted`, `melancholic`, `surprised`, `calm`: ```json { "input": "今天真是令人开心的一天！", "voice": "demo_voice", "extra_params": { "lang": "zh", "emo_vector": [1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], "emo_alpha": 0.8 } } ``` Infer emotion from text: ```json { "input": "我们终于完成了这项工作！", "voice": "demo_voice", "extra_params": { "lang": "zh", "use_emo_text": true, "emo_text": "开心、兴奋而且充满活力", "emo_alpha": 0.8 } } ``` Transfer expression from another recording while preserving the uploaded speaker identity: ```json { "input": "请用情绪参考音频中的表达方式朗读这句话。", "voice": "demo_voice", "extra_params": { "lang": "zh", "emo_audio": "file:///path/to/emotion_reference.wav", "emo_alpha": 0.8 } } ``` When several emotion sources are supplied, precedence is `use_emo_text` > `emo_vector` > `emo_audio` > the speaker-reference emotion. ## Validation notes The integration has completed end-to-end generation with voice cloning and Whisper-small transcript checks. In the fixed-seed Chinese validation, `speed=1.0` produced 5.503 s and `speed=2.0` produced 2.752 s; both matched the requested sentence. These measurements are functional evidence, not a general throughput claim. ## References - [IndexTTS-2.5 model page](https://huggingface.co/IndexTeam/IndexTTS-2.5) - [vLLM-Omni IndexTTS-2.5 recipe](https://github.com/vllm-project/vllm-omni/blob/main/recipes/IndexTeam/IndexTTS-2_5.md) - [vLLM-Omni repository](https://github.com/vllm-project/vllm-omni) --- # OpenMOSS-Team/MOSS-Transcribe-Diarize on vLLM Source: https://recipes.vllm.ai/OpenMOSS-Team/MOSS-Transcribe-Diarize Provider: OpenMOSS Released: 2026-05-19T08:19:23.000Z Updated: 2026-08-10 Architecture: dense Parameters: 0.9B Min vLLM: 0.27.0 Tasks: multimodal Summary: OpenMOSS's 0.9B end-to-end multi-speaker long-audio transcription model with timestamps and speaker labels, served through vLLM's OpenAI-compatible /v1/audio/transcriptions API. ## Variants - default: precision=bf16, vram_minimum_gb=4 MOSS-Transcribe-Diarize-0.9B - single-GPU long-form multi-speaker transcription ## Recommended base flags ```bash vllm serve 'OpenMOSS-Team/MOSS-Transcribe-Diarize' \ --trust-remote-code ``` ## Guide ## Overview [MOSS-Transcribe-Diarize](https://huggingface.co/OpenMOSS-Team/MOSS-Transcribe-Diarize) is a 0.9B end-to-end speech-to-text model for long-form multi-speaker transcription. It uses a Whisper-style audio encoder and a Qwen3-style causal decoder to generate structured transcripts with text, timestamps, and speaker labels in a single pass. Example output: ```text [0.11][S01] Good morning! [1.03] [1.11][S02] Morning, let's start the meeting. [2.42] ``` Key capabilities: - End-to-end multi-speaker transcription without a separate ASR, diarization, or alignment pipeline. - Structured text output with speaker IDs and timestamps. - Long-audio transcription for meetings, interviews, podcasts, and customer support recordings. - Prompt-based hotwords for names, organizations, product names, project codes, and domain vocabulary. ## Prerequisites Install vLLM with audio dependencies. ```bash uv venv source .venv/bin/activate uv pip install -U "vllm[audio]" --torch-backend auto ``` ## Launching with vLLM ```bash vllm serve OpenMOSS-Team/MOSS-Transcribe-Diarize \ --trust-remote-code \ --served-model-name moss-mtd ``` The model uses custom Hugging Face processor code, so `--trust-remote-code` is required. ## Client Usage ### OpenAI Transcription API ```python from openai import OpenAI client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY") with open("meeting.wav", "rb") as audio_file: transcription = client.audio.transcriptions.create( model="moss-mtd", file=audio_file, response_format="diarized_json", temperature=0, ) print(transcription.text) for segment in transcription.segments: print( f"[{segment.start:.2f} - {segment.end:.2f}] " f"{segment.speaker}: {segment.text}" ) ``` ### cURL ```bash curl http://localhost:8000/v1/audio/transcriptions \ -F "model=moss-mtd" \ -F "file=@meeting.wav" \ -F "response_format=diarized_json" \ -F "temperature=0" ``` ## Output Format With `response_format=diarized_json`, vLLM returns an OpenAI-compatible response with structured speaker segments: ```json { "task": "transcribe", "duration": 5.1, "text": "Let's review the deployment plan. The vLLM server is already running.", "segments": [ { "type": "transcript.text.segment", "id": "seg_0", "start": 0.41, "end": 2.8, "text": "Let's review the deployment plan.", "speaker": "S01" }, { "type": "transcript.text.segment", "id": "seg_1", "start": 3.12, "end": 5.1, "text": "The vLLM server is already running.", "speaker": "S02" } ], "usage": { "type": "duration", "seconds": 6 } } ``` Speaker IDs such as `S01`, `S02`, and `S03` are generated by the model and are intended to remain consistent within the audio. Use `response_format=json` if you want the model's native timestamped transcript instead. ```text [0.41][S01] Let's review the deployment plan. [2.80] [3.12][S02] The vLLM server is already running. [5.10] ``` ## References - [Model card](https://huggingface.co/OpenMOSS-Team/MOSS-Transcribe-Diarize) - [GitHub repository](https://github.com/OpenMOSS/MOSS-Transcribe-Diarize) - [vLLM audio support](https://docs.vllm.ai/en/latest/features/multimodal_inputs.html) --- # thinkingmachines/Inkling on vLLM Source: https://recipes.vllm.ai/thinkingmachines/Inkling Provider: Thinking Machines Lab Released: 2026-07-14T13:23:14.000Z Updated: 2026-08-04 Architecture: moe Parameters: 975B Context length: 1,048,576 Min vLLM: 0.26.0 Tasks: multimodal Verified hardware: GB200 NVL4, B200, H200, MI300X, MI355X Summary: Natively multimodal 1T-parameter MoE from Thinking Machines Lab — text/image/audio in, text out, up to 1M context — with relative attention, short convolution, and shared expert sinks. ## Variants - default: precision=nvfp4, vram_minimum_gb=651 Weight-only NVFP4 (group size 16) on the routed MoE expert matmuls only; attention, norms, sconv, shared experts, routers, embeddings, and the layer 0-1 dense MLPs stay BF16. ~542 GB on disk. On Hopper, the MoE kernel dequantizes the NVFP4 expert weights to BF16 on the fly. - mxfp4: precision=mxfp4, vram_minimum_gb=540, model_id=lightseekorg/Inkling-MXFP4 Quark OCP MXFP4 checkpoint for AMD Instinct: TP4 on MI355X with native AITER MXFP4 MoE, or TP4 on MI300X/MI325X with Triton MXFP4 emulation. Full 1M context is validated on MI300X and MI355X; MI325X uses the same gfx942 path but is not yet smoke-tested. - bf16: precision=bf16, vram_minimum_gb=2400, model_id=thinkingmachines/Inkling Full BF16 checkpoint (~1.8 TB on disk). Runs on Hopper and Blackwell. - fp8: precision=fp8, vram_minimum_gb=1170, model_id=RedHatAI/Inkling-FP8-dynamic RedHatAI FP8 (E4M3) with dynamic per-token activation quantization - redhat_fp8: precision=fp8, vram_minimum_gb=1170, model_id=RedHatAI/Inkling-FP8-block RedHatAI FP8 block quantization - redhat_nvfp4: precision=nvfp4, vram_minimum_gb=618, model_id=RedHatAI/Inkling-NVFP4 RedHatAI NVFP4 quantization ## Recommended base flags ```bash vllm serve 'thinkingmachines/Inkling-NVFP4' \ --trust-remote-code \ --tokenizer-mode \ inkling ``` Environment: ```bash VLLM_USE_V2_MODEL_RUNNER='1' ``` ## Guide ## Overview TML Inkling is a natively multimodal 1T-parameter Mixture-of-Experts model from Thinking Machines Lab. It accepts text, image, and audio inputs and generates text, with a context length of up to 1M tokens. vLLM supports Inkling on Day 0 with full feature parity — LoRA, speculative decoding (MTP), TP/DP/EP/PP parallelism, prefix caching, and disaggregated serving. Architectural highlights: - **Attention.** 66-layer decoder backbone — 11 full-attention layers and 55 sliding-window layers — all grouped-query attention with head size 128. Inkling replaces RoPE with **relative attention**: a learned relative-position term added to the pre-softmax logits. - **Short convolution (sconv).** Each layer applies four window-4 sconv modules (on attention keys, values, output, and MoE output). vLLM manages the sconv cache as the KV cache of a virtual sliding-window layer, so prefix caching works seamlessly, and shards sconv across the channel dimension (reduce-scatter / all-gather) to avoid replicating cache and compute across TP ranks. - **MoE with expert sinks.** 256 routed experts (top-6) plus 2 shared experts — 8 experts per token. The 2 shared experts act as *expert sinks*: they participate in routing-score computation but are excluded from the top-6 candidate pool. - **MTP.** 8 chained MTP heads (single-layer full-attention transformers with a dense MLP, all BF16) enable up to 9 tokens per forward step for speculative decoding. ## Variants - **NVFP4 (default):** Only the routed experts are quantized to NVFP4; shared experts and the qkvr linears remain BF16. This is a mixed-precision checkpoint. Blackwell uses its native FP4 tensor cores; on Hopper, the MoE kernel dequantizes the NVFP4 expert weights to BF16 on the fly. - **BF16:** Full BF16 weights (`thinkingmachines/Inkling`). Runs on both **Hopper** (H200&H20) and Blackwell, at the cost of substantially more VRAM. - **MXFP4 (AMD):** `lightseekorg/Inkling-MXFP4` keeps the full 1M-token context at TP4 on MI355X and MI300X. MI355X runs native AITER MXFP4 MoE; MI300X uses Triton on-the-fly MXFP4 emulation. ## Prerequisites - **Hardware:** NVIDIA Hopper (H200/H20) or Blackwell for NVFP4 and BF16; AMD MI300X, MI325X, or MI355X with TP4 for MXFP4. - **vLLM:** Nightly wheels (Day-0 support just landed). - **Parallelism:** 1T parameters requires multi-GPU. The featured configuration is 4x GB200 (`--tensor-parallel-size 4`) for the NVFP4 variant. On Hopper, NVFP4 uses TP8 on H200; the BF16 variant needs multi-node. ## Client Usage Launch the server (NVFP4 on 4x GB200): ```bash export VLLM_USE_V2_MODEL_RUNNER=1 export FLASH_ATTENTION_CUTE_DSL_CACHE_ENABLED=1 vllm serve thinkingmachines/Inkling-NVFP4 \ --tokenizer-mode inkling \ --reasoning-parser inkling \ --tool-call-parser inkling \ --enable-auto-tool-choice \ --tensor-parallel-size 4 \ --disable-flashinfer-autotune \ --trust-remote-code ``` For a multi-node BF16 deployment on Hopper (H200/H20), select the **Multi-Node TP + Data Parallel** strategy with **Nodes = 2** in the command builder above — it renders the per-node `vllm serve` commands (TP within each node, DP across nodes, EP over all ranks). On AMD, select the **MXFP4** variant. TP4 is validated with the native 1M context on both GPU families. Equivalent MI355X launch: ```bash export VLLM_USE_V2_MODEL_RUNNER=1 export VLLM_ROCM_USE_AITER=1 export VLLM_ROCM_USE_AITER_MOE=1 export INKLING_GFX950_GLUON=0 vllm serve lightseekorg/Inkling-MXFP4 \ --tensor-parallel-size 4 \ --tokenizer-mode inkling \ --reasoning-parser inkling \ --tool-call-parser inkling \ --enable-auto-tool-choice \ --moe-backend aiter \ --block-size 128 \ --trust-remote-code ``` Query the server with the OpenAI SDK, including an image input: ```python from openai import OpenAI client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1") response = client.chat.completions.create( model="thinkingmachines/Inkling-NVFP4", messages=[{ "role": "user", "content": [ {"type": "text", "text": "Describe this image."}, {"type": "image_url", "image_url": {"url": "https://example.com/photo.jpg"}}, ], }], ) print(response.choices[0].message.content) ``` ## Performance vLLM implements Inkling with a series of optimizations: sconv-aware TP sharding (each rank stores only its channel shard of the sconv cache), low-latency fused reduce-scatter / all-gather collectives built on FlashInfer's Lamport-protocol all-reduce, the FA4 sheared-bias attention kernel for relative attention, plus kernel fusion, PDL, and multi-streaming. Enabling MTP speculative decoding adds further per-user throughput. ## References - [TML Inkling (NVFP4)](https://huggingface.co/thinkingmachines/Inkling-NVFP4) - [TML Inkling (BF16)](https://huggingface.co/thinkingmachines/Inkling) - [Inkling MXFP4](https://huggingface.co/lightseekorg/Inkling-MXFP4) - [vLLM integration PR](https://github.com/Inferact/mlj/pull/16) --- # moonshotai/Kimi-K2.5 on vLLM Source: https://recipes.vllm.ai/moonshotai/Kimi-K2.5 Provider: Moonshot AI Released: 2026-01-01T06:06:03.000Z Updated: 2026-07-29 Architecture: moe Parameters: 1T Context length: 262,144 Min vLLM: 0.19.1 Tasks: multimodal, text Verified hardware: H200, GB200 NVL4, MI300X, MI325X, MI355X Summary: Open-source native multimodal agentic MoE model with vision-language understanding, tool calling, and thinking modes ## Variants - default: precision=int4, vram_minimum_gb=714 Packed INT4 via compressed-tensors (~595 GB on disk); fits 8×H200 - nvfp4: precision=nvfp4, vram_minimum_gb=600, model_id=nvidia/Kimi-K2.5-NVFP4 NVIDIA NVFP4 quantized weights for Blackwell GPUs (e.g. GB200) - mxfp4: precision=mxfp4, vram_minimum_gb=669, model_id=amd/Kimi-K2.5-MXFP4 AMD Quark OCP MXFP4 weights for MI355X (gfx950); ~557 GB on disk. ## Recommended base flags ```bash vllm serve 'moonshotai/Kimi-K2.5' \ --trust-remote-code ``` ## Guide ## Overview Kimi K2.5 is an open-source, native multimodal agentic model built through continual pretraining on approximately 15 trillion mixed visual and text tokens atop Kimi-K2-Base. It seamlessly integrates vision and language understanding with advanced agentic capabilities, instant and thinking modes, as well as conversational and agentic paradigms. ## Prerequisites - **vLLM version:** >= 0.15.0 (speculative decoding with Eagle3 requires >= 0.18.0) - **Hardware (BF16):** 8x H200 GPUs (verified), or equivalent aggregate VRAM (~640 GB) - **Hardware (NVFP4):** 4x Blackwell GPUs (e.g. GB200) - **AMD support:** 8x MI300X / MI325X / MI355X with ROCm 7.2.1 and Python 3.12 ### Install vLLM **Pip (NVIDIA):** ```bash uv venv source .venv/bin/activate uv pip install vllm --torch-backend auto ``` **Pip (AMD ROCm):** ```bash uv venv --python 3.12 source .venv/bin/activate uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm ``` **Docker (NVIDIA):** ```bash docker pull vllm/vllm-openai:latest ``` ### AMD MI300X/MI325X On 8x MI300X or MI325X (`gfx942`), use the standard W4A16 MoE path with AITER and INT4 QuickReduce. ```bash export VLLM_ROCM_USE_AITER=1 export VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4 vllm serve moonshotai/Kimi-K2.5 \ --host 0.0.0.0 \ --port 8000 \ --trust-remote-code \ --tensor-parallel-size 8 \ --tool-call-parser kimi_k2 \ --enable-auto-tool-choice \ --reasoning-parser kimi_k2 \ --mm-encoder-tp-mode data ``` ### AMD MI350X/MI355X On 8x MI350X or MI355X (`gfx950`), add `--moe-backend flydsl` to use the optimized FlyDSL W4A16 MoE kernel. Keep LoRA disabled for this path. ```bash export VLLM_ROCM_USE_AITER=1 export VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4 vllm serve moonshotai/Kimi-K2.5 \ --tensor-parallel-size 8 \ --trust-remote-code \ --mm-encoder-tp-mode data \ --moe-backend flydsl \ --compilation-config '{"pass_config": {"fuse_allreduce_rms": false}}' ``` Notes: - The FlyDSL INT4 MoE path does not support expert parallelism; do not add `--enable-expert-parallel`. - Keep `--compilation-config '{"pass_config": {"fuse_allreduce_rms": false}}'`; it is required for this FlyDSL path on MI350X / MI355X. - vLLM has tuned MI350X/MI355X FlyDSL configs for this Kimi shape at TP=8 and TP=4. - Keep vLLM's default block size unless you are tuning long-context throughput; `--block-size 64` is safe to try. ### AMD MI355X (MXFP4 checkpoint) The **MXFP4** variant serves the AMD Quark checkpoint [`amd/Kimi-K2.5-MXFP4`](https://huggingface.co/amd/Kimi-K2.5-MXFP4) — a different quantization from the packed-INT4 `moonshotai/Kimi-K2.5` FlyDSL path above. It runs the AITER MXFP4 MoE path and adds an fp8 KV cache. ```bash export VLLM_ROCM_USE_AITER=1 export VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4 vllm serve amd/Kimi-K2.5-MXFP4 \ --tensor-parallel-size 8 \ --trust-remote-code \ --mm-encoder-tp-mode data \ --kv-cache-dtype fp8 \ --gpu-memory-utilization 0.90 ``` Notes: - **fp8 KV cache** is the main MI355X win: it doubles resident KV capacity and halves KV read bandwidth. Measured +27% throughput at 8k/1k input with no accuracy change (GSM8K 0.9697, unchanged from bf16 KV). - **AITER RMSNorm** stays on. A local TP4 GSM8K check on the current nightly measured 0.9697. - **Old firmware only:** if `rocm-smi --showfw` reports a MEC firmware version below 177, add `export HSA_NO_SCRATCH_RECLAIM=1` — older firmware can't reclaim RCCL scratch and vLLM crashes without it. Current MI355X firmware doesn't need it. - For maximum throughput on fixed-length benchmark 8k/1k or 1k/1k workloads, see the benchmark reproduction section below. These are throughput-sweep tunings; leave vLLM's defaults for general serving. ### MXFP4 benchmark reproduction (InferenceX MI355X sweep) The command above is the default serving configuration. The [SemiAnalysis InferenceX](https://inferencex.semianalysis.com/) MI355X benchmark lane for this checkpoint is a separate **high-concurrency benchmark** configuration — not the recipe default. To reproduce that sweep, start from the default command above and apply these deltas: - Use `--tensor-parallel-size 4` (the sweep runs TP4; the default recipe keeps TP8 for KV-cache and multimodal-encoder headroom). - Add `--kv-cache-dtype fp8`. - Use the benchmark block size and scheduler knobs: `--block-size 16`, `--max-num-batched-tokens 16384`, `--max-num-seqs 512`, `--async-scheduling`, and `--no-enable-prefix-caching`. - Run on a current `vllm/vllm-openai-rocm:nightly` that contains the required AITER Kimi MXFP4 backend. - Export the AITER and INT4 quantized all-reduce env vars before launch: ```bash # Kernel selection and benchmark runtime knobs. export VLLM_ROCM_USE_AITER=1 export VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4 export VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS=1 export VLLM_ROCM_USE_SKINNY_GEMM=0 export VLLM_ROCM_USE_AITER_RMSNORM=0 export AITER_MXFP4_INTERMEDIATE=1 export AITER_BYPASS_TUNE_CONFIG=0 export AITER_MOE_SORT_BACKEND=auto export OMP_NUM_THREADS=1 ``` The full reproduction serve command is then: ```bash vllm serve amd/Kimi-K2.5-MXFP4 \ --port "${PORT:-8000}" \ --tensor-parallel-size 4 \ --gpu-memory-utilization 0.85 \ --max-model-len "$MAX_MODEL_LEN" \ --kv-cache-dtype fp8 \ --block-size 16 \ --max-num-batched-tokens 16384 \ --max-num-seqs 512 \ --async-scheduling \ --trust-remote-code \ --no-enable-prefix-caching \ --mm-encoder-tp-mode data ``` The benchmark script enforces `MAX_MODEL_LEN >= 9472` so the 1k/1k configuration matches the locally validated server. The env vars and scheduler knobs are benchmark-specific throughput settings, not default serving choices. #### Tuned AITER MXFP4 MoE (MI355X) For higher MoE throughput on MI355X, run the AITER MXFP4 (RadeonFlow) intermediate GEMM path with fused shared experts. This requires a `vllm/vllm-openai-rocm:nightly` built after vLLM [#48683](https://github.com/vllm-project/vllm/pull/48683) (AITER `v0.1.16.post5`, which includes the [ROCm/aiter#3832](https://github.com/ROCm/aiter/pull/3832) gfx950 MXFP4 MoE backend). Export the AITER MoE kernel-selection vars and add the batching/scheduling flags below. `AITER_MXFP4_INTERMEDIATE=1` is what selects the RadeonFlow MXFP4 intermediate path — without it (and `VLLM_ROCM_USE_AITER=1`) the MoE oracle falls back to a different kernel. Runs at TP4 or TP8; for **TP < 8** disable AITER RMSNorm. ```bash export VLLM_ROCM_USE_AITER=1 export VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4 export VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS=1 export VLLM_ROCM_USE_SKINNY_GEMM=0 export AITER_MXFP4_INTERMEDIATE=1 export AITER_BYPASS_TUNE_CONFIG=0 export AITER_MOE_SORT_BACKEND=auto export OMP_NUM_THREADS=1 # TP < 8 only: export VLLM_ROCM_USE_AITER_RMSNORM=0 vllm serve amd/Kimi-K2.5-MXFP4 \ --tensor-parallel-size 4 \ --trust-remote-code \ --gpu-memory-utilization 0.90 \ --kv-cache-dtype fp8 \ --block-size 16 \ --async-scheduling \ --no-enable-prefix-caching \ --mm-encoder-tp-mode data ``` These AITER MXFP4 MoE env vars are kernel-selection choices (same MXFP4 math, faster kernel), not precision reductions. For fixed-length throughput sweeping, additionally cap `--max-model-len` to the working sequence length (e.g. `9472` for an 8k/1k workload) and set `--max-num-batched-tokens 16384 --max-num-seqs 512`. Leave these off for general serving. ## Client Usage Once the vLLM server is running, consume it via the OpenAI-compatible API: ```python import time from openai import OpenAI client = OpenAI( api_key="EMPTY", base_url="http://localhost:8000/v1", timeout=3600 ) messages = [ { "role": "user", "content": [ { "type": "image_url", "image_url": { "url": "https://ofasys-multimodal-wlcb-3-toshanghai.oss-accelerate.aliyuncs.com/wpf272043/keepme/image/receipt.png" } }, { "type": "text", "text": "Read all the text in the image." } ] } ] start = time.time() response = client.chat.completions.create( model="moonshotai/Kimi-K2.5", messages=messages, max_tokens=2048 ) print(f"Response costs: {time.time() - start:.2f}s") print(f"Generated text: {response.choices[0].message.content}") ``` ## Troubleshooting - **OOM errors:** Lower `--gpu-memory-utilization` or adjust TP/EP to match your GPU count. - **Vision encoder performance:** Use `--mm-encoder-tp-mode data` to run the vision encoder in data-parallel mode. The encoder is small, so TP adds communication overhead with little gain. - **Unique multimodal inputs:** Pass `--mm-processor-cache-gb 0` to avoid caching overhead. For repeated inputs, `--mm-processor-cache-type shm` uses host shared memory for better performance at high TP settings. - **MoE kernel tuning:** Use the `benchmark_moe` script from vLLM to tune Triton kernels for your specific hardware. - **Async scheduling:** Enabled by default for better throughput. Disable if you encounter issues, and file a bug report to vLLM. ## References - [Kimi-K2.5 on Hugging Face](https://huggingface.co/moonshotai/Kimi-K2.5) - [NVIDIA Kimi-K2.5-NVFP4 on Hugging Face](https://huggingface.co/nvidia/Kimi-K2.5-NVFP4) - [Eagle3 MLA speculative decoding model](https://huggingface.co/lightseekorg/kimi-k2.5-eagle3-mla) - [vLLM multimodal inputs guide](https://docs.vllm.ai/en/latest/features/multimodal_inputs.html) - [vLLM Expert Parallelism docs](https://docs.vllm.ai/en/latest/serving/expert_parallel_deployment.html) - [vLLM NixlConnector usage guide](https://docs.vllm.ai/en/latest/features/nixl_connector_usage.html) --- # moonshotai/Kimi-K2.6 on vLLM Source: https://recipes.vllm.ai/moonshotai/Kimi-K2.6 Provider: Moonshot AI Released: 2026-04-14T04:23:36.000Z Updated: 2026-07-29 Architecture: moe Parameters: 1T Context length: 262,144 Min vLLM: 0.25.0 Tasks: multimodal, text Verified hardware: H200, B300, GB200 NVL4, MI300X, MI325X, MI355X Summary: Open-source native multimodal agentic MoE model with vision-language understanding, tool calling, and thinking modes ## Variants - default: precision=int4, vram_minimum_gb=714 Packed INT4 via compressed-tensors (~595 GB on disk); fits 8×H200 - nvfp4: precision=nvfp4, vram_minimum_gb=600, model_id=nvidia/Kimi-K2.6-NVFP4 NVIDIA NVFP4 quantized weights for Blackwell GPUs (e.g. GB200) - fp8: precision=fp8, vram_minimum_gb=1200, model_id=RedHatAI/Kimi-K2.6-FP8-BLOCK RedHatAI FP8 block quantization - redhat_nvfp4: precision=nvfp4, vram_minimum_gb=600, model_id=RedHatAI/Kimi-K2.6-NVFP4 RedHatAI NVFP4 quantization ## Recommended base flags ```bash vllm serve 'moonshotai/Kimi-K2.6' \ --trust-remote-code ``` ## Guide ## Overview Kimi K2.6 is an open-source, native multimodal agentic model built through continual pretraining on approximately 15 trillion mixed visual and text tokens atop Kimi-K2-Base. It seamlessly integrates vision and language understanding with advanced agentic capabilities, instant and thinking modes, as well as conversational and agentic paradigms. ## Prerequisites - **vLLM version:** >= 0.25.0 (required for the B300 EAGLE3 and native CPU KV offload paths below) - **Hardware (INT4):** 8× H200 GPUs (verified), or equivalent aggregate VRAM (~640 GB) - **Hardware (B300 / NVFP4):** 4× B300 (verified); use `vllm/vllm-openai:latest` - **AMD support:** 8x MI300X / MI325X / MI355X with ROCm 7.2.1 and Python 3.12 ### NVIDIA B300: NVFP4 with Eagle3 ```bash export VLLM_FLASHINFER_ALLREDUCE_BACKEND=trtllm vllm serve nvidia/Kimi-K2.6-NVFP4 \ --tensor-parallel-size 4 \ --trust-remote-code \ --kv-cache-dtype fp8 \ --block-size 64 \ --gpu-memory-utilization 0.90 \ --attention-backend TOKENSPEED_MLA \ --attention-config '{"use_prefill_query_quantization":true}' \ --compilation-config '{"cudagraph_mode":"FULL_AND_PIECEWISE","custom_ops":["all"]}' \ --max-cudagraph-capture-size 2048 \ --speculative-config '{"method":"eagle3","model":"lightseekorg/kimi-k2.6-eagle3-mla","num_speculative_tokens":4}' ``` ### Native CPU KV offload Select **Simple** in the command builder's **KV Offload** row to extend the prefix cache into host DRAM with `SimpleCPUOffloadConnector`. The shared option uses 220 GiB per rank by default. The verified B300 TP4 run used 1,199 GiB total (299.75 GiB per rank); the equivalent explicit command is: ```bash CPU_OFFLOAD_BYTES_PER_RANK=321854111744 vllm serve nvidia/Kimi-K2.6-NVFP4 \ --tensor-parallel-size 4 \ --trust-remote-code \ --kv-cache-dtype fp8 \ --block-size 64 \ --gpu-memory-utilization 0.90 \ --attention-backend TOKENSPEED_MLA \ --attention-config '{"use_prefill_query_quantization":true}' \ --disable-hybrid-kv-cache-manager \ --kv-transfer-config "{\"kv_connector\":\"SimpleCPUOffloadConnector\",\"kv_role\":\"kv_both\",\"kv_connector_extra_config\":{\"cpu_bytes_to_use_per_rank\":${CPU_OFFLOAD_BYTES_PER_RANK},\"lazy_offload\":false}}" ``` ### Decode context parallelism For higher concurrency on the B300 NVFP4 path, add `--decode-context-parallel-size 4` (match TP size). The command builder's **Advanced** row also exposes DCP for this recipe. ### AMD MI300X/MI325X On 8x MI300X or MI325X (`gfx942`), use the standard W4A16 MoE path with AITER and INT4 QuickReduce. ```bash export VLLM_ROCM_USE_AITER=1 export VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4 vllm serve moonshotai/Kimi-K2.6 \ --host 0.0.0.0 \ --port 8000 \ --trust-remote-code \ --tensor-parallel-size 8 \ --tool-call-parser kimi_k2 \ --enable-auto-tool-choice \ --reasoning-parser kimi_k2 \ --mm-encoder-tp-mode data ``` ### AMD MI350X/MI355X On 8x MI350X or MI355X (`gfx950`), add `--moe-backend flydsl` to use the optimized FlyDSL W4A16 MoE kernel. Keep LoRA disabled for this path. ```bash export VLLM_ROCM_USE_AITER=1 export VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4 vllm serve moonshotai/Kimi-K2.6 \ --tensor-parallel-size 8 \ --trust-remote-code \ --mm-encoder-tp-mode data \ --moe-backend flydsl \ --compilation-config '{"pass_config": {"fuse_allreduce_rms": false}}' ``` Notes: - The FlyDSL INT4 MoE path does not support expert parallelism; do not add `--enable-expert-parallel`. - Keep `--compilation-config '{"pass_config": {"fuse_allreduce_rms": false}}'`; it is required for this FlyDSL path on MI350X / MI355X. - vLLM has tuned MI350X/MI355X FlyDSL configs for this Kimi shape at TP=8 and TP=4. - Keep vLLM's default block size unless you are tuning long-context throughput; `--block-size 64` is safe to try. ## Client Usage Once the vLLM server is running, consume it via the OpenAI-compatible API: ```python import time from openai import OpenAI client = OpenAI( api_key="EMPTY", base_url="http://localhost:8000/v1", timeout=3600 ) messages = [ { "role": "user", "content": [ { "type": "image_url", "image_url": { "url": "https://ofasys-multimodal-wlcb-3-toshanghai.oss-accelerate.aliyuncs.com/wpf272043/keepme/image/receipt.png" } }, { "type": "text", "text": "Read all the text in the image." } ] } ] start = time.time() response = client.chat.completions.create( model="moonshotai/Kimi-K2.6", messages=messages, max_tokens=2048 ) print(f"Response costs: {time.time() - start:.2f}s") print(f"Generated text: {response.choices[0].message.content}") ``` ## Troubleshooting - **OOM errors:** Lower `--gpu-memory-utilization` or adjust TP/EP to match your GPU count. - **Vision encoder performance:** Use `--mm-encoder-tp-mode data` to run the vision encoder in data-parallel mode. The encoder is small, so TP adds communication overhead with little gain. - **Unique multimodal inputs:** Pass `--mm-processor-cache-gb 0` to avoid caching overhead. For repeated inputs, `--mm-processor-cache-type shm` uses host shared memory for better performance at high TP settings. - **MoE kernel tuning:** Use the `benchmark_moe` script from vLLM to tune Triton kernels for your specific hardware. - **Async scheduling:** Enabled by default for better throughput. Disable if you encounter issues, and file a bug report to vLLM. ## References - [Kimi-K2.6 on Hugging Face](https://huggingface.co/moonshotai/Kimi-K2.6) - [NVIDIA Kimi-K2.6-NVFP4 on Hugging Face](https://huggingface.co/nvidia/Kimi-K2.6-NVFP4) - [vLLM multimodal inputs guide](https://docs.vllm.ai/en/latest/features/multimodal_inputs.html) - [vLLM Expert Parallelism docs](https://docs.vllm.ai/en/latest/serving/expert_parallel_deployment.html) - [vLLM NixlConnector usage guide](https://docs.vllm.ai/en/latest/features/nixl_connector_usage.html) --- # moonshotai/Kimi-K2.7-Code on vLLM Source: https://recipes.vllm.ai/moonshotai/Kimi-K2.7-Code Provider: Moonshot AI Released: 2026-06-11T07:51:47.000Z Updated: 2026-07-29 Architecture: moe Parameters: 1T Context length: 262,144 Min vLLM: 0.19.1 Tasks: multimodal, text Verified hardware: H200, MI300X, MI325X, MI355X Summary: Coding-focused agentic MoE built on Kimi-K2.6, tuned for long-horizon software-engineering tasks with thinking-only reasoning, tool calling, and vision-language input ## Variants - default: precision=int4, vram_minimum_gb=714 Native INT4 via compressed-tensors (~595 GB on disk); fits 8×H200 ## Recommended base flags ```bash vllm serve 'moonshotai/Kimi-K2.7-Code' \ --trust-remote-code ``` ## Guide ## Overview [Kimi-K2.7-Code](https://huggingface.co/moonshotai/Kimi-K2.7-Code) is a coding-focused agentic model built on top of Kimi-K2.6. It substantially improves real-world long-horizon coding performance — strengthening end-to-end task completion across complex software-engineering workflows while improving token efficiency, reducing thinking-token usage by ~30% versus Kimi-K2.6. It shares the same architecture as Kimi-K2.5 / Kimi-K2.6 (1T-parameter MoE, 32B activated, DeepSeek-V3 backbone with MLA attention and a MoonViT vision encoder), so the K2.6 deployment method can be reused directly. Kimi-K2.7-Code runs in **thinking mode only** with `preserve_thinking` forced on; keep the reasoning parser enabled. ## Prerequisites - **vLLM version:** >= 0.19.1 (manually verified). The model is also available in the vLLM nightly wheel, but nightly builds are experimental. - **Hardware (INT4):** 8x H200 GPUs (verified), or equivalent aggregate VRAM (~640 GB) ```bash uv venv source .venv/bin/activate uv pip install -U vllm --torch-backend=auto ``` ## Launch command Serve on a single H200 node with TP8 (vision encoder in data-parallel mode): ```bash vllm serve moonshotai/Kimi-K2.7-Code \ --tensor-parallel-size 8 \ --mm-encoder-tp-mode data \ --trust-remote-code \ --tool-call-parser kimi_k2 \ --enable-auto-tool-choice \ --reasoning-parser kimi_k2 ``` **Key notes** - `--tool-call-parser kimi_k2` — required for enabling tool calling. - `--reasoning-parser kimi_k2` — Kimi-K2.7-Code supports thinking mode only; pass this for correct reasoning processing. - `--mm-encoder-tp-mode data` — runs the small MoonViT encoder data-parallel to avoid TP communication overhead. ### AMD MI300X/MI325X On 8x MI300X or MI325X (`gfx942`), use the standard W4A16 MoE path with AITER and INT4 QuickReduce. ```bash export VLLM_ROCM_USE_AITER=1 export VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4 vllm serve moonshotai/Kimi-K2.7-Code \ --host 0.0.0.0 \ --port 8000 \ --trust-remote-code \ --tensor-parallel-size 8 \ --tool-call-parser kimi_k2 \ --enable-auto-tool-choice \ --reasoning-parser kimi_k2 \ --mm-encoder-tp-mode data ``` ### AMD MI350X/MI355X On 8x MI350X or MI355X (`gfx950`), add `--moe-backend flydsl` to use the optimized FlyDSL W4A16 MoE kernel. Keep LoRA disabled for this path. ```bash export VLLM_ROCM_USE_AITER=1 export VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4 vllm serve moonshotai/Kimi-K2.7-Code \ --tensor-parallel-size 8 \ --trust-remote-code \ --mm-encoder-tp-mode data \ --moe-backend flydsl \ --tool-call-parser kimi_k2 \ --enable-auto-tool-choice \ --reasoning-parser kimi_k2 \ --compilation-config '{"pass_config": {"fuse_allreduce_rms": false}}' ``` Notes: - The FlyDSL INT4 MoE path does not support expert parallelism; do not add `--enable-expert-parallel`. - Keep `--compilation-config '{"pass_config": {"fuse_allreduce_rms": false}}'`; it is required for this FlyDSL path on MI350X / MI355X. - vLLM has tuned MI350X/MI355X FlyDSL configs for this Kimi shape at TP=8 and TP=4. - Keep vLLM's default block size unless you are tuning long-context throughput; `--block-size 64` is safe to try. ## Client Usage Once the vLLM server is running, consume it via the OpenAI-compatible API. The recommended sampling for thinking mode is `temperature=1.0` and `top_p=0.95`. ```python import time from openai import OpenAI client = OpenAI( api_key="EMPTY", base_url="http://localhost:8000/v1", timeout=3600, ) messages = [ { "role": "user", "content": [ { "type": "image_url", "image_url": { "url": "https://huggingface.co/moonshotai/Kimi-K2.7-Code/resolve/main/figures/kimi-logo.png" }, }, {"type": "text", "text": "Describe this image in detail."}, ], } ] start = time.time() response = client.chat.completions.create( model="moonshotai/Kimi-K2.7-Code", messages=messages, max_tokens=4096, temperature=1.0, top_p=0.95, ) print(f"Response costs: {time.time() - start:.2f}s") print(f"Reasoning: {response.choices[0].message.reasoning}") print(f"Response: {response.choices[0].message.content}") ``` ## Troubleshooting - **OOM errors:** Lower `--gpu-memory-utilization` or adjust TP/EP to match your GPU count. - **Vision encoder performance:** Use `--mm-encoder-tp-mode data` to run the vision encoder in data-parallel mode. The encoder is small, so TP adds communication overhead with little gain. - **Unique multimodal inputs:** Pass `--mm-processor-cache-gb 0` to avoid caching overhead. For repeated inputs, `--mm-processor-cache-type shm` uses host shared memory for better performance at high TP settings. - **Text-only workloads:** Pass `--language-model-only` to skip loading MoonViT and free VRAM for KV cache. ## References - [Kimi-K2.7-Code on Hugging Face](https://huggingface.co/moonshotai/Kimi-K2.7-Code) - [Kimi-K2.7-Code deployment guide](https://huggingface.co/moonshotai/Kimi-K2.7-Code/blob/main/docs/deploy_guidance.md) - [Kimi-K2.6 recipe](https://recipes.vllm.ai/moonshotai/Kimi-K2.6) - [vLLM multimodal inputs guide](https://docs.vllm.ai/en/latest/features/multimodal_inputs.html) --- # mindlab-research/Macaron-V1-Coding-Venti on vLLM Source: https://recipes.vllm.ai/mindlab-research/Macaron-V1-Coding-Venti Provider: MindLab Research Released: 2026-07-20T15:49:37.000Z Updated: 2026-07-23 Architecture: moe Parameters: 743B Context length: 1,048,576 Min vLLM: 0.25.0 Tasks: text Verified hardware: B300 Summary: Macaron-V1-Coding-Venti — MindLab's coding-specialist checkpoint: the Macaron-V1-Venti L2 Coding LoRA merged into the GLM-5.2 BF16 base. Same MoE architecture and launch as GLM-5.2 (~743B total, 39B active), no runtime adapter. ## Variants - default: precision=bf16, vram_minimum_gb=1786 Full-precision BF16 merged checkpoint — requires multi-node (≈1786 GB weights + KV; a single 8xH200/8xB200 node does not fit). - fp8: precision=fp8, vram_minimum_gb=893 Same BF16 checkpoint, quantized to FP8 at load time via --quantization fp8 — not a native FP8 weight set. Halves weight memory so it fits a single node; on a B300 node it leaves ~2.7× the GPU KV-cache space of BF16 (≈3.07M vs 1.12M). Slower startup; on GSM8K (5-shot, exact-match) it matches native BF16 within noise (0.9424 vs 0.9431, ±0.0064). ## Recommended base flags ```bash vllm serve 'mindlab-research/Macaron-V1-Coding-Venti' \ --kv-cache-dtype \ fp8_e4m3 ``` ## Guide ## Overview **Macaron-V1-Coding-Venti** is MindLab Research's coding-specialist checkpoint in the Macaron-V1 family. It is produced by **merging the Macaron-V1-Venti L2 Coding LoRA specialist into the GLM-5.2 BF16 base**, yielding a single merged checkpoint — no PEFT adapter or Mixture-of-LoRA (MoL) router is needed at runtime. It targets code understanding, repository-level software engineering, terminal use, and coding-agent workflows, and reports strong coding/terminal slices (SWE Verified 85.6, TerminalBench 2.1 87.6, DeepSWE 58.4) on the Macaron evaluation suite. Because it is a weight-level merge of a LoRA onto GLM-5.2, the runtime architecture is **identical to GLM-5.2** — same `GlmMoeDsaForCausalLM` MoE (~743B total / 39B active), same 1M native context, same MTP indexer. **Launch it with the same serving stack as [GLM-5.2 recipe](/zai-org/GLM-5.2).** This model ships **BF16 only** (no native FP8 / NVFP4 checkpoint). At full BF16 precision (≈1786 GB) it needs multi-node; the **FP8 (on-the-fly)** variant loads the same weights as FP8 (`--quantization fp8`, ≈893 GB) to fit a single 8×H200 / 8×B200 node — or to free VRAM on larger trays. ## Prerequisites - **vLLM 0.25.0** (stable) — the `GlmMoeDsaForCausalLM` handler is the same one GLM-5.2 uses. - **GPU:** at full **BF16** precision (≈1786 GB) a single 8×H200 / 8×B200 node does not fit — run **multi-node** (e.g. 2× 8-GPU nodes, tensor-parallel 16), or pick the **FP8 (on-the-fly)** variant (≈893 GB, `--quantization fp8`; `fp8_per_channel` on AMD) to run single-node. The native FP8/NVFP4 GLM-5.2 checkpoints are a calibrated-quant alternative. ## Installation ### Docker The BF16 variant needs two nodes (the FP8 (on-the-fly) variant fits one — see Launching below). Run one container per node (each `--gpus all`, 8 local GPUs); both pass `--tensor-parallel-size 16 --nnodes 2`, varying `--node-rank`. This is the head node — node 1 repeats with `--node-rank 1 --headless`. ```bash docker run --gpus all \ -p 8000:8000 \ --ipc=host \ --network host \ -v ~/.cache/huggingface:/root/.cache/huggingface \ vllm/vllm-openai:v0.28.0 mindlab-research/Macaron-V1-Coding-Venti \ --tensor-parallel-size 16 \ --nnodes 2 \ --node-rank 0 \ --master-addr $HEAD_IP \ --kv-cache-dtype fp8 \ --tool-call-parser glm47 \ --reasoning-parser glm45 \ --enable-auto-tool-choice \ --served-model-name macaron-v1-coding-venti ``` ### From source ```bash uv venv source .venv/bin/activate uv pip install "vllm==0.25.0" --torch-backend=auto uv pip install "transformers>=5.9.0" ``` ## Launching the server Pick the **Variant** that matches your hardware: **BF16** for full precision on multi-node, or **FP8 (on-the-fly)** to fit a single node (or free VRAM on a larger tray). The builder renders the exact flags per your Variant / Strategy / Nodes picks. ### BF16 — 2× 8xH200 (multi-node TP) Full-precision BF16 (≈1786 GB) needs multi-node. The command below is a 2-node tensor-parallel example (16 GPUs). Run the head command on node 0 and the worker on node 1, pointing `--master-addr` at the head's IP. Pick the **Multi-node TP** strategy in the builder to render both tabs. ```bash # Node 0 (head) vllm serve mindlab-research/Macaron-V1-Coding-Venti \ --kv-cache-dtype fp8 \ --tensor-parallel-size 16 \ --tool-call-parser glm47 \ --reasoning-parser glm45 \ --enable-auto-tool-choice \ --served-model-name macaron-v1-coding-venti # Node 1 vllm serve mindlab-research/Macaron-V1-Coding-Venti \ --kv-cache-dtype fp8 \ --tensor-parallel-size 16 \ --tool-call-parser glm47 \ --reasoning-parser glm45 \ --enable-auto-tool-choice \ --served-model-name macaron-v1-coding-venti ``` > The builder emits the exact multi-node multiprocessing args (`--nnodes`, > `--node-rank`, `--master-addr` for TP) when you select **Multi-node TP**. ### FP8 (on-the-fly) on a single node Pick the **FP8 (on-the-fly)** variant to load the same BF16 checkpoint as FP8 via `--quantization fp8` (runtime quantization — not a native FP8 weight set). At ≈893 GB it fits a single 8×H200 / 8×B200 node, and on larger trays (e.g. B300) it frees VRAM for longer context or more concurrent sequences. Startup is slower while the quantization pass runs at load; the accuracy / KV-cache trade-off is small (see the B300 numbers below). ```bash vllm serve mindlab-research/Macaron-V1-Coding-Venti \ --quantization fp8 \ --tensor-parallel-size 8 \ --kv-cache-dtype fp8 \ --tool-call-parser glm47 \ --reasoning-parser glm45 \ --enable-auto-tool-choice \ --served-model-name macaron-v1-coding-venti ``` - **`--max-model-len`** — raise toward the 1M window when KV-cache headroom allows; lower on OOM. - **`--max-num-seqs 32`** — the main knob for fitting long context under concurrency. - **`--gpu-memory-utilization 0.80`** — leaves ROCm runtime headroom for MTP graph capture. On a **B300** node, on-the-fly FP8 trades a negligible accuracy drop for a large KV-cache headroom gain: | Variant | GSM8K 5-shot (exact-match) | B300 GPU KV-cache size | |---|---:|---:| | BF16 | 0.9431 (±0.0064) | ≈1.12M | | FP8 (on-the-fly) | 0.9424 (±0.0064) | ≈3.07M | The GSM8K scores sit within stderr of each other, while FP8 leaves ~2.7× the KV-cache space — so on B300 (where BF16 already fits) FP8 buys far more concurrency / context at almost no accuracy cost. ### Speculative decoding (MTP) The merged weights inherit GLM-5.2's built-in Multi-Token Prediction head, so MTP works exactly as on GLM-5.2 — on both the BF16 and FP8 (on-the-fly) variants. Toggle **spec_decoding → MTP** in the builder: ```bash vllm serve mindlab-research/Macaron-V1-Coding-Venti \ --kv-cache-dtype fp8 \ --tensor-parallel-size 16 \ --speculative-config '{"method":"mtp","num_speculative_tokens":5}' \ --tool-call-parser glm47 \ --reasoning-parser glm45 \ --enable-auto-tool-choice \ --served-model-name macaron-v1-coding-venti ``` ### AMD MI300X / MI355X Use the AITER backends under ROCm, same as GLM-5.2. On AMD the FP8 (on-the-fly) variant uses `--quantization fp8_per_channel`. FP8 (on-the-fly): ```bash VLLM_ROCM_USE_AITER=1 \ VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS=1 \ vllm serve mindlab-research/Macaron-V1-Coding-Venti \ --kv-cache-dtype fp8_e4m3 \ --tensor-parallel-size 8 \ --quantization fp8_per_channel \ --moe-backend aiter \ --tool-call-parser glm47 \ --enable-auto-tool-choice \ --reasoning-parser glm45 ``` BF16: ```bash VLLM_ROCM_USE_AITER=1 \ VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS=1 \ vllm serve mindlab-research/Macaron-V1-Coding-Venti \ --kv-cache-dtype fp8_e4m3 \ --tensor-parallel-size 8 \ --max-model-len 262144 \ --linear-backend aiter \ --moe-backend aiter \ --tool-call-parser glm47 \ --enable-auto-tool-choice \ --reasoning-parser glm45 ``` On 8×MI355X, GSM8K 5-shot (strict-match) is 0.9469 for both BF16 and FP8 (on-the-fly). ## Reasoning modes Thinking is **on by default** and reuses the GLM/DeepSeek-V4 `reasoning_effort` mechanism, identical to GLM-5.2. The chat template resolves effort to **`max` unless `reasoning_effort` is explicitly `"high"`**. | Mode | How to request | Behavior | |------|----------------|----------| | **Think Max** (default) | omit `reasoning_effort`, or set `"max"` | Deepest reasoning — hard math, multi-step planning, agentic tasks. | | **Think High** | `"reasoning_effort": "high"` | Balanced depth and latency. | | **Non-think** | `chat_template_kwargs.enable_thinking: false` | Fast, no chain-of-thought. | ## Client usage ```python from openai import OpenAI client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1") msgs = [{"role": "user", "content": "Refactor this function to handle empty input."}] # Think Max (default) — just omit reasoning_effort client.chat.completions.create(model="macaron-v1-coding-venti", messages=msgs, max_tokens=4096) # Think High client.chat.completions.create( model="macaron-v1-coding-venti", messages=msgs, max_tokens=4096, extra_body={"chat_template_kwargs": {"reasoning_effort": "high"}}, ) # Non-think client.chat.completions.create( model="macaron-v1-coding-venti", messages=msgs, max_tokens=4096, extra_body={"chat_template_kwargs": {"enable_thinking": False}}, ) ``` ## Benchmarking Add `--no-enable-prefix-caching` to the server command for a clean measurement. ```bash vllm bench serve \ --model mindlab-research/Macaron-V1-Coding-Venti \ --dataset-name random \ --random-input 8000 \ --random-output 1024 \ --request-rate 10 \ --num-prompts 32 \ --ignore-eos ``` > **Note:** pure throughput benchmarks tend to under-report real speed, because MTP's > acceptance rate is usually low in synthetic runs. ## References - [Model card](https://huggingface.co/mindlab-research/Macaron-V1-Coding-Venti) - [Base model: GLM-5.2 recipe](/zai-org/GLM-5.2) - [Introducing Macaron-V1 (blog)](https://macaron.im/mindlab/research/introducing-macaron-v1) - [Macaron Artifacts (GitHub)](https://github.com/MindLab-Research/macaron-artifacts) --- # poolside/Laguna-S-2.1 on vLLM Source: https://recipes.vllm.ai/poolside/Laguna-S-2.1 Provider: Poolside Released: 2026-07-13T13:58:35.000Z Updated: 2026-07-22 Architecture: moe Parameters: 118B Context length: 262,144 Min vLLM: 0.25.0 Tasks: text Verified hardware: H200, MI325X Summary: Poolside's 118B total / 8B activated MoE coding model with mixed sliding-window + global attention, native interleaved reasoning, and 256K context — the larger sibling of Laguna XS-2.1, tuned for agentic coding. ## Variants - default: precision=bf16, vram_minimum_gb=282 BF16 weights (~235GB) — needs a multi-GPU node (≥2× H200/B200, or a full HGX node) - fp8: precision=fp8, vram_minimum_gb=145, model_id=poolside/Laguna-S-2.1-FP8 Block-FP8 weights + FP8 KV cache (~121GB) — a hair over one 141GB H200, so TP2 (or a single 268GB B300) - nvfp4: precision=nvfp4, vram_minimum_gb=86, model_id=poolside/Laguna-S-2.1-NVFP4 NVFP4 weights + FP8 KV cache (~72GB, Blackwell only) — fits a single B200/B300 - int4: precision=int4, vram_minimum_gb=86, model_id=poolside/Laguna-S-2.1-INT4 INT4 (W4A16) weights + FP8 KV cache (~72GB) — fits a single 96GB+ GPU (H200/B200) ## Recommended base flags ```bash vllm serve 'poolside/Laguna-S-2.1' \ --trust-remote-code ``` ## Guide ## Overview [Laguna S-2.1](https://huggingface.co/poolside/Laguna-S-2.1) is Poolside's 118B-total / 8B-activated Mixture-of-Experts model — the larger sibling of [Laguna XS-2.1](https://huggingface.co/poolside/Laguna-XS-2.1), built for agentic coding and long-horizon work. It shares the `laguna` architecture: mixed sliding-window + global attention (3:1 across 48 layers) with softplus per-head gating and an FP8 KV cache, plus native interleaved reasoning and tool calling. ### Key features - **Mixed SWA + global attention**: 36 sliding-window layers (window=512) interleaved with 12 global-attention layers (3:1), each with per-layer rotary scaling. - **256 experts + 1 shared expert** with top-10 routing. - **Grouped-query attention**: 8 KV heads, head dim 128, per-head softplus output gating. - **Native FP8 KV cache** to reduce memory per token. - **Interleaved reasoning**: thinking blocks emitted between tool calls; toggled per-request via `enable_thinking`. - **Tool calling**: Poolside's XML-style tool-call protocol, parsed via `poolside_v1`. Same architecture as XS-2.1, so vLLM needs no S-specific code — support lands with the shared `laguna` implementation. ## Prerequisites ### pip ```bash uv pip install -U 'vllm>=0.25.0' ``` ### Docker ```bash docker pull vllm/vllm-openai:latest ``` ## Variants and VRAM Laguna S-2.1 is large: the BF16 weights are ~235GB, so unlike XS-2.1 the default **does not fit a single GPU** — plan for a multi-GPU node or pick a quantized checkpoint. Select one with the **Variant** control above: | Variant | Checkpoint | Weights | ~Size | Fits on | |---------|-----------|---------|-------|---------| | BF16 | [Laguna-S-2.1](https://huggingface.co/poolside/Laguna-S-2.1) | BF16 | 235 GB | ≥2× H200/B200, or a full HGX node | | FP8 | [Laguna-S-2.1-FP8](https://huggingface.co/poolside/Laguna-S-2.1-FP8) | Block-FP8 | 121 GB | TP2 on H200, or one B300; set `VLLM_BLOCKSCALE_FP8_GEMM_FLASHINFER=0` | | NVFP4 | [Laguna-S-2.1-NVFP4](https://huggingface.co/poolside/Laguna-S-2.1-NVFP4) | NVFP4 | 72 GB | single B200/B300 (Blackwell only) | | INT4 | [Laguna-S-2.1-INT4](https://huggingface.co/poolside/Laguna-S-2.1-INT4) | INT4 (W4A16) | 72 GB | single H200/B200 | Quantization is detected automatically from each checkpoint's `quantization_config`, so the launch command is identical — just swap the model ID. All three quantized checkpoints ship an **FP8-quantized KV cache**. Precision-matched DFlash speculators ship for each quantization — [DFlash-FP8](https://huggingface.co/poolside/Laguna-S-2.1-DFlash-FP8), [DFlash-NVFP4](https://huggingface.co/poolside/Laguna-S-2.1-DFlash-NVFP4), [DFlash-INT4](https://huggingface.co/poolside/Laguna-S-2.1-DFlash-INT4) — to pair speculative decoding with a quantized base at matching precision. > **Context length.** The BF16 checkpoint supports up to 1,048,576 tokens; > the quantized checkpoints are configured for 256K. The commands below use > `--max-model-len 262144`; raise or lower it to trade context for KV memory. ## Launch command ### Multi-GPU node (H200/B200, BF16) ```bash vllm serve poolside/Laguna-S-2.1 \ --trust-remote-code \ --tensor-parallel-size 8 \ --max-model-len 262144 \ --enable-auto-tool-choice \ --tool-call-parser poolside_v1 \ --reasoning-parser poolside_v1 ``` ### AMD ROCm — 8× MI325X For FP8, the MI325X override disables AITER MoE because its blockscale kernel currently corrupts logits; other AITER operators remain enabled, while vLLM uses Triton MoE. ### Single GPU (INT4 on H200, or NVFP4 on B200) ```bash vllm serve poolside/Laguna-S-2.1-INT4 \ --trust-remote-code \ --max-model-len 262144 \ --enable-auto-tool-choice \ --tool-call-parser poolside_v1 \ --reasoning-parser poolside_v1 ``` ## Controlling reasoning Reasoning is **off by default** in the chat template. Enable it per-request: ```python from openai import OpenAI client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY") resp = client.chat.completions.create( model="poolside/Laguna-S-2.1", messages=[{"role": "user", "content": "Write a Python retry wrapper with exponential backoff."}], extra_body={"chat_template_kwargs": {"enable_thinking": True}}, temperature=0.7, top_p=1.0, extra_query={"top_k": 20}, ) print(resp.choices[0].message.reasoning) print(resp.choices[0].message.content) ``` Or default-on with `--default-chat-template-kwargs '{"enable_thinking": true}'`. The model's `generation_config.json` sets `top_k 20` (eval-certified truncation) so clients that send no sampling params still get sane output. Do **not** add `min_p` when speculative decoding is on — vLLM rejects `min_p`/`logit_bias` with speculation; use `top_k`/`top_p`. ## Speculative decoding (DFlash) Enable the **Spec Decoding** toggle above to attach Poolside's [DFlash draft model](https://huggingface.co/poolside/Laguna-S-2.1-DFlash) — a parallel drafter that proposes up to 15 draft tokens per step. Measured mean accepted length is ~3.1 tokens/step on natural text (higher on code), matching Poolside's production deployment. Requires: - DFlash drafter support for Laguna, merged in [PR #46853](https://github.com/vllm-project/vllm/pull/46853) (2026-07-03) — use a vLLM build that includes it (validated on 0.25.1). - `--moe-backend triton` — the default DeepGEMM MoE backend is incompatible with the DFlash draft path. Enabling the toggle adds this flag. ```bash vllm serve poolside/Laguna-S-2.1 \ --trust-remote-code \ --tensor-parallel-size 8 \ --max-model-len 16384 \ --enable-auto-tool-choice \ --tool-call-parser poolside_v1 \ --reasoning-parser poolside_v1 \ --speculative-config '{"model":"poolside/Laguna-S-2.1-DFlash","num_speculative_tokens":15,"method":"dflash"}' \ --moe-backend triton ``` ### DGX Spark (GB10) The NVFP4 checkpoint is validated on a single DGX Spark (GB10, `sm_121a`) with vLLM 0.25.1 + a FlashInfer nightly and DFlash — see the [Laguna-S-2.1-NVFP4 model card](https://huggingface.co/poolside/Laguna-S-2.1-NVFP4) for the full one-box recipe. ## References - [Model card](https://huggingface.co/poolside/Laguna-S-2.1) - [DFlash draft model](https://huggingface.co/poolside/Laguna-S-2.1-DFlash) - [vLLM DFlash spec-decoding PR #46853](https://github.com/vllm-project/vllm/pull/46853) - [Laguna XS-2.1 recipe](https://huggingface.co/poolside/Laguna-XS-2.1) --- # PaddlePaddle/PaddleOCR-VL-1.6 on vLLM Source: https://recipes.vllm.ai/PaddlePaddle/PaddleOCR-VL-1.6 Provider: PaddlePaddle Released: 2026-05-27T06:27:27.000Z Updated: 2026-07-08 Architecture: dense Parameters: 0.9B Context length: 131,072 Min vLLM: 0.11.1 Tasks: multimodal Summary: PaddleOCR-VL-1.6 (0.9B) — region-aware data optimization + progressive post-training; new SOTA 96.33% on OmniDocBench v1.6, drop-in replacement for 1.5 ## Variants - default: precision=bf16, vram_minimum_gb=2 BF16 weights — small footprint, runs on most GPUs ## Recommended base flags ```bash vllm serve 'PaddlePaddle/PaddleOCR-VL-1.6' \ --trust-remote-code \ --max-num-batched-tokens \ 16384 \ --no-enable-prefix-caching \ --mm-processor-cache-gb \ 0 ``` ## Guide ## Overview [PaddleOCR-VL-1.6](https://huggingface.co/PaddlePaddle/PaddleOCR-VL-1.6) upgrades PaddleOCR-VL-1.5 with a region-aware data optimization framework (targeted enhancement of weak regions found in the previous model) and a progressive post-training recipe based on curated data selection and reinforcement learning: - **96.33% on OmniDocBench v1.6** (new SOTA), plus new records on OmniDocBench v1.5 and Real5-OmniDocBench (scanning, warping, screen-photography, illumination, skew) - Major gains in table, Chinese ancient document, and Chinese rare character recognition - Improved seal/stamp recognition, text spotting, and chart recognition The architecture is **fully compatible with PaddleOCR-VL-1.5** (same 0.9B NaViT-style dynamic-resolution vision encoder + ERNIE-4.5-0.3B LM), so migration is plug-and-play: the vLLM launch command and feature flags are unchanged. ## Prerequisites - Hardware: 1x GPU (small VRAM footprint) - vLLM >= 0.11.1 ### Install vLLM ```bash uv venv source .venv/bin/activate uv pip install -U vllm ``` ## Launch command ```bash vllm serve PaddlePaddle/PaddleOCR-VL-1.6 \ --trust-remote-code \ --max-num-batched-tokens 16384 \ --no-enable-prefix-caching \ --mm-processor-cache-gb 0 ``` Tip: OCR workloads don't benefit much from prefix caching or image reuse, so disable those to avoid hashing/caching overhead. ## Client Usage Task-specific prompts (same task set as 1.5, including `spotting` and `seal`): ```python from openai import OpenAI client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1", timeout=3600) TASKS = { "ocr": "OCR:", "table": "Table Recognition:", "formula": "Formula Recognition:", "chart": "Chart Recognition:", "spotting": "Spotting:", "seal": "Seal Recognition:", } response = client.chat.completions.create( model="PaddlePaddle/PaddleOCR-VL-1.6", messages=[{ "role": "user", "content": [ {"type": "image_url", "image_url": {"url": "https://.../receipt.png"}}, {"type": "text", "text": TASKS["spotting"]}, ], }], temperature=0.0, ) print(response.choices[0].message.content) ``` ## Offline Inference with PP-DocLayoutV3 Use separate venvs for `vllm` and `paddlepaddle` to avoid conflicts. The 1.6 pipeline requires `paddleocr[doc-parser] >= 3.6.0` and `pipeline_version="v1.6"`, which automatically uses [PP-DocLayoutV3](https://huggingface.co/PaddlePaddle/PP-DocLayoutV3) for layout detection (PP-DocLayoutV2 belongs to the v1 pipeline). Pass `vl_rec_api_model_name="PaddlePaddle/PaddleOCR-VL-1.6"` so the client matches the model name the vLLM server exposes — otherwise it defaults to `PaddleOCR-VL-1.6-0.9B` and fails with "The model PaddleOCR-VL-1.6-0.9B does not exist." ```bash uv pip install paddlepaddle-gpu==3.2.1 --extra-index-url https://www.paddlepaddle.org.cn/packages/stable/cu126/ uv pip install -U "paddleocr[doc-parser]>=3.6.0" uv pip install safetensors ``` ```python from paddleocr import PaddleOCRVL pipeline = PaddleOCRVL( pipeline_version="v1.6", vl_rec_backend="vllm-server", vl_rec_server_url="http://localhost:8000/v1", vl_rec_api_model_name="PaddlePaddle/PaddleOCR-VL-1.6", ) output = pipeline.predict("https://paddle-model-ecology.bj.bcebos.com/paddlex/imgs/demo_image/paddleocr_vl_demo.png") for i, res in enumerate(output): res.save_to_json(save_path=f"output_{i}.json") res.save_to_markdown(save_path=f"output_{i}.md") ``` ## References - [PaddleOCR-VL-1.6 on Hugging Face](https://huggingface.co/PaddlePaddle/PaddleOCR-VL-1.6) - [Technical report (arXiv)](https://arxiv.org/pdf/2606.03264) - [PaddleOCR GitHub](https://github.com/PaddlePaddle/PaddleOCR) - [vLLM support PR](https://github.com/vllm-project/vllm/pull/27758) --- # poolside/Laguna-XS-2.1 on vLLM Source: https://recipes.vllm.ai/poolside/Laguna-XS-2.1 Provider: Poolside Released: 2026-06-20T14:14:26.000Z Updated: 2026-07-02 Architecture: moe Parameters: 33B Context length: 262,144 Min vLLM: 0.21.0 Tasks: text Verified hardware: H200 Summary: Poolside's 33B total / 3B activated MoE coding model with mixed sliding-window + global attention, native interleaved reasoning, and 256K context — designed for agentic coding. ## Variants - default: precision=bf16, vram_minimum_gb=80 BF16 weights — fits on a single 80GB+ GPU (H100/H200/B200) - fp8: precision=fp8, vram_minimum_gb=40, model_id=poolside/Laguna-XS-2.1-FP8 Block-FP8 weights + FP8 KV cache — halves VRAM, fits on a single 40GB+ GPU - nvfp4: precision=nvfp4, vram_minimum_gb=26, model_id=poolside/Laguna-XS-2.1-NVFP4 NVFP4 weights + FP8 KV cache (Blackwell only) — the most compact variant at ~26GB - int4: precision=int4, vram_minimum_gb=29, model_id=poolside/Laguna-XS-2.1-INT4 INT4 (W4A16) weights + FP8 KV cache — fits on a single 32GB+ GPU ## Recommended base flags ```bash vllm serve 'poolside/Laguna-XS-2.1' \ --trust-remote-code ``` ## Guide ## Overview [Laguna XS-2.1](https://huggingface.co/poolside/Laguna-XS-2.1) is Poolside's 33B-total / 3B-activated Mixture-of-Experts model purpose-built for agentic coding and long-horizon work. It combines mixed sliding-window + global attention (3:1 across 40 layers) with sigmoid per-head gating and FP8 KV cache, so it stays compact enough to run locally while supporting a 256K-token context. The 2.1 point release improves coding quality over XS.2 (SWE-bench Multilingual 57.7% → 63.1%, SWE-bench Verified 69.9% → 70.9%). ### Key features - **Mixed SWA + global attention**: 30 sliding-window layers (window=512) interleaved with 10 global-attention layers, each with per-layer rotary scaling. - **Native FP8 KV cache**: KV cache is quantized to FP8 to reduce memory per token. - **Interleaved reasoning**: thinking blocks emitted between tool calls; toggled per-request via `enable_thinking`. - **Tool calling**: Poolside-specific XML-style tool-call protocol, parsed via `poolside_v1`. - **256 experts + 1 shared expert** with top-8 routing. ## Prerequisites Laguna XS-2.1 support is available in vLLM 0.21.0 and later (via [PR #41129](https://github.com/vllm-project/vllm/pull/41129)). ### pip ```bash uv pip install -U 'vllm>=0.21.0' ``` ### Docker ```bash docker pull vllm/vllm-openai:latest ``` ## Quantized variants Pre-quantized checkpoints are published alongside the BF16 weights — select one with the **Variant** control above: | Variant | Checkpoint | Weights | ~VRAM | Notes | |---------|-----------|---------|-------|-------| | BF16 | [Laguna-XS-2.1](https://huggingface.co/poolside/Laguna-XS-2.1) | BF16 | 80 GB | Default | | FP8 | [Laguna-XS-2.1-FP8](https://huggingface.co/poolside/Laguna-XS-2.1-FP8) | Block-FP8 | 40 GB | Set `VLLM_BLOCKSCALE_FP8_GEMM_FLASHINFER=0` | | NVFP4 | [Laguna-XS-2.1-NVFP4](https://huggingface.co/poolside/Laguna-XS-2.1-NVFP4) | NVFP4 | 26 GB | Blackwell only | | INT4 | [Laguna-XS-2.1-INT4](https://huggingface.co/poolside/Laguna-XS-2.1-INT4) | INT4 (W4A16) | 29 GB | Hopper or Blackwell | Quantization is detected automatically from each checkpoint's `quantization_config`, so the launch command is identical — just swap the model ID (select the variant above). All three quantized checkpoints ship an **FP8-quantized KV cache**. > **vLLM ≥ 0.22.0 required for the quantized variants.** The FP8 KV cache needs the per-layer attention-head fix in [vllm#42650](https://github.com/vllm-project/vllm/pull/42650); earlier versions produce scrambled output on non-Hopper GPUs. On older vLLM you can disable the FP8 KV cache with `--kv-cache-dtype-skip-layers $(seq 0 39)`. Precision-matched DFlash speculators are also available for each quantization — [DFlash-FP8](https://huggingface.co/poolside/Laguna-XS-2.1-DFlash-FP8), [DFlash-NVFP4](https://huggingface.co/poolside/Laguna-XS-2.1-DFlash-NVFP4), and [DFlash-INT4](https://huggingface.co/poolside/Laguna-XS-2.1-DFlash-INT4) — if you want to pair speculative decoding with a quantized base at matching precision. ## Launch command ### Single GPU (H100/H200/B200, BF16) ```bash vllm serve poolside/Laguna-XS-2.1 \ --trust-remote-code \ --max-model-len 262144 \ --enable-auto-tool-choice \ --tool-call-parser poolside_v1 \ --reasoning-parser poolside_v1 ``` ### Docker ```bash docker run -itd --name laguna-xs21 \ --ipc=host --network host --shm-size 16G --gpus all \ -v ~/.cache/huggingface:/root/.cache/huggingface \ vllm/vllm-openai:latest \ --model poolside/Laguna-XS-2.1 \ --trust-remote-code \ --max-model-len 262144 \ --enable-auto-tool-choice \ --tool-call-parser poolside_v1 \ --reasoning-parser poolside_v1 \ --host 0.0.0.0 --port 8000 ``` ## Controlling reasoning Reasoning is **off by default** in the chat template. Enable it per-request: ```python from openai import OpenAI client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY") resp = client.chat.completions.create( model="poolside/Laguna-XS-2.1", messages=[{"role": "user", "content": "Write a Python retry wrapper with exponential backoff."}], extra_body={"chat_template_kwargs": {"enable_thinking": True}}, temperature=0.7, top_p=1.0, extra_query={"top_k": 20}, ) print(resp.choices[0].message.reasoning) print(resp.choices[0].message.content) ``` Or default-on with `--default-chat-template-kwargs '{"enable_thinking": true}'`. ## Speculative decoding (DFlash) Enable the **Spec Decoding** toggle above to attach Poolside's [DFlash draft model](https://huggingface.co/poolside/Laguna-XS-2.1-DFlash) — a 5-layer Llama-style speculator that proposes up to 15 tokens per step. Reported end-to-end speedup is 1.67x–2.64x across evaluation datasets, with a mean accepted length of 3.55–4.57 tokens per step. Requires: - vLLM built from [PR #46853](https://github.com/vllm-project/vllm/pull/46853) (adds DFlash drafter support for Laguna-XS-2.1) — not yet in a stable release. - `--moe-backend triton` — the default DeepGEMM MoE backend is currently incompatible with the DFlash draft path, so force the Triton MoE backend. Enabling the **Spec Decoding** toggle above adds this flag automatically. Example: ```bash vllm serve poolside/Laguna-XS-2.1 \ --trust-remote-code \ --max-model-len 16384 \ --enable-auto-tool-choice \ --tool-call-parser poolside_v1 \ --reasoning-parser poolside_v1 \ --speculative-config '{"model":"poolside/Laguna-XS-2.1-DFlash","num_speculative_tokens":15,"method":"dflash"}' \ --moe-backend triton ``` ## References - [Model card](https://huggingface.co/poolside/Laguna-XS-2.1) - [Release blog post](https://poolside.ai/blog/laguna-a-deeper-dive) - [vLLM support PR #41129](https://github.com/vllm-project/vllm/pull/41129) - [vLLM DFlash spec-decoding PR #46853](https://github.com/vllm-project/vllm/pull/46853) - [DFlash draft model](https://huggingface.co/poolside/Laguna-XS-2.1-DFlash) --- # baidu/Unlimited-OCR on vLLM Source: https://recipes.vllm.ai/baidu/Unlimited-OCR Provider: Ernie (Baidu) Released: 2026-06-19T09:40:33.000Z Updated: 2026-06-28 Architecture: dense Parameters: 3B Context length: 32,768 Min vLLM: 0.25.0 Tasks: multimodal Summary: Baidu's state-of-the-art document-parsing model with Reference Sliding Window Attention (R-SWA), optimized for full-page OCR and markdown generation. ## Variants - default: precision=bf16, vram_minimum_gb=8 Full precision BF16 (~3B params) ## Recommended base flags ```bash vllm serve 'baidu/Unlimited-OCR' \ --trust-remote-code \ --logits_processors \ vllm.model_executor.models.unlimited_ocr:NGramPerReqLogitsProcessor \ --no-enable-prefix-caching \ --mm-processor-cache-gb \ 0 ``` ## Guide ## Overview [Unlimited-OCR](https://huggingface.co/baidu/Unlimited-OCR) is Baidu's DeepSeek-OCR-lineage document-parsing model. It shares the DeepSeek-OCR *gundam* vision stack (SAM-ViT-B + CLIP-L DeepEncoder, `base_size=1024` / `image_size=640` / crop) and ships with the same n-gram logits processor for high-quality, repetition-free document → markdown generation. ## Prerequisites - **Hardware**: a single GPU with >=8 GB VRAM is enough for BF16 inference. - **vLLM**: served from the dedicated release image (the architecture is not yet in a stable pip wheel). Pull the image shown in the Install → Docker tab. ## Serving recipe (required) The model has **no chat template** and is trained for a specific prompt/decode recipe — without it the model returns empty output. The required pieces: - register the no-repeat-ngram logits processor on the server: `--logits_processors vllm.model_executor.models.unlimited_ocr:NGramPerReqLogitsProcessor` - prompt text must begin with a literal `` (e.g. `document parsing.`) - `skip_special_tokens=False` - pass the processor args per request: `ngram_size=35`, `window_size=128` (use `window_size=1024` for multi-page / PDF input) ## Client Usage ### Online OCR serving ```bash docker run --rm --gpus all --network host --ipc host \ vllm/vllm-openai:unlimited-ocr \ baidu/Unlimited-OCR \ --trust-remote-code \ --logits_processors vllm.model_executor.models.unlimited_ocr:NGramPerReqLogitsProcessor \ --no-enable-prefix-caching \ --mm-processor-cache-gb 0 ``` ```python import time from openai import OpenAI client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1", timeout=3600) messages = [ { "role": "user", "content": [ {"type": "text", "text": "document parsing."}, {"type": "image_url", "image_url": {"url": "https://huggingface.co/baidu/Unlimited-OCR/resolve/main/assets/baidu.png"}}, ], } ] start = time.time() response = client.chat.completions.create( model="baidu/Unlimited-OCR", messages=messages, max_tokens=8192, temperature=0.0, extra_body={ "skip_special_tokens": False, "vllm_xargs": {"ngram_size": 35, "window_size": 128}, }, ) print(f"Response costs: {time.time() - start:.2f}s") print(f"Generated text: {response.choices[0].message.content}") ``` The raw output carries `<|ref|>…<|/ref|>` / `<|det|>…<|/det|>` grounding tokens; unwrap the `<|ref|>` text and drop the `<|det|>` coordinate boxes to get clean markdown. ## Troubleshooting / Configuration Tips - **The custom logits processor is required** — without it long documents loop on `<|det|>` coordinate tokens. - Empty output almost always means the prompt is missing the literal `` prefix or `skip_special_tokens` was left `True`. - OCR tasks don't benefit from prefix caching / image reuse, so prefix caching and the mm-processor cache are disabled in the recipe. - Single-image input uses gundam (crop) mode; multi-image requests automatically fall back to non-crop (base) mode — use `window_size=1024` for those. ## References - [Unlimited-OCR on Hugging Face](https://huggingface.co/baidu/Unlimited-OCR) - [vLLM multimodal inputs guide](https://docs.vllm.ai/en/latest/features/multimodal_inputs.html#offline-inference) --- # LiquidAI/LFM2.5-230M on vLLM Source: https://recipes.vllm.ai/LiquidAI/LFM2.5-230M Provider: LiquidAI Released: 2026-06-24T23:14:11.000Z Updated: 2026-06-25 Architecture: dense Parameters: 230M Context length: 32,768 Min vLLM: 0.23.0 Tasks: text Verified hardware: H100, H200, B200 Summary: Liquid AI's most compact LFM2.5 chat model (230M) on the LFM2 hybrid conv+attention backbone — distilled from LFM2.5-350M, tool calling and 32K context, built for edge / on-device serving. ## Variants - default: precision=bf16, vram_minimum_gb=1 Full BF16 — fits essentially any GPU (~0.5 GB of weights) ## Guide ## Overview [LFM2.5-230M](https://huggingface.co/LiquidAI/LFM2.5-230M) is the most compact chat model in [Liquid AI's](https://www.liquid.ai/) LFM2.5 family, built on the **LFM2 hybrid backbone**: short-range gated convolution blocks interleaved with grouped-query attention. It was **distilled from LFM2.5-350M** and refined with multi-stage reinforcement learning, then trained on a 19T-token budget — keeping memory and latency low enough for the tightest edge and on-device budgets while still supporting tool calling and a 32K context window through vLLM's OpenAI-compatible API. ### Key Features - **Hybrid backbone**: 14 layers — 8 double-gated short-convolution blocks interleaved with 6 grouped-query attention blocks — for a smaller KV cache and lower decode latency than a same-size full-attention transformer. - **32K context**: Long-context support (`context_length = 32768`). - **Tool calling**: Pythonic tool calls (`<|tool_call_start|>…<|tool_call_end|>`) surfaced as OpenAI `tool_calls` by vLLM's native `lfm2` parser. - **Native vLLM support**: Served via the `Lfm2ForCausalLM` architecture — no `--trust-remote-code` required. - **Edge-ready**: ~0.5 GB of BF16 weights — runs from low-cost CPUs to production GPUs (213 tok/s decode on a Galaxy S25 Ultra, 42 tok/s on a Raspberry Pi 5). Best suited for data extraction and lightweight on-device agentic pipelines; the model card notes it is **not** recommended for reasoning-heavy workloads (advanced math, code generation, creative writing) — reach for a larger sibling there. ### Supported Variants Dense: - `LiquidAI/LFM2.5-230M` (230M) - `LiquidAI/LFM2.5-350M` (350M) - `LiquidAI/LFM2.5-1.2B-Instruct` (1.2B) - `LiquidAI/LFM2.5-1.2B-Thinking` (1.2B, reasoning) - `LiquidAI/LFM2.5-1.2B-JP` / `LiquidAI/LFM2.5-1.2B-JP-202606` (Japanese) - `LiquidAI/LFM2.5-1.2B-Base` (pretrained base) MoE: - `LiquidAI/LFM2.5-8B-A1B` (8B total / ~1B active) Vision-Language: - `LiquidAI/LFM2.5-VL-450M`, `LiquidAI/LFM2.5-VL-1.6B` See the [LFM2.5 usage guide](../../LiquidAI/LFM2.5) for the full family. ## Prerequisites - **Hardware**: 1× GPU (~1 GB VRAM for weights; any modern GPU works). Verified on H100. - **vLLM**: ≥ 0.23.0 — the LFM2 architecture ships in the 0.23.0 stable release. ### pip (NVIDIA CUDA) ```bash uv venv source .venv/bin/activate uv pip install -U vllm --torch-backend auto ``` ## Deployment Configurations ### Quick Start (Single GPU, BF16) ```bash vllm serve LiquidAI/LFM2.5-230M ``` ### Full-Featured Server Launch Enables tool calling: ```bash vllm serve LiquidAI/LFM2.5-230M \ --enable-auto-tool-choice \ --tool-call-parser lfm2 \ --host 0.0.0.0 --port 8000 ``` ### Docker (NVIDIA) ```bash docker run -itd --name lfm2.5-230m \ --ipc=host --network host --shm-size 16G --gpus all \ -v ~/.cache/huggingface:/root/.cache/huggingface \ vllm/vllm-openai:latest \ --model LiquidAI/LFM2.5-230M \ --host 0.0.0.0 --port 8000 ``` ## Client Usage ### Text Generation The model card recommends temperature `0.1`, `top_k 50`, `repetition_penalty 1.05` (`top_k` and `repetition_penalty` ride in `extra_body`). ```python from openai import OpenAI client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY") response = client.chat.completions.create( model="LiquidAI/LFM2.5-230M", messages=[{"role": "user", "content": "Give me three uses for baking soda."}], temperature=0.1, extra_body={"top_k": 50, "repetition_penalty": 1.05}, ) print(response.choices[0].message.content) ``` ### Tool Calling Launch with `--enable-auto-tool-choice --tool-call-parser lfm2`, then pass `tools=[…]`; the `lfm2` parser converts the model's Pythonic call into a standard `tool_calls` array. ```python response = client.chat.completions.create( model="LiquidAI/LFM2.5-230M", messages=[{"role": "user", "content": "What's the weather in Paris?"}], tools=[{ "type": "function", "function": { "name": "get_weather", "description": "Get current weather for a location", "parameters": {"type": "object", "properties": {"location": {"type": "string"}}, "required": ["location"]}, }, }], temperature=0.1, extra_body={"top_k": 50, "repetition_penalty": 1.05}, ) print(response.choices[0].message.tool_calls) ``` ### Structured Outputs vLLM's guided decoding constrains output to a JSON schema via `response_format`. The model does not see the schema itself — put semantic instructions in the system prompt. Data extraction is a primary use case for this model, so structured outputs pair naturally with it. ## Configuration Tips - At 230M the model fits any GPU; raise `--max-num-seqs` to push batch throughput. - Set `--max-model-len` to match your workload (up to 32K). - `--gpu-memory-utilization 0.90–0.95` maximizes KV cache capacity. - Sampling presets are per-request client defaults — don't bake them into `vllm serve`. ## References - [Model card](https://huggingface.co/LiquidAI/LFM2.5-230M) - [Blog post](https://www.liquid.ai/blog/lfm2-5-230m) - [LFM2.5 usage guide](../../LiquidAI/LFM2.5) - [Liquid AI](https://www.liquid.ai/) - [vLLM supported models](https://docs.vllm.ai/en/latest/models/supported_models.html) --- # LiquidAI/LFM2.5-1.2B-Base on vLLM Source: https://recipes.vllm.ai/LiquidAI/LFM2.5-1.2B-Base Provider: LiquidAI Released: 2026-01-05T16:27:32.000Z Updated: 2026-06-24 Architecture: dense Parameters: 1.2B Context length: 32,768 Min vLLM: 0.23.0 Tasks: text Verified hardware: H100, H200, B200 Summary: Liquid AI's 1.2B pretrained base model on the LFM2 hybrid conv+attention backbone — a text-completion and fine-tuning foundation (no chat template). ## Variants - default: precision=bf16, vram_minimum_gb=3 Full BF16 — single 8 GB+ NVIDIA GPU with room for KV cache ## Guide ## Overview [LFM2.5-1.2B-Base](https://huggingface.co/LiquidAI/LFM2.5-1.2B-Base) is the pretrained base checkpoint behind [Liquid AI's](https://www.liquid.ai/) LFM2.5-1.2B instruct models, built on the **LFM2 hybrid backbone** (short-range gated convolution blocks interleaved with grouped-query attention). It is **not** instruction-tuned and has no chat template — use the `/v1/completions` endpoint (raw text continuation), or fine-tune it for your own task. ### Key Features - **Pretrained base**: No chat template, no instruction tuning — a clean foundation for fine-tuning or raw completion. - **Hybrid backbone**: Gated short convolutions interleaved with grouped-query attention — a smaller KV cache and lower decode latency than a same-size full-attention transformer. - **32K context**: 32,768-token context window. - **Native vLLM support**: Served via the `Lfm2ForCausalLM` architecture. ### Supported Variants Dense: - `LiquidAI/LFM2.5-230M` (230M) - `LiquidAI/LFM2.5-350M` (350M) - `LiquidAI/LFM2.5-1.2B-Instruct` (1.2B) - `LiquidAI/LFM2.5-1.2B-Thinking` (1.2B, reasoning) - `LiquidAI/LFM2.5-1.2B-JP` / `LiquidAI/LFM2.5-1.2B-JP-202606` (Japanese) - `LiquidAI/LFM2.5-1.2B-Base` (pretrained base) MoE: - `LiquidAI/LFM2.5-8B-A1B` (8B total / ~1B active) Vision-Language: - `LiquidAI/LFM2.5-VL-450M`, `LiquidAI/LFM2.5-VL-1.6B` See the [LFM2.5 usage guide](../../LiquidAI/LFM2.5) for the full family. ## Prerequisites - **Hardware**: 1× GPU with ≥8 GB VRAM. - **vLLM**: ≥ 0.23.0. ### pip (NVIDIA CUDA) ```bash uv venv source .venv/bin/activate uv pip install -U vllm --torch-backend auto ``` ## Deployment Configurations ### Quick Start (Single GPU, BF16) ```bash vllm serve LiquidAI/LFM2.5-1.2B-Base ``` ### Docker (NVIDIA) ```bash docker run -itd --name lfm2.5-base \ --ipc=host --network host --shm-size 16G --gpus all \ -v ~/.cache/huggingface:/root/.cache/huggingface \ vllm/vllm-openai:latest \ --model LiquidAI/LFM2.5-1.2B-Base \ --host 0.0.0.0 --port 8000 ``` ## Client Usage ### Text Completion This is a base model — use the **completions** endpoint, not chat. Recommended sampling: temperature `0.3`, `min_p 0.15`, `repetition_penalty 1.05`. ```python from openai import OpenAI client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY") response = client.completions.create( model="LiquidAI/LFM2.5-1.2B-Base", prompt="The three laws of thermodynamics are:", temperature=0.3, extra_body={"min_p": 0.15, "repetition_penalty": 1.05}, ) print(response.choices[0].text) ``` ## Configuration Tips - Use `/v1/completions` (raw continuation) — there is no chat template on a base model. - For fine-tuning, this checkpoint is the recommended starting point for custom LFM2.5-1.2B tasks. - Set `--max-model-len` to match your workload (up to 32K). - Sampling presets are per-request client defaults — don't bake them into `vllm serve`. ## References - [Model card](https://huggingface.co/LiquidAI/LFM2.5-1.2B-Base) - [LFM2.5 usage guide](../../LiquidAI/LFM2.5) - [Liquid AI](https://www.liquid.ai/) - [vLLM supported models](https://docs.vllm.ai/en/latest/models/supported_models.html) --- # LiquidAI/LFM2.5-1.2B-Instruct on vLLM Source: https://recipes.vllm.ai/LiquidAI/LFM2.5-1.2B-Instruct Provider: LiquidAI Released: 2026-01-06T00:28:46.000Z Updated: 2026-06-24 Architecture: dense Parameters: 1.2B Context length: 32,768 Min vLLM: 0.23.0 Tasks: text Verified hardware: H100, H200, B200 Summary: Liquid AI's 1.2B instruction-tuned model on the LFM2 hybrid conv+attention backbone, with tool calling and a 32K context window on a single small GPU. ## Variants - default: precision=bf16, vram_minimum_gb=3 Full BF16 — single 8 GB+ NVIDIA GPU with room for KV cache ## Guide ## Overview [LFM2.5-1.2B-Instruct](https://huggingface.co/LiquidAI/LFM2.5-1.2B-Instruct) is a 1.2B instruction-tuned model from [Liquid AI](https://www.liquid.ai/), built on the **LFM2 hybrid backbone**: short-range gated convolution blocks interleaved with grouped-query attention. The hybrid design keeps the KV cache small and decode fast, so the model serves comfortably on a single small GPU while supporting a 32K context window and Pythonic tool calling — all through vLLM's OpenAI-compatible API. ### Key Features - **Hybrid backbone**: Gated short convolutions interleaved with grouped-query attention — a smaller KV cache and lower decode latency than a same-size full-attention transformer. - **32K context**: 32,768-token context window. - **Tool calling**: Pythonic tool calls (`<|tool_call_start|>…<|tool_call_end|>`) surfaced as OpenAI `tool_calls` by vLLM's native `lfm2` parser. - **Native vLLM support**: Served via the `Lfm2ForCausalLM` architecture. - **Edge-ready family**: Day-one support across llama.cpp, MLX, ONNX, vLLM, and SGLang. ### Supported Variants Dense: - `LiquidAI/LFM2.5-230M` (230M) - `LiquidAI/LFM2.5-350M` (350M) - `LiquidAI/LFM2.5-1.2B-Instruct` (1.2B) - `LiquidAI/LFM2.5-1.2B-Thinking` (1.2B, reasoning) - `LiquidAI/LFM2.5-1.2B-JP` / `LiquidAI/LFM2.5-1.2B-JP-202606` (Japanese) - `LiquidAI/LFM2.5-1.2B-Base` (pretrained base) MoE: - `LiquidAI/LFM2.5-8B-A1B` (8B total / ~1B active) Vision-Language: - `LiquidAI/LFM2.5-VL-450M`, `LiquidAI/LFM2.5-VL-1.6B` See the [LFM2.5 usage guide](../../LiquidAI/LFM2.5) for the full family. ## Prerequisites - **Hardware**: 1× GPU with ≥8 GB VRAM (e.g. L4, A10, RTX 4090, H100). - **vLLM**: ≥ 0.23.0. ### pip (NVIDIA CUDA) ```bash uv venv source .venv/bin/activate uv pip install -U vllm --torch-backend auto ``` ## Deployment Configurations ### Quick Start (Single GPU, BF16) ```bash vllm serve LiquidAI/LFM2.5-1.2B-Instruct ``` Cap the context to fit a smaller GPU (the model supports up to 32K): ```bash vllm serve LiquidAI/LFM2.5-1.2B-Instruct --max-model-len 32768 ``` ### Full-Featured Server Launch Enables tool calling: ```bash vllm serve LiquidAI/LFM2.5-1.2B-Instruct \ --enable-auto-tool-choice \ --tool-call-parser lfm2 \ --host 0.0.0.0 --port 8000 ``` ### Docker (NVIDIA) ```bash docker run -itd --name lfm2.5-1.2b \ --ipc=host --network host --shm-size 16G --gpus all \ -v ~/.cache/huggingface:/root/.cache/huggingface \ vllm/vllm-openai:latest \ --model LiquidAI/LFM2.5-1.2B-Instruct \ --host 0.0.0.0 --port 8000 ``` ## Client Usage ### Text Generation The model card recommends temperature `0.1`, `top_k 50`, `repetition_penalty 1.05`. `top_k` and `repetition_penalty` are vLLM extra sampling params — pass them via `extra_body`. ```python from openai import OpenAI client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY") response = client.chat.completions.create( model="LiquidAI/LFM2.5-1.2B-Instruct", messages=[{"role": "user", "content": "What is C. elegans? Answer in one sentence."}], temperature=0.1, extra_body={"top_k": 50, "repetition_penalty": 1.05}, ) print(response.choices[0].message.content) ``` ### Tool Calling Launch with `--enable-auto-tool-choice --tool-call-parser lfm2`, then pass `tools=[…]`. The `lfm2` parser converts the model's Pythonic call into a standard `tool_calls` array. ```python response = client.chat.completions.create( model="LiquidAI/LFM2.5-1.2B-Instruct", messages=[{"role": "user", "content": "What's the weather in Paris?"}], tools=[{ "type": "function", "function": { "name": "get_weather", "description": "Get current weather for a location", "parameters": {"type": "object", "properties": {"location": {"type": "string"}}, "required": ["location"]}, }, }], temperature=0.1, extra_body={"top_k": 50, "repetition_penalty": 1.05}, ) print(response.choices[0].message.tool_calls) ``` ### Structured Outputs vLLM's guided decoding constrains output to a JSON schema via `response_format`. The model does not see the schema itself — put semantic instructions (units, formatting) in the system prompt. ## Configuration Tips - Set `--max-model-len` to match your workload (up to 32K); lowering it frees VRAM for KV cache. - `--gpu-memory-utilization 0.90–0.95` maximizes KV cache capacity. - FP8 KV cache (`--kv-cache-dtype fp8`) roughly halves KV memory. - Sampling presets are per-request client defaults — don't bake them into `vllm serve`. ## References - [Model card](https://huggingface.co/LiquidAI/LFM2.5-1.2B-Instruct) - [LFM2.5 usage guide](../../LiquidAI/LFM2.5) - [Liquid AI](https://www.liquid.ai/) - [vLLM supported models](https://docs.vllm.ai/en/latest/models/supported_models.html) --- # LiquidAI/LFM2.5-1.2B-JP-202606 on vLLM Source: https://recipes.vllm.ai/LiquidAI/LFM2.5-1.2B-JP-202606 Provider: LiquidAI Released: 2026-05-26T04:50:17.000Z Updated: 2026-06-24 Architecture: dense Parameters: 1.2B Context length: 32,768 Min vLLM: 0.23.0 Tasks: text Verified hardware: H100, H200, B200 Summary: Liquid AI's updated (2026-06) 1.2B Japanese chat model on the LFM2 hybrid conv+attention backbone — adds tool calling, with a 32K context window. ## Variants - default: precision=bf16, vram_minimum_gb=3 Full BF16 — single 8 GB+ NVIDIA GPU with room for KV cache ## Guide ## Overview [LFM2.5-1.2B-JP-202606](https://huggingface.co/LiquidAI/LFM2.5-1.2B-JP-202606) is the updated (June 2026) Japanese-specialized 1.2B chat model from [Liquid AI](https://www.liquid.ai/), adding tool calling on top of the **LFM2 hybrid backbone** (short-range gated convolution blocks interleaved with grouped-query attention). It serves on a single small GPU through vLLM's OpenAI-compatible API. ### Key Features - **Japanese-specialized**: Updated June 2026 checkpoint tuned for Japanese chat. - **Hybrid backbone**: Gated short convolutions interleaved with grouped-query attention — a smaller KV cache and lower decode latency than a same-size full-attention transformer. - **Tool calling**: Pythonic tool calls (`<|tool_call_start|>…<|tool_call_end|>`) surfaced as OpenAI `tool_calls` by vLLM's native `lfm2` parser. - **32K context**: 32,768-token context window. - **Native vLLM support**: Served via the `Lfm2ForCausalLM` architecture. ### Supported Variants Dense: - `LiquidAI/LFM2.5-230M` (230M) - `LiquidAI/LFM2.5-350M` (350M) - `LiquidAI/LFM2.5-1.2B-Instruct` (1.2B) - `LiquidAI/LFM2.5-1.2B-Thinking` (1.2B, reasoning) - `LiquidAI/LFM2.5-1.2B-JP` / `LiquidAI/LFM2.5-1.2B-JP-202606` (Japanese) - `LiquidAI/LFM2.5-1.2B-Base` (pretrained base) MoE: - `LiquidAI/LFM2.5-8B-A1B` (8B total / ~1B active) Vision-Language: - `LiquidAI/LFM2.5-VL-450M`, `LiquidAI/LFM2.5-VL-1.6B` See the [LFM2.5 usage guide](../../LiquidAI/LFM2.5) for the full family. ## Prerequisites - **Hardware**: 1× GPU with ≥8 GB VRAM. - **vLLM**: ≥ 0.23.0. ### pip (NVIDIA CUDA) ```bash uv venv source .venv/bin/activate uv pip install -U vllm --torch-backend auto ``` ## Deployment Configurations ### Quick Start (Single GPU, BF16) ```bash vllm serve LiquidAI/LFM2.5-1.2B-JP-202606 ``` ### Full-Featured Server Launch Enables tool calling: ```bash vllm serve LiquidAI/LFM2.5-1.2B-JP-202606 \ --enable-auto-tool-choice \ --tool-call-parser lfm2 \ --host 0.0.0.0 --port 8000 ``` ### Docker (NVIDIA) ```bash docker run -itd --name lfm2.5-jp-202606 \ --ipc=host --network host --shm-size 16G --gpus all \ -v ~/.cache/huggingface:/root/.cache/huggingface \ vllm/vllm-openai:latest \ --model LiquidAI/LFM2.5-1.2B-JP-202606 \ --host 0.0.0.0 --port 8000 ``` ## Client Usage ### Text Generation The updated checkpoint uses temperature `0.1`, `top_k 50`, `repetition_penalty 1.05` (`top_k` and `repetition_penalty` ride in `extra_body`). ```python from openai import OpenAI client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY") response = client.chat.completions.create( model="LiquidAI/LFM2.5-1.2B-JP-202606", messages=[{"role": "user", "content": "京都でおすすめの観光スポットを3つ教えてください。"}], temperature=0.1, extra_body={"top_k": 50, "repetition_penalty": 1.05}, ) print(response.choices[0].message.content) ``` ### Tool Calling Launch with `--enable-auto-tool-choice --tool-call-parser lfm2`, then pass `tools=[…]`; the `lfm2` parser converts the model's Pythonic call into a standard `tool_calls` array. ## Configuration Tips - Set `--max-model-len` to match your workload (up to 32K); lowering it frees VRAM for KV cache. - `--gpu-memory-utilization 0.90–0.95` maximizes KV cache capacity. - Sampling presets are per-request client defaults — don't bake them into `vllm serve`. ## References - [Model card](https://huggingface.co/LiquidAI/LFM2.5-1.2B-JP-202606) - [LFM2.5 usage guide](../../LiquidAI/LFM2.5) - [Liquid AI](https://www.liquid.ai/) - [vLLM supported models](https://docs.vllm.ai/en/latest/models/supported_models.html) --- # LiquidAI/LFM2.5-1.2B-JP on vLLM Source: https://recipes.vllm.ai/LiquidAI/LFM2.5-1.2B-JP Provider: LiquidAI Released: 2026-01-04T23:45:18.000Z Updated: 2026-06-24 Architecture: dense Parameters: 1.2B Context length: 32,768 Min vLLM: 0.23.0 Tasks: text Verified hardware: H100, H200, B200 Summary: Liquid AI's 1.2B Japanese-specialized chat model on the LFM2 hybrid conv+attention backbone, with a 32K context window on a single small GPU. ## Variants - default: precision=bf16, vram_minimum_gb=3 Full BF16 — single 8 GB+ NVIDIA GPU with room for KV cache ## Guide ## Overview [LFM2.5-1.2B-JP](https://huggingface.co/LiquidAI/LFM2.5-1.2B-JP) is the Japanese-specialized 1.2B chat model from [Liquid AI](https://www.liquid.ai/), built on the **LFM2 hybrid backbone** (short-range gated convolution blocks interleaved with grouped-query attention). It serves on a single small GPU through vLLM's OpenAI-compatible API. For the latest Japanese checkpoint with tool calling, see [LFM2.5-1.2B-JP-202606](https://huggingface.co/LiquidAI/LFM2.5-1.2B-JP-202606). ### Key Features - **Japanese-specialized**: Tuned for Japanese chat and instruction following. - **Hybrid backbone**: Gated short convolutions interleaved with grouped-query attention — a smaller KV cache and lower decode latency than a same-size full-attention transformer. - **32K context**: 32,768-token context window. - **Native vLLM support**: Served via the `Lfm2ForCausalLM` architecture. ### Supported Variants Dense: - `LiquidAI/LFM2.5-230M` (230M) - `LiquidAI/LFM2.5-350M` (350M) - `LiquidAI/LFM2.5-1.2B-Instruct` (1.2B) - `LiquidAI/LFM2.5-1.2B-Thinking` (1.2B, reasoning) - `LiquidAI/LFM2.5-1.2B-JP` / `LiquidAI/LFM2.5-1.2B-JP-202606` (Japanese) - `LiquidAI/LFM2.5-1.2B-Base` (pretrained base) MoE: - `LiquidAI/LFM2.5-8B-A1B` (8B total / ~1B active) Vision-Language: - `LiquidAI/LFM2.5-VL-450M`, `LiquidAI/LFM2.5-VL-1.6B` See the [LFM2.5 usage guide](../../LiquidAI/LFM2.5) for the full family. ## Prerequisites - **Hardware**: 1× GPU with ≥8 GB VRAM. - **vLLM**: ≥ 0.23.0. ### pip (NVIDIA CUDA) ```bash uv venv source .venv/bin/activate uv pip install -U vllm --torch-backend auto ``` ## Deployment Configurations ### Quick Start (Single GPU, BF16) ```bash vllm serve LiquidAI/LFM2.5-1.2B-JP ``` ### Docker (NVIDIA) ```bash docker run -itd --name lfm2.5-jp \ --ipc=host --network host --shm-size 16G --gpus all \ -v ~/.cache/huggingface:/root/.cache/huggingface \ vllm/vllm-openai:latest \ --model LiquidAI/LFM2.5-1.2B-JP \ --host 0.0.0.0 --port 8000 ``` ## Client Usage ### Text Generation The card recommends temperature `0.3`, `min_p 0.15`, `repetition_penalty 1.05` for the JP checkpoint (`min_p` and `repetition_penalty` ride in `extra_body`). ```python from openai import OpenAI client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY") response = client.chat.completions.create( model="LiquidAI/LFM2.5-1.2B-JP", messages=[{"role": "user", "content": "日本の四季について簡単に説明してください。"}], temperature=0.3, extra_body={"min_p": 0.15, "repetition_penalty": 1.05}, ) print(response.choices[0].message.content) ``` ## Configuration Tips - Set `--max-model-len` to match your workload (up to 32K); lowering it frees VRAM for KV cache. - `--gpu-memory-utilization 0.90–0.95` maximizes KV cache capacity. - Sampling presets are per-request client defaults — don't bake them into `vllm serve`. ## References - [Model card](https://huggingface.co/LiquidAI/LFM2.5-1.2B-JP) - [LFM2.5 usage guide](../../LiquidAI/LFM2.5) - [Liquid AI](https://www.liquid.ai/) - [vLLM supported models](https://docs.vllm.ai/en/latest/models/supported_models.html) --- # LiquidAI/LFM2.5-1.2B-Thinking on vLLM Source: https://recipes.vllm.ai/LiquidAI/LFM2.5-1.2B-Thinking Provider: LiquidAI Released: 2026-01-20T13:55:42.000Z Updated: 2026-06-24 Architecture: dense Parameters: 1.2B Context length: 32,768 Min vLLM: 0.23.0 Tasks: text Verified hardware: H100, H200, B200 Summary: Liquid AI's 1.2B reasoning model on the LFM2 hybrid conv+attention backbone — chain-of-thought, tool calling, and 32K context on a single small GPU. ## Variants - default: precision=bf16, vram_minimum_gb=3 Full BF16 — single 8 GB+ NVIDIA GPU with room for KV cache ## Guide ## Overview [LFM2.5-1.2B-Thinking](https://huggingface.co/LiquidAI/LFM2.5-1.2B-Thinking) is the reasoning variant of [Liquid AI's](https://www.liquid.ai/) 1.2B LFM2.5 model, built on the **LFM2 hybrid backbone** (short-range gated convolution blocks interleaved with grouped-query attention). On reasoning-heavy tasks it produces an explicit `…` chain-of-thought before its final answer, so it reasons well above its weight class while still serving on a single small GPU through vLLM's OpenAI-compatible API. ### Key Features - **Hybrid backbone**: Gated short convolutions interleaved with grouped-query attention — a smaller KV cache and lower decode latency than a same-size full-attention transformer. - **`` reasoning**: Emits an explicit `…` chain-of-thought on non-trivial problems; vLLM's `qwen3` parser splits it into a separate `reasoning` field. - **32K context**: 32,768-token context window. - **Tool calling**: Pythonic tool calls surfaced as OpenAI `tool_calls` by vLLM's native `lfm2` parser. - **Native vLLM support**: Served via the `Lfm2ForCausalLM` architecture. ### Supported Variants Dense: - `LiquidAI/LFM2.5-230M` (230M) - `LiquidAI/LFM2.5-350M` (350M) - `LiquidAI/LFM2.5-1.2B-Instruct` (1.2B) - `LiquidAI/LFM2.5-1.2B-Thinking` (1.2B, reasoning) - `LiquidAI/LFM2.5-1.2B-JP` / `LiquidAI/LFM2.5-1.2B-JP-202606` (Japanese) - `LiquidAI/LFM2.5-1.2B-Base` (pretrained base) MoE: - `LiquidAI/LFM2.5-8B-A1B` (8B total / ~1B active, also a reasoning model) Vision-Language: - `LiquidAI/LFM2.5-VL-450M`, `LiquidAI/LFM2.5-VL-1.6B` See the [LFM2.5 usage guide](../../LiquidAI/LFM2.5) for the full family. ## Prerequisites - **Hardware**: 1× GPU with ≥8 GB VRAM. - **vLLM**: ≥ 0.23.0. ### pip (NVIDIA CUDA) ```bash uv venv source .venv/bin/activate uv pip install -U vllm --torch-backend auto ``` ## Deployment Configurations ### Quick Start (Single GPU, BF16) Enable the reasoning parser so the chain-of-thought is returned in `reasoning`: ```bash vllm serve LiquidAI/LFM2.5-1.2B-Thinking \ --reasoning-parser qwen3 ``` ### Full-Featured Server Launch Reasoning **and** tool calling: ```bash vllm serve LiquidAI/LFM2.5-1.2B-Thinking \ --reasoning-parser qwen3 \ --enable-auto-tool-choice \ --tool-call-parser lfm2 \ --host 0.0.0.0 --port 8000 ``` ### Docker (NVIDIA) ```bash docker run -itd --name lfm2.5-thinking \ --ipc=host --network host --shm-size 16G --gpus all \ -v ~/.cache/huggingface:/root/.cache/huggingface \ vllm/vllm-openai:latest \ --model LiquidAI/LFM2.5-1.2B-Thinking \ --reasoning-parser qwen3 \ --host 0.0.0.0 --port 8000 ``` ## Client Usage ### Reasoning Mode The card recommends a low temperature for the reasoning model — temperature `0.05`, `top_k 50`, `repetition_penalty 1.05`. Do **not** set `max_tokens` — capping it can truncate the chain-of-thought mid-stream. ```python from openai import OpenAI client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY") response = client.chat.completions.create( model="LiquidAI/LFM2.5-1.2B-Thinking", messages=[{"role": "user", "content": "A snail climbs 3 ft up a 20 ft well each day and slides 2 ft back each night. How many days to reach the top?"}], temperature=0.05, extra_body={"top_k": 50, "repetition_penalty": 1.05}, ) msg = response.choices[0].message print("reasoning:", msg.reasoning) print("answer:", msg.content) ``` > **Note**: the model opens the `` channel for non-trivial problems; a simple prompt may > be answered directly, in which case `reasoning` is empty. That is expected behavior — > the `qwen3` parser extracts the block whenever it is present. ### Tool Calling Add `--enable-auto-tool-choice --tool-call-parser lfm2` at launch, then pass `tools=[…]`; the model can reason about which tool to call before emitting it. ### Structured Outputs vLLM's guided decoding constrains output to a JSON schema via `response_format`. The model does not see the schema itself — put semantic instructions in the system prompt. ## Configuration Tips - Give reasoning room: leave `max_tokens` unset so the `` block and answer aren't truncated. - Set `--max-model-len` to match your workload (up to 32K). - `--gpu-memory-utilization 0.90–0.95` maximizes KV cache capacity. - Sampling presets are per-request client defaults — don't bake them into `vllm serve`. ## References - [Model card](https://huggingface.co/LiquidAI/LFM2.5-1.2B-Thinking) - [LFM2.5 usage guide](../../LiquidAI/LFM2.5) - [Liquid AI](https://www.liquid.ai/) - [vLLM supported models](https://docs.vllm.ai/en/latest/models/supported_models.html) --- # LiquidAI/LFM2.5-350M on vLLM Source: https://recipes.vllm.ai/LiquidAI/LFM2.5-350M Provider: LiquidAI Released: 2026-03-31T06:21:25.000Z Updated: 2026-06-24 Architecture: dense Parameters: 350M Context length: 32,768 Min vLLM: 0.23.0 Tasks: text Verified hardware: H100, H200, B200 Summary: Liquid AI's smallest LFM2.5 chat model (350M) on the LFM2 hybrid conv+attention backbone — tool calling and 32K context, light enough for edge GPUs. ## Variants - default: precision=bf16, vram_minimum_gb=1 Full BF16 — fits essentially any GPU (~0.7 GB of weights) ## Guide ## Overview [LFM2.5-350M](https://huggingface.co/LiquidAI/LFM2.5-350M) is the smallest chat model in [Liquid AI's](https://www.liquid.ai/) LFM2.5 family, built on the **LFM2 hybrid backbone**: short-range gated convolution blocks interleaved with grouped-query attention. It shares that backbone with its larger siblings, which keeps memory and latency low enough for edge and on-device deployment while still supporting tool calling and a 32K context window — all through vLLM's OpenAI-compatible API. ### Key Features - **Hybrid backbone**: Gated short convolutions interleaved with grouped-query attention — a smaller KV cache and lower decode latency than a same-size full-attention transformer. - **32K context**: 32,768-token context window. - **Tool calling**: Pythonic tool calls (`<|tool_call_start|>…<|tool_call_end|>`) surfaced as OpenAI `tool_calls` by vLLM's native `lfm2` parser. - **Native vLLM support**: Served via the `Lfm2ForCausalLM` architecture. - **Edge-ready**: ~0.7 GB of BF16 weights — runs on commodity and on-device GPUs. ### Supported Variants Dense: - `LiquidAI/LFM2.5-230M` (230M) - `LiquidAI/LFM2.5-350M` (350M) - `LiquidAI/LFM2.5-1.2B-Instruct` (1.2B) - `LiquidAI/LFM2.5-1.2B-Thinking` (1.2B, reasoning) - `LiquidAI/LFM2.5-1.2B-JP` / `LiquidAI/LFM2.5-1.2B-JP-202606` (Japanese) - `LiquidAI/LFM2.5-1.2B-Base` (pretrained base) MoE: - `LiquidAI/LFM2.5-8B-A1B` (8B total / ~1B active) Vision-Language: - `LiquidAI/LFM2.5-VL-450M`, `LiquidAI/LFM2.5-VL-1.6B` See the [LFM2.5 usage guide](../../LiquidAI/LFM2.5) for the full family. ## Prerequisites - **Hardware**: 1× GPU (~1 GB VRAM for weights; any modern GPU works). - **vLLM**: ≥ 0.23.0. ### pip (NVIDIA CUDA) ```bash uv venv source .venv/bin/activate uv pip install -U vllm --torch-backend auto ``` ## Deployment Configurations ### Quick Start (Single GPU, BF16) ```bash vllm serve LiquidAI/LFM2.5-350M ``` ### Full-Featured Server Launch Enables tool calling: ```bash vllm serve LiquidAI/LFM2.5-350M \ --enable-auto-tool-choice \ --tool-call-parser lfm2 \ --host 0.0.0.0 --port 8000 ``` ### Docker (NVIDIA) ```bash docker run -itd --name lfm2.5-350m \ --ipc=host --network host --shm-size 16G --gpus all \ -v ~/.cache/huggingface:/root/.cache/huggingface \ vllm/vllm-openai:latest \ --model LiquidAI/LFM2.5-350M \ --host 0.0.0.0 --port 8000 ``` ## Client Usage ### Text Generation The model card recommends temperature `0.1`, `top_k 50`, `repetition_penalty 1.05` (`top_k` and `repetition_penalty` ride in `extra_body`). ```python from openai import OpenAI client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY") response = client.chat.completions.create( model="LiquidAI/LFM2.5-350M", messages=[{"role": "user", "content": "Give me three uses for baking soda."}], temperature=0.1, extra_body={"top_k": 50, "repetition_penalty": 1.05}, ) print(response.choices[0].message.content) ``` ### Tool Calling Launch with `--enable-auto-tool-choice --tool-call-parser lfm2`, then pass `tools=[…]`; the `lfm2` parser converts the model's Pythonic call into a standard `tool_calls` array. ```python response = client.chat.completions.create( model="LiquidAI/LFM2.5-350M", messages=[{"role": "user", "content": "What's the weather in Paris?"}], tools=[{ "type": "function", "function": { "name": "get_weather", "description": "Get current weather for a location", "parameters": {"type": "object", "properties": {"location": {"type": "string"}}, "required": ["location"]}, }, }], temperature=0.1, extra_body={"top_k": 50, "repetition_penalty": 1.05}, ) print(response.choices[0].message.tool_calls) ``` ### Structured Outputs vLLM's guided decoding constrains output to a JSON schema via `response_format`. The model does not see the schema itself — put semantic instructions in the system prompt. ## Configuration Tips - At 350M the model fits any GPU; raise `--max-num-seqs` to push batch throughput. - Set `--max-model-len` to match your workload (up to 32K). - `--gpu-memory-utilization 0.90–0.95` maximizes KV cache capacity. - Sampling presets are per-request client defaults — don't bake them into `vllm serve`. ## References - [Model card](https://huggingface.co/LiquidAI/LFM2.5-350M) - [LFM2.5 usage guide](../../LiquidAI/LFM2.5) - [Liquid AI](https://www.liquid.ai/) - [vLLM supported models](https://docs.vllm.ai/en/latest/models/supported_models.html) --- # LiquidAI/LFM2.5-8B-A1B on vLLM Source: https://recipes.vllm.ai/LiquidAI/LFM2.5-8B-A1B Provider: LiquidAI Released: 2026-05-28T09:43:54.000Z Updated: 2026-06-24 Architecture: moe Parameters: 8B Context length: 128,000 Min vLLM: 0.23.0 Tasks: text Verified hardware: H100, H200, B200 Summary: Liquid AI's 8B mixture-of-experts model (~1B active) on the LFM2 hybrid conv+attention backbone, with reasoning and tool calling at ~1B decode cost. ## Variants - default: precision=bf16, vram_minimum_gb=21 Full BF16 — all 8B of experts live in VRAM; single 24 GB+ NVIDIA GPU ## Guide ## Overview [LFM2.5-8B-A1B](https://huggingface.co/LiquidAI/LFM2.5-8B-A1B) is [Liquid AI's](https://www.liquid.ai/) mixture-of-experts model: ~8B total parameters with only ~1B activated per token. It pairs the **LFM2 hybrid backbone** (short-range gated convolution blocks interleaved with grouped-query attention) with sparse MoE feed-forward layers, so it reaches 8B-class quality at roughly 1B decode cost. It supports explicit `` reasoning and Pythonic tool calling through vLLM's OpenAI-compatible API. ### Key Features - **Hybrid + MoE**: Gated short convolutions + grouped-query attention with sparse expert FFNs — ~1B active parameters per token out of ~8B total. - **`` reasoning**: Emits an explicit `…` chain-of-thought on non-trivial problems; vLLM's `qwen3` parser splits it into `reasoning`. - **Tool calling**: Pythonic tool calls surfaced as OpenAI `tool_calls` by vLLM's native `lfm2` parser. - **128K context**: Long-context support (`max_position_embeddings = 128000`). - **Native vLLM support**: Served via the `Lfm2MoeForCausalLM` architecture. ### Supported Variants Dense: - `LiquidAI/LFM2.5-230M` (230M) - `LiquidAI/LFM2.5-350M` (350M) - `LiquidAI/LFM2.5-1.2B-Instruct` (1.2B) - `LiquidAI/LFM2.5-1.2B-Thinking` (1.2B, reasoning) - `LiquidAI/LFM2.5-1.2B-JP` / `LiquidAI/LFM2.5-1.2B-JP-202606` (Japanese) - `LiquidAI/LFM2.5-1.2B-Base` (pretrained base) MoE: - `LiquidAI/LFM2.5-8B-A1B` (8B total / ~1B active) Vision-Language: - `LiquidAI/LFM2.5-VL-450M`, `LiquidAI/LFM2.5-VL-1.6B` See the [LFM2.5 usage guide](../../LiquidAI/LFM2.5) for the full family. > **Sizing**: MoE keeps every expert resident in VRAM, so size the GPU for the full ~8B of > weights (≈17 GB BF16 + KV cache) even though only ~1B is active per token. ## Prerequisites - **Hardware**: 1× GPU with ≥24 GB VRAM. - **vLLM**: ≥ 0.23.0. ### pip (NVIDIA CUDA) ```bash uv venv source .venv/bin/activate uv pip install -U vllm --torch-backend auto ``` ## Deployment Configurations ### Quick Start (Single GPU, BF16) ```bash vllm serve LiquidAI/LFM2.5-8B-A1B ``` ### Full-Featured Server Launch Reasoning **and** tool calling: ```bash vllm serve LiquidAI/LFM2.5-8B-A1B \ --reasoning-parser qwen3 \ --enable-auto-tool-choice \ --tool-call-parser lfm2 \ --host 0.0.0.0 --port 8000 ``` ### Multi-GPU This model fits on a single 24 GB+ GPU, so tensor parallelism gives no meaningful speedup on basically any GPU — there's nothing to split that helps. You can still try it on a multi-GPU node if you want, but expect no throughput gain: ```bash vllm serve LiquidAI/LFM2.5-8B-A1B --tensor-parallel-size 2 ``` ### Docker (NVIDIA) ```bash docker run -itd --name lfm2.5-8b-a1b \ --ipc=host --network host --shm-size 16G --gpus all \ -v ~/.cache/huggingface:/root/.cache/huggingface \ vllm/vllm-openai:latest \ --model LiquidAI/LFM2.5-8B-A1B \ --reasoning-parser qwen3 \ --host 0.0.0.0 --port 8000 ``` ## Client Usage ### Reasoning Mode The model card recommends temperature `0.2`, `top_k 80`, `repetition_penalty 1.05`. Leave `max_tokens` unset so the `` block and answer aren't truncated mid-stream. ```python from openai import OpenAI client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY") response = client.chat.completions.create( model="LiquidAI/LFM2.5-8B-A1B", messages=[{"role": "user", "content": "A snail climbs 3 ft up a 20 ft well each day and slides 2 ft back each night. How many days to reach the top?"}], temperature=0.2, extra_body={"top_k": 80, "repetition_penalty": 1.05}, ) msg = response.choices[0].message print("reasoning:", msg.reasoning) print("answer:", msg.content) ``` > **Note**: the model opens the `` channel for non-trivial problems; a simple prompt may > be answered directly, in which case `reasoning` is empty. That is expected behavior — > the `qwen3` parser extracts the block whenever it is present. ### Tool Calling Add `--enable-auto-tool-choice --tool-call-parser lfm2` at launch, then pass `tools=[…]`; the `lfm2` parser converts the model's Pythonic call into a standard `tool_calls` array. ### Structured Outputs vLLM's guided decoding constrains output to a JSON schema via `response_format`. The model does not see the schema itself — put semantic instructions in the system prompt. ## Configuration Tips - Size the GPU for the full ~8B of weights — all experts are resident even though ~1B is active. - It fits on one GPU; tensor parallelism gives no real speedup on basically any GPU (you can still try `--tensor-parallel-size N`). - Set `--max-model-len` to match your workload (up to 128K). - `--gpu-memory-utilization 0.90–0.95` maximizes KV cache capacity. - Sampling presets are per-request client defaults — don't bake them into `vllm serve`. ## References - [Model card](https://huggingface.co/LiquidAI/LFM2.5-8B-A1B) - [LFM2.5 usage guide](../../LiquidAI/LFM2.5) - [Liquid AI](https://www.liquid.ai/) - [vLLM supported models](https://docs.vllm.ai/en/latest/models/supported_models.html) --- # LiquidAI/LFM2.5-VL-1.6B on vLLM Source: https://recipes.vllm.ai/LiquidAI/LFM2.5-VL-1.6B Provider: LiquidAI Released: 2026-01-05T19:07:50.000Z Updated: 2026-06-24 Architecture: dense Parameters: 1.6B Context length: 32,768 Min vLLM: 0.23.0 Tasks: multimodal Verified hardware: H100, H200, B200 Summary: Liquid AI's 1.6B vision-language model — LFM2 hybrid LM backbone plus a SigLIP2 vision tower for image+text chat on a single small GPU. ## Variants - default: precision=bf16, vram_minimum_gb=4 Full BF16 (LM + vision encoder) — single 8 GB+ NVIDIA GPU ## Guide ## Overview [LFM2.5-VL-1.6B](https://huggingface.co/LiquidAI/LFM2.5-VL-1.6B) is the larger vision-language model in [Liquid AI's](https://www.liquid.ai/) LFM2.5-VL family. It pairs the **LFM2 hybrid language backbone** (short-range gated convolution blocks interleaved with grouped-query attention) with a SigLIP2 vision encoder for image understanding, while remaining small enough to serve on a single commodity GPU through vLLM's OpenAI-compatible API. ### Key Features - **Vision-language**: SigLIP2 vision encoder on top of the LFM2 hybrid language backbone — single- and multi-image prompts. - **Hybrid LM backbone**: Gated short convolutions interleaved with grouped-query attention — a smaller KV cache and lower decode latency than a same-size full-attention transformer. - **32K context**: 32,768-token context window. - **Native vLLM support**: Served via the `Lfm2VlForConditionalGeneration` architecture. ### Supported Variants Vision-Language: - `LiquidAI/LFM2.5-VL-450M` (450M) - `LiquidAI/LFM2.5-VL-1.6B` (1.6B) Text (same LFM2 family): - Dense: `LiquidAI/LFM2.5-230M`, `LiquidAI/LFM2.5-350M`, `LiquidAI/LFM2.5-1.2B-Instruct`, `LiquidAI/LFM2.5-1.2B-Thinking`, `LiquidAI/LFM2.5-1.2B-JP`, `LiquidAI/LFM2.5-1.2B-JP-202606`, `LiquidAI/LFM2.5-1.2B-Base` - MoE: `LiquidAI/LFM2.5-8B-A1B` See the [LFM2.5 usage guide](../../LiquidAI/LFM2.5) for the full family. ## Prerequisites - **Hardware**: 1× GPU with ≥8 GB VRAM. - **vLLM**: ≥ 0.23.0. ### pip (NVIDIA CUDA) ```bash uv venv source .venv/bin/activate uv pip install -U vllm --torch-backend auto ``` ## Deployment Configurations ### Quick Start (Single GPU, BF16) ```bash vllm serve LiquidAI/LFM2.5-VL-1.6B ``` ### Multiple Images per Request ```bash vllm serve LiquidAI/LFM2.5-VL-1.6B \ --limit-mm-per-prompt '{"image": 4}' \ --host 0.0.0.0 --port 8000 ``` ### Docker (NVIDIA) ```bash docker run -itd --name lfm2.5-vl-1.6b \ --ipc=host --network host --shm-size 16G --gpus all \ -v ~/.cache/huggingface:/root/.cache/huggingface \ vllm/vllm-openai:latest \ --model LiquidAI/LFM2.5-VL-1.6B \ --host 0.0.0.0 --port 8000 ``` ## Client Usage ### Image Understanding Send an image + text turn via the OpenAI chat API. The card recommends temperature `0.1`, `min_p 0.15`, `repetition_penalty 1.05` (`min_p` and `repetition_penalty` ride in `extra_body`). ```python from openai import OpenAI client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY") response = client.chat.completions.create( model="LiquidAI/LFM2.5-VL-1.6B", messages=[{ "role": "user", "content": [ {"type": "image_url", "image_url": {"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"}}, {"type": "text", "text": "What is in this image?"}, ], }], temperature=0.1, extra_body={"min_p": 0.15, "repetition_penalty": 1.05}, ) print(response.choices[0].message.content) ``` ### Multiple Images Launch with `--limit-mm-per-prompt '{"image": N}'`, then include several `image_url` blocks in one message to compare or reason across images. ## Performance Tensor parallelism gives no speedup — the model fits on a single GPU. ## Configuration Tips - `--limit-mm-per-prompt '{"image": N}'` caps images per request (default 1). - Set `--max-model-len` to match your workload (up to 32K). - `--gpu-memory-utilization 0.90–0.95` maximizes KV cache capacity. - Sampling presets are per-request client defaults — don't bake them into `vllm serve`. ## References - [Model card](https://huggingface.co/LiquidAI/LFM2.5-VL-1.6B) - [LFM2.5 usage guide](../../LiquidAI/LFM2.5) - [Liquid AI](https://www.liquid.ai/) - [vLLM supported models](https://docs.vllm.ai/en/latest/models/supported_models.html) --- # LiquidAI/LFM2.5-VL-450M on vLLM Source: https://recipes.vllm.ai/LiquidAI/LFM2.5-VL-450M Provider: LiquidAI Released: 2026-04-08T14:57:35.000Z Updated: 2026-06-24 Architecture: dense Parameters: 450M Context length: 32,768 Min vLLM: 0.23.0 Tasks: multimodal Verified hardware: H100, H200, B200 Summary: Liquid AI's smallest vision-language model (450M) — LFM2 hybrid LM backbone plus a SigLIP2 vision tower for image+text chat, light enough for edge GPUs. ## Variants - default: precision=bf16, vram_minimum_gb=2 Full BF16 (LM + vision encoder) — fits essentially any GPU ## Guide ## Overview [LFM2.5-VL-450M](https://huggingface.co/LiquidAI/LFM2.5-VL-450M) is the smallest vision-language model in [Liquid AI's](https://www.liquid.ai/) LFM2.5-VL family — a SigLIP2 vision encoder on top of the **LFM2 hybrid language backbone** (short-range gated convolution blocks interleaved with grouped-query attention). It is light enough for edge and on-device image understanding, served through vLLM's OpenAI-compatible API. ### Key Features - **Vision-language**: SigLIP2 vision encoder on top of the LFM2 hybrid language backbone — single- and multi-image prompts. - **Edge-ready**: ~1 GB of BF16 weights — runs on commodity and on-device GPUs. - **Hybrid LM backbone**: Gated short convolutions interleaved with grouped-query attention — a smaller KV cache and lower decode latency than a same-size full-attention transformer. - **32K context**: 32,768-token context window. - **Native vLLM support**: Served via the `Lfm2VlForConditionalGeneration` architecture. ### Supported Variants Vision-Language: - `LiquidAI/LFM2.5-VL-450M` (450M) - `LiquidAI/LFM2.5-VL-1.6B` (1.6B) Text (same LFM2 family): - Dense: `LiquidAI/LFM2.5-230M`, `LiquidAI/LFM2.5-350M`, `LiquidAI/LFM2.5-1.2B-Instruct`, `LiquidAI/LFM2.5-1.2B-Thinking`, `LiquidAI/LFM2.5-1.2B-JP`, `LiquidAI/LFM2.5-1.2B-JP-202606`, `LiquidAI/LFM2.5-1.2B-Base` - MoE: `LiquidAI/LFM2.5-8B-A1B` See the [LFM2.5 usage guide](../../LiquidAI/LFM2.5) for the full family. ## Prerequisites - **Hardware**: 1× GPU (~1–2 GB VRAM for weights; any modern GPU works). - **vLLM**: ≥ 0.23.0. ### pip (NVIDIA CUDA) ```bash uv venv source .venv/bin/activate uv pip install -U vllm --torch-backend auto ``` ## Deployment Configurations ### Quick Start (Single GPU, BF16) ```bash vllm serve LiquidAI/LFM2.5-VL-450M ``` ### Multiple Images per Request ```bash vllm serve LiquidAI/LFM2.5-VL-450M \ --limit-mm-per-prompt '{"image": 4}' \ --host 0.0.0.0 --port 8000 ``` ### Docker (NVIDIA) ```bash docker run -itd --name lfm2.5-vl-450m \ --ipc=host --network host --shm-size 16G --gpus all \ -v ~/.cache/huggingface:/root/.cache/huggingface \ vllm/vllm-openai:latest \ --model LiquidAI/LFM2.5-VL-450M \ --host 0.0.0.0 --port 8000 ``` ## Client Usage ### Image Understanding Send an image + text turn via the OpenAI chat API. The card recommends temperature `0.1`, `min_p 0.15`, `repetition_penalty 1.05` (`min_p` and `repetition_penalty` ride in `extra_body`). ```python from openai import OpenAI client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY") response = client.chat.completions.create( model="LiquidAI/LFM2.5-VL-450M", messages=[{ "role": "user", "content": [ {"type": "image_url", "image_url": {"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"}}, {"type": "text", "text": "What is in this image?"}, ], }], temperature=0.1, extra_body={"min_p": 0.15, "repetition_penalty": 1.05}, ) print(response.choices[0].message.content) ``` ### Multiple Images Launch with `--limit-mm-per-prompt '{"image": N}'`, then include several `image_url` blocks in one message to compare or reason across images. ## Configuration Tips - At 450M the model fits any GPU; ideal for edge / on-device image understanding. - `--limit-mm-per-prompt '{"image": N}'` caps images per request (default 1). - Set `--max-model-len` to match your workload (up to 32K). - Sampling presets are per-request client defaults — don't bake them into `vllm serve`. ## References - [Model card](https://huggingface.co/LiquidAI/LFM2.5-VL-450M) - [LFM2.5 usage guide](../../LiquidAI/LFM2.5) - [Liquid AI](https://www.liquid.ai/) - [vLLM supported models](https://docs.vllm.ai/en/latest/models/supported_models.html) --- # poolside/Laguna-M.1 on vLLM Source: https://recipes.vllm.ai/poolside/Laguna-M.1 Provider: Poolside Released: 2026-06-15T09:17:53.000Z Updated: 2026-06-18 Architecture: moe Parameters: 225B Context length: 262,144 Min vLLM: 0.21.0 Tasks: text Summary: Poolside's 225B total / 23B activated MoE coding model with global attention, native interleaved reasoning, and 256K context — designed for agentic coding and long-horizon work. ## Variants - default: precision=bf16, vram_minimum_gb=540 BF16 weights (225B) — fits on a single 8×H200/B200 node - fp8: precision=fp8, vram_minimum_gb=270, model_id=poolside/Laguna-M.1-FP8 Block-FP8 weights + FP8 KV cache — halves VRAM, runs on a single 4×H200/B200 node - nvfp4: precision=nvfp4, vram_minimum_gb=135, model_id=poolside/Laguna-M.1-NVFP4 NVFP4 weights (Blackwell only) — fits within ~135GB; pairs with FP8 KV cache ## Guide ## Overview [Laguna M.1](https://huggingface.co/poolside/Laguna-M.1) is Poolside's 225B-total / 23B-activated Mixture-of-Experts model purpose-built for agentic coding and long-horizon work. It is competitive with state-of-the-art open-weight and frontier models on SWE-bench Verified (74.6%), SWE-bench Multilingual, SWE-Bench Pro, and Terminal-Bench 2.0. ### Key features - **Large sparse MoE**: 70-layer transformer — the first 3 layers are dense SwiGLU, the remaining 67 are sparse MoE with **256 experts + 1 shared expert** and top-k=16 routing with auxiliary-loss-free load balancing. - **Global attention**: global attention across all layers with 64 Q-heads, 8 KV-heads, head dim 128, and softplus attention output gating. - **256K context**: RoPE with YaRN extends the context window to 262,144 tokens. - **Native interleaved reasoning**: thinking blocks emitted before and between tool calls, with *preserved thinking* (returning prior `reasoning` in history) recommended for best agentic performance. Toggled per-request via `enable_thinking`. - **Tool calling**: Poolside-specific tool-call protocol, parsed via `poolside_v1`. ## Prerequisites Laguna M.1 support landed in vLLM via [PR #41129](https://github.com/vllm-project/vllm/pull/41129) (shared implementation with [Laguna XS.2](https://huggingface.co/poolside/Laguna-XS.2)) and ships in **vLLM 0.21.0 and later**. ```bash uv pip install -U 'vllm>=0.21.0' ``` FP8 and NVFP4 quantized checkpoints are available at [Laguna-M.1-FP8](https://huggingface.co/poolside/Laguna-M.1-FP8) and [Laguna-M.1-NVFP4](https://huggingface.co/poolside/Laguna-M.1-NVFP4). Quantization is detected automatically from the checkpoint's `quantization_config`, so the launch command is identical — just swap the model ID (select the variant above). ## Launch command ### Single node (8×H200, BF16) ```bash vllm serve poolside/Laguna-M.1 \ --tensor-parallel-size 8 \ --served-model-name laguna \ --enable-auto-tool-choice \ --tool-call-parser poolside_v1 \ --reasoning-parser poolside_v1 \ --default-chat-template-kwargs '{"enable_thinking": true}' ``` The BF16 checkpoint (~450 GB of weights) fits on a single 8×H200 node. Use the **FP8** variant to run on 4 GPUs, or the **NVFP4** variant (Blackwell only) to fit within ~135 GB. ## Controlling reasoning Reasoning is enabled at the server when you pass `--default-chat-template-kwargs '{"enable_thinking": true}'` (included above). Poolside recommends **preserved thinking** for agentic coding — return the previous assistant turn's `reasoning` in the message history. ```python from openai import OpenAI client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY") resp = client.chat.completions.create( model="laguna", messages=[{"role": "user", "content": "Write a Python retry wrapper with exponential backoff."}], temperature=1.0, extra_query={"top_k": 20}, stream=True, ) for chunk in resp: delta = chunk.choices[0].delta rc = delta.reasoning if rc: print(rc, end="") if delta.content: print(delta.content, end="") ``` Recommended sampling (matching Poolside's benchmark runs): **temperature=1.0, top_k=20**, thinking enabled. To disable thinking, omit `--default-chat-template-kwargs` at launch, or pass `extra_body={"chat_template_kwargs": {"enable_thinking": False}}` per request. ## References - [Model card](https://huggingface.co/poolside/Laguna-M.1) - [Release blog post](https://poolside.ai/blog/laguna-a-deeper-dive) - [Technical report](https://poolside.ai/assets/laguna/laguna-m1-xs2-technical-report.pdf) - [vLLM support PR #41129](https://github.com/vllm-project/vllm/pull/41129) --- # OpenMOSS-Team/MOSS-SoundEffect on vLLM Source: https://recipes.vllm.ai/OpenMOSS-Team/MOSS-SoundEffect Provider: OpenMOSS Released: 2026-02-08T17:44:45.000Z Updated: 2026-06-15 Architecture: dense Parameters: 8B Min vLLM: 0.22.0 Tasks: omni Summary: OpenMOSS's 8B sound-effect generation model — environmental, urban, biological, human-action and musical sounds with controllable duration, no reference audio — served via vLLM-Omni through the OpenAI /v1/audio/speech API (24 kHz mono). ## Variants - default: precision=bf16, vram_minimum_gb=32 MOSS-SoundEffect 8B (MossTTSDelay, n_vq=16) — text-to-sound-effect ## Recommended base flags ```bash vllm serve 'OpenMOSS-Team/MOSS-SoundEffect' \ --omni \ --deploy-config \ vllm_omni/deploy/moss_sound_effect.yaml ``` ## Guide ## Overview [MOSS-SoundEffect](https://huggingface.co/OpenMOSS-Team/MOSS-SoundEffect) is the **sound-effect generation** member of OpenMOSS's MOSS-TTS Family, served through **vLLM-Omni** with the OpenAI-compatible `/v1/audio/speech` API. It generates audio for natural environments, urban scenes, biological sounds, human actions, and musical fragments — suitable for film, games, and interactive experiences. Output is **24 kHz mono**. Unlike the speech members of the family, MOSS-SoundEffect takes **no reference audio**: the `input` field is a text description of the sound (mapped to the upstream `ambient_sound` parameter). Generation runs at ~12.5 tokens/second; longer descriptions (or a higher token budget) produce longer audio. ## Prerequisites - **Hardware**: a single CUDA GPU comparable to the 8B MOSS-TTS profile (~18 GB talker + ~8 GB codec). - **vLLM-Omni** targeting vLLM >= 0.22. ## Installation ```bash uv venv && source .venv/bin/activate uv pip install git+https://github.com/vllm-project/vllm-omni.git ``` ## Launch the server MOSS-SoundEffect shares `model_type=moss_tts_delay`, so pass its deploy config explicitly: ```bash vllm serve OpenMOSS-Team/MOSS-SoundEffect --omni \ --deploy-config vllm_omni/deploy/moss_sound_effect.yaml --port 8000 ``` ## Client usage ```bash curl -X POST http://localhost:8000/v1/audio/speech \ -H "Content-Type: application/json" \ -d '{ "model": "OpenMOSS-Team/MOSS-SoundEffect", "input": "Thunder rumbling, rain pattering on a tin roof.", "response_format": "wav" }' --output thunder.wav ``` ## Known limitations - Output is **24 kHz mono**. - No `ref_audio` is accepted; `input` is a text description of the sound. ## References - [Model card](https://huggingface.co/OpenMOSS-Team/MOSS-SoundEffect) - [MOSS-TTS Family](https://huggingface.co/OpenMOSS-Team/MOSS-TTS) · [GitHub](https://github.com/OpenMOSS/MOSS-TTS) --- # OpenMOSS-Team/MOSS-TTS-Realtime on vLLM Source: https://recipes.vllm.ai/OpenMOSS-Team/MOSS-TTS-Realtime Provider: OpenMOSS Released: 2026-02-09T02:41:24.000Z Updated: 2026-06-15 Architecture: dense Parameters: 1.7B Min vLLM: 0.22.0 Tasks: omni Summary: OpenMOSS's 1.7B real-time streaming TTS for low-latency voice agents (TTFB ~180 ms) — multi-turn context-aware incremental synthesis — served via vLLM-Omni through the OpenAI /v1/audio/speech API (24 kHz mono). ## Variants - default: precision=bf16, vram_minimum_gb=16 MOSS-TTS-Realtime 1.7B (MossTTSRealtime) — lowest-latency streaming variant (reference run on 1x A10G 24 GB) ## Recommended base flags ```bash vllm serve 'OpenMOSS-Team/MOSS-TTS-Realtime' \ --omni ``` ## Guide ## Overview [MOSS-TTS-Realtime](https://huggingface.co/OpenMOSS-Team/MOSS-TTS-Realtime) is the **1.7B real-time streaming** member of OpenMOSS's MOSS-TTS Family, served through **vLLM-Omni** with the OpenAI-compatible `/v1/audio/speech` API. It is a multi-turn, context-aware model for **low-latency voice agents**, using a local depth transformer (no delay warm-up) for incremental synthesis. TTFB (time to first byte) is **~180 ms**, and LLM-first-sentence + TTS-TTFB is ~377 ms. Output is **24 kHz mono**. ## Prerequisites - **Hardware**: a single CUDA GPU. Reference run: 1x A10G 24 GB — ~6 GB for the 1.7B talker + ~8 GB for the codec decoder. - **vLLM-Omni** targeting vLLM >= 0.22. ## Installation ```bash uv venv && source .venv/bin/activate uv pip install git+https://github.com/vllm-project/vllm-omni.git ``` ## Launch the server MOSS-TTS-Realtime uses `model_type=moss_tts_realtime`, so its deploy config (`vllm_omni/deploy/moss_tts_realtime.yaml`, `codec_chunk_frames: 15` for low TTFA) is auto-discovered: ```bash vllm serve OpenMOSS-Team/MOSS-TTS-Realtime --omni --port 8000 ``` ## Client usage ### Streaming voice clone (curl) ```bash curl -X POST http://localhost:8000/v1/audio/speech \ -H "Content-Type: application/json" \ -d '{ "model": "OpenMOSS-Team/MOSS-TTS-Realtime", "input": "This is a low-latency streaming TTS test.", "voice": "default", "ref_audio": "https://raw.githubusercontent.com/OpenMOSS/MOSS-TTS/main/assets/audio/zh_1.wav", "response_format": "wav", "stream": true }' --output output.wav ``` ## Known limitations - Output is **24 kHz mono**. - Shares the `OpenMOSS-Team/MOSS-Audio-Tokenizer` codec (~7 GB, auto-downloaded; override with `MOSS_TTS_CODEC_PATH`). ## References - [Model card](https://huggingface.co/OpenMOSS-Team/MOSS-TTS-Realtime) - [MOSS-TTS Family](https://huggingface.co/OpenMOSS-Team/MOSS-TTS) · [GitHub](https://github.com/OpenMOSS/MOSS-TTS) --- # OpenMOSS-Team/MOSS-TTS on vLLM Source: https://recipes.vllm.ai/OpenMOSS-Team/MOSS-TTS Provider: OpenMOSS Released: 2026-02-06T04:14:06.000Z Updated: 2026-06-15 Architecture: dense Parameters: 8B Min vLLM: 0.22.0 Tasks: omni Summary: Flagship 8B model of the OpenMOSS MOSS-TTS Family — high-fidelity zero-shot voice cloning, ultra-long stable speech, token-level duration and phoneme control, 20-language code-switched synthesis — served via vLLM-Omni through the OpenAI /v1/audio/speech API (24 kHz mono). ## Variants - default: precision=bf16, vram_minimum_gb=32 MOSS-TTS 8B (MossTTSDelay) — flagship production TTS (reference run on 1x H100 80 GB) ## Recommended base flags ```bash vllm serve 'OpenMOSS-Team/MOSS-TTS' \ --omni \ --deploy-config \ vllm_omni/deploy/moss_tts.yaml ``` ## Guide ## Overview [MOSS-TTS](https://huggingface.co/OpenMOSS-Team/MOSS-TTS) is the **flagship 8B model** of OpenMOSS's [MOSS-TTS Family](https://huggingface.co/OpenMOSS-Team/MOSS-TTS), a production-grade TTS foundation model served through **vLLM-Omni** with the OpenAI-compatible `/v1/audio/speech` API. It delivers high-fidelity zero-shot voice cloning, **ultra-long stable speech (up to ~1 hour)**, token-level duration control, Pinyin/IPA phoneme control, and **20-language** / code-switched synthesis. Output is **24 kHz mono** via the shared `OpenMOSS-Team/MOSS-Audio-Tokenizer` codec (~7 GB, auto-downloaded; override with `MOSS_TTS_CODEC_PATH`). The MOSS-TTS Family ships five production-ready models, each with its own recipe: **MOSS-TTS** (this page, flagship TTS), **MOSS-TTS-Realtime** (low-latency streaming), **MOSS-TTSD** (multi-speaker dialogue), **MOSS-VoiceGenerator** (zero-shot voice design), and **MOSS-SoundEffect** (sound-effect synthesis). MOSS-TTS is released in two architectures: the 8B `MossTTSDelay` (this checkpoint, production-recommended) and a 1.7B `MossTTSLocal` (`OpenMOSS-Team/MOSS-TTS-Local-Transformer`) tuned for streaming-oriented systems. A `MOSS-TTS-v1.5` point release adds 31-language support and steadier cloning with `[pause Xs]` markers (same serving path; set `language` for best results). ## Prerequisites - **Hardware**: a single CUDA GPU. Reference run: 1x H100 80 GB — ~18 GB for the 8B talker + ~8 GB for the codec decoder on the same device (`gpu_memory_utilization: 0.85` in `moss_tts.yaml`). - **vLLM-Omni** targeting vLLM >= 0.22. ## Installation ```bash uv venv && source .venv/bin/activate uv pip install git+https://github.com/vllm-project/vllm-omni.git ``` ## Launch the server All MossTTSDelay checkpoints (MOSS-TTS, MOSS-TTSD, MOSS-SoundEffect, MOSS-VoiceGenerator) share `model_type=moss_tts_delay`, so the deploy config must be passed explicitly to select the right pipeline: ```bash vllm serve OpenMOSS-Team/MOSS-TTS --omni \ --deploy-config vllm_omni/deploy/moss_tts.yaml --port 8000 ``` ## Client usage ### Voice cloning (curl) ```bash curl -X POST http://localhost:8000/v1/audio/speech \ -H "Content-Type: application/json" \ -d '{ "model": "OpenMOSS-Team/MOSS-TTS", "input": "Hello, this is a voice cloning test.", "voice": "default", "ref_audio": "https://raw.githubusercontent.com/OpenMOSS/MOSS-TTS/main/assets/audio/zh_1.wav", "response_format": "wav" }' --output output.wav ``` ## Known limitations - Output is **24 kHz mono**. - `MOSS_TTS_CODEC_PATH` overrides the codec checkpoint location if you have a local copy. ## References - [MOSS-TTS model card](https://huggingface.co/OpenMOSS-Team/MOSS-TTS) - [MOSS-TTS Family](https://huggingface.co/OpenMOSS-Team/MOSS-TTS) · [GitHub](https://github.com/OpenMOSS/MOSS-TTS) - [vLLM-Omni MOSS-TTS offline example](https://github.com/vllm-project/vllm-omni/tree/main/examples/offline_inference/text_to_speech/moss_tts) --- # OpenMOSS-Team/MOSS-TTSD-v1.0 on vLLM Source: https://recipes.vllm.ai/OpenMOSS-Team/MOSS-TTSD-v1.0 Provider: OpenMOSS Released: 2026-02-08T14:41:48.000Z Updated: 2026-06-15 Architecture: dense Parameters: 8B Min vLLM: 0.22.0 Tasks: omni Summary: OpenMOSS's 8B spoken-dialogue generation model for expressive, multi-speaker, ultra-long conversations — served via vLLM-Omni through the OpenAI /v1/audio/speech API (24 kHz mono). ## Variants - default: precision=bf16, vram_minimum_gb=32 MOSS-TTSD-v1.0 8B (MossTTSDelay, n_vq=16) — multi-speaker dialogue TTS ## Recommended base flags ```bash vllm serve 'OpenMOSS-Team/MOSS-TTSD-v1.0' \ --omni \ --deploy-config \ vllm_omni/deploy/moss_ttsd.yaml ``` ## Guide ## Overview [MOSS-TTSD-v1.0](https://huggingface.co/OpenMOSS-Team/MOSS-TTSD-v1.0) is the **spoken-dialogue** member of OpenMOSS's MOSS-TTS Family, served through **vLLM-Omni** with the OpenAI-compatible `/v1/audio/speech` API. It generates expressive, **multi-speaker, ultra-long dialogues** and (per OpenMOSS) leads on objective metrics while outperforming top closed-source systems in subjective evaluations. Output is **24 kHz mono**. Dialogue formatting (speaker turns, e.g. `[S1] ... [S2] ...`) and any multi-speaker reference conditioning follow the upstream [MOSS-TTSD](https://github.com/OpenMOSS/MOSS-TTSD) conventions — consult the upstream repo for the exact turn/reference schema. ## Prerequisites - **Hardware**: a single CUDA GPU comparable to the 8B MOSS-TTS profile (~18 GB talker + ~8 GB codec; e.g. an 80 GB H100 with headroom for long dialogues). - **vLLM-Omni** targeting vLLM >= 0.22. ## Installation ```bash uv venv && source .venv/bin/activate uv pip install git+https://github.com/vllm-project/vllm-omni.git ``` ## Launch the server MOSS-TTSD shares `model_type=moss_tts_delay` with the other MossTTSDelay checkpoints, so pass its deploy config explicitly: ```bash vllm serve OpenMOSS-Team/MOSS-TTSD-v1.0 --omni \ --deploy-config vllm_omni/deploy/moss_ttsd.yaml --port 8000 ``` ## Client usage ```bash curl -X POST http://localhost:8000/v1/audio/speech \ -H "Content-Type: application/json" \ -d '{ "model": "OpenMOSS-Team/MOSS-TTSD-v1.0", "input": "[S1] Hi, how was your day? [S2] Pretty good, thanks for asking!", "voice": "default", "response_format": "wav" }' --output dialogue.wav ``` ## Known limitations - Output is **24 kHz mono**. - Shares the `OpenMOSS-Team/MOSS-Audio-Tokenizer` codec (~7 GB, auto-downloaded; override with `MOSS_TTS_CODEC_PATH`). ## References - [Model card](https://huggingface.co/OpenMOSS-Team/MOSS-TTSD-v1.0) - [MOSS-TTSD GitHub](https://github.com/OpenMOSS/MOSS-TTSD) - [MOSS-TTS Family](https://huggingface.co/OpenMOSS-Team/MOSS-TTS) --- # OpenMOSS-Team/MOSS-VoiceGenerator on vLLM Source: https://recipes.vllm.ai/OpenMOSS-Team/MOSS-VoiceGenerator Provider: OpenMOSS Released: 2026-02-08T15:23:36.000Z Updated: 2026-06-15 Architecture: dense Parameters: 1.7B Min vLLM: 0.22.0 Tasks: omni Summary: OpenMOSS's 1.7B zero-shot voice-design model — generate diverse voices and styles directly from a text prompt with no reference speech — served via vLLM-Omni through the OpenAI /v1/audio/speech API (24 kHz mono). ## Variants - default: precision=bf16, vram_minimum_gb=16 MOSS-VoiceGenerator 1.7B (MossTTSDelay, n_vq=16) — zero-shot voice design ## Recommended base flags ```bash vllm serve 'OpenMOSS-Team/MOSS-VoiceGenerator' \ --omni \ --deploy-config \ vllm_omni/deploy/moss_voice_generator.yaml ``` ## Guide ## Overview [MOSS-VoiceGenerator](https://huggingface.co/OpenMOSS-Team/MOSS-VoiceGenerator) is the **voice-design** member of OpenMOSS's MOSS-TTS Family, served through **vLLM-Omni** with the OpenAI-compatible `/v1/audio/speech` API. It generates diverse voices and styles **directly from a text prompt, with no reference speech**, unifying voice design, style control, and synthesis. It can run standalone or as a design layer that produces a voice for the downstream TTS members of the family. Output is **24 kHz mono**. ## Prerequisites - **Hardware**: a single CUDA GPU. ~6 GB for the 1.7B talker + ~8 GB for the codec decoder (comparable to MOSS-TTS-Realtime). - **vLLM-Omni** targeting vLLM >= 0.22. ## Installation ```bash uv venv && source .venv/bin/activate uv pip install git+https://github.com/vllm-project/vllm-omni.git ``` ## Launch the server MOSS-VoiceGenerator shares `model_type=moss_tts_delay`, so pass its deploy config explicitly: ```bash vllm serve OpenMOSS-Team/MOSS-VoiceGenerator --omni \ --deploy-config vllm_omni/deploy/moss_voice_generator.yaml --port 8000 ``` ## Client usage ```bash curl -X POST http://localhost:8000/v1/audio/speech \ -H "Content-Type: application/json" \ -d '{ "model": "OpenMOSS-Team/MOSS-VoiceGenerator", "input": "Hello, this voice was designed entirely from a text prompt.", "instructions": "A warm, friendly young female voice with a calm pace.", "response_format": "wav" }' --output designed.wav ``` ## Known limitations - Output is **24 kHz mono**. - No reference speech is used — voice identity comes from the text prompt / `instructions`. The exact prompt schema follows the upstream model card. ## References - [Model card](https://huggingface.co/OpenMOSS-Team/MOSS-VoiceGenerator) - [MOSS-TTS Family](https://huggingface.co/OpenMOSS-Team/MOSS-TTS) · [GitHub](https://github.com/OpenMOSS/MOSS-TTS) --- # Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice on vLLM Source: https://recipes.vllm.ai/Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice Provider: Qwen Released: 2026-01-21T08:56:49.000Z Updated: 2026-06-15 Architecture: dense Parameters: 1.7B Min vLLM: 0.20.0 Tasks: omni Summary: Text-to-speech served via vLLM-Omni with predefined speaker voices and optional style/emotion control, exposed through the OpenAI /v1/audio/speech API. Sibling VoiceDesign and Base (voice-clone) checkpoints share the same serving path. ## Variants - default: precision=bf16, vram_minimum_gb=16 CustomVoice 1.7B — predefined speaker voices + optional style/emotion control - voicedesign-1.7b: precision=bf16, vram_minimum_gb=16, model_id=Qwen/Qwen3-TTS-12Hz-1.7B-VoiceDesign VoiceDesign 1.7B — generate a voice from a natural-language description - base-1.7b: precision=bf16, vram_minimum_gb=16, model_id=Qwen/Qwen3-TTS-12Hz-1.7B-Base Base 1.7B — voice cloning from reference audio + transcript - customvoice-0.6b: precision=bf16, vram_minimum_gb=8, model_id=Qwen/Qwen3-TTS-12Hz-0.6B-CustomVoice CustomVoice 0.6B — smaller, lower-VRAM predefined voices - base-0.6b: precision=bf16, vram_minimum_gb=8, model_id=Qwen/Qwen3-TTS-12Hz-0.6B-Base Base 0.6B — smaller voice-clone checkpoint ## Recommended base flags ```bash vllm serve 'Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice' \ --omni ``` ## Guide ## Overview [Qwen3-TTS](https://huggingface.co/Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice) is a text-to-speech family served through **vLLM-Omni** with the OpenAI-compatible `/v1/audio/speech` API. Three task types are each backed by a dedicated checkpoint: | Task type | Model | Use case | |-------------|---------------------------------------------|---------------------------------------------------| | CustomVoice | `Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice` | Predefined speaker voices + optional style/emotion | | VoiceDesign | `Qwen/Qwen3-TTS-12Hz-1.7B-VoiceDesign` | Speech from a natural-language voice description | | Base | `Qwen/Qwen3-TTS-12Hz-1.7B-Base` | Voice cloning from reference audio + transcript | Smaller 0.6B variants are also published for CustomVoice and Base. The server serves **one** checkpoint at a time — switch task type by restarting with the matching model. ## Prerequisites - **Hardware**: a single CUDA GPU (the 1.7B talker + codec decoder share one device; the bundled deploy config requests `gpu_memory_utilization: 0.3` per stage). - **vLLM-Omni** targeting vLLM >= 0.20. See the [installation guide](https://docs.vllm.ai/projects/vllm-omni/en/latest/getting_started/installation/gpu/). Use the **Variant** selector above to switch checkpoint (CustomVoice / VoiceDesign / Base, 1.7B or 0.6B); the served `model_id` updates accordingly. ## Installation ```bash uv venv && source .venv/bin/activate uv pip install git+https://github.com/vllm-project/vllm-omni.git ``` ## Launch the server The two-stage deploy config (`vllm_omni/deploy/qwen3_tts.yaml`, async chunking on for low first-audio latency) is auto-discovered from `model_type=qwen3_tts`: ```bash # CustomVoice (predefined speakers) vllm serve Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice --omni --port 8000 # VoiceDesign (natural-language voice description) vllm serve Qwen/Qwen3-TTS-12Hz-1.7B-VoiceDesign --omni --port 8000 # Base (voice cloning from reference audio) vllm serve Qwen/Qwen3-TTS-12Hz-1.7B-Base --omni --port 8000 ``` > `--omni` is what selects the Omni pipeline, so it is required. The > `vllm-omni serve` console-script is equivalent only when `--omni` is passed > to it as well; without the flag it falls through to stock `vllm serve`. ## Client usage ### Predefined voice (curl) ```bash curl -X POST http://localhost:8000/v1/audio/speech \ -H "Content-Type: application/json" \ -d '{"model": "Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice", "input": "Hello, how are you?", "voice": "vivian", "language": "English"}' \ --output output.wav ``` ### Emotion / style instruction ```bash curl -X POST http://localhost:8000/v1/audio/speech \ -H "Content-Type: application/json" \ -d '{"model": "Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice", "input": "I am so excited!", "voice": "vivian", "instructions": "Speak with great enthusiasm"}' \ --output excited.wav ``` ### List available voices ```bash curl http://localhost:8000/v1/audio/voices ``` ### Streaming PCM (low latency) ```bash curl -X POST http://localhost:8000/v1/audio/speech \ -H "Content-Type: application/json" \ -d '{"model": "Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice", "input": "Hello, how are you?", "voice": "vivian", "language": "English", "stream": true, "response_format": "pcm"}' \ --no-buffer | play -t raw -r 24000 -e signed -b 16 -c 1 - ``` Streaming requires `stream: true` with `response_format: "pcm"` (24 kHz mono). ### Python client and offline inference ```bash # Online client (CustomVoice / VoiceDesign / Base) python examples/online_serving/text_to_speech/qwen3_tts/openai_speech_client.py \ --text "Hello, how are you?" --speaker vivian --language English # Offline (no server) python examples/offline_inference/text_to_speech/qwen3_tts/end2end.py \ --query-type CustomVoice [--streaming] ``` ## Known limitations - Each task type requires its matching checkpoint — a CustomVoice model cannot serve a Base (voice-clone) request. - One model variant is served per process; switching task type means restarting. ## References - [Model card](https://huggingface.co/Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice) - [vLLM-Omni Qwen3-TTS online example](https://github.com/vllm-project/vllm-omni/tree/main/examples/online_serving/text_to_speech/qwen3_tts) - [vLLM-Omni Qwen3-TTS offline example](https://github.com/vllm-project/vllm-omni/tree/main/examples/offline_inference/text_to_speech/qwen3_tts) --- # bosonai/higgs-audio-v3-tts-4b on vLLM Source: https://recipes.vllm.ai/bosonai/higgs-audio-v3-tts-4b Provider: Boson AI Released: 2026-06-04T05:37:32.000Z Updated: 2026-06-15 Architecture: dense Parameters: 4B Min vLLM: 0.22.0 Tasks: omni Summary: Boson AI's ~4B Qwen3-backbone text-to-speech model served via vLLM-Omni — 24 kHz speech, 100+ languages, zero-shot voice cloning with inline emotion/style/prosody control tokens — through the OpenAI /v1/audio/speech API. ## Variants - default: precision=bf16, vram_minimum_gb=24 BF16 weights — single GPU (reference run on 1x H100 80 GB; ~4B talker + codec decoder share the device) ## Recommended base flags ```bash vllm serve 'bosonai/higgs-audio-v3-tts-4b' \ --omni \ --trust-remote-code ``` ## Guide ## Overview [Higgs Audio V3 TTS](https://huggingface.co/bosonai/higgs-audio-v3-tts-4b) is Boson AI's multilingual text-to-speech model served through **vLLM-Omni** with the OpenAI-compatible `/v1/audio/speech` API. It generates **24 kHz** speech, supports zero-shot voice cloning from a reference clip, and handles **100+ languages** with inline control tokens for emotion, style, and prosody. The architecture is a ~4B Qwen3 backbone with a fused multi-codebook embedding/head (8 codebooks × 1026 vocab, MusicGen-style delay pattern). ## Prerequisites - **Hardware**: a single CUDA GPU. Reference run: 1x H100 80 GB — Stage 0 (talker, ~4B) ~60% GPU memory, Stage 1 (codec decoder) ~25%. - **vLLM-Omni** targeting vLLM >= 0.22. - `--trust-remote-code` is required. ## Installation ```bash uv venv && source .venv/bin/activate uv pip install git+https://github.com/vllm-project/vllm-omni.git ``` ## Launch the server The deploy config (`vllm_omni/deploy/higgs_multimodal_qwen3.yaml`) is auto-discovered from `model_type=higgs_multimodal_qwen3`: ```bash vllm serve bosonai/higgs-audio-v3-tts-4b --omni --trust-remote-code \ --host 0.0.0.0 --port 8000 ``` ## Client usage ### Basic TTS (curl) ```bash curl -X POST http://localhost:8000/v1/audio/speech \ -H "Content-Type: application/json" \ -d '{"model": "bosonai/higgs-audio-v3-tts-4b", "input": "Hello, how are you?"}' \ --output hello.wav ``` ### Voice clone (Python client) ```bash python examples/online_serving/text_to_speech/higgs_audio_v3/batch_speech_client.py \ --base-url http://localhost:8000 \ --model bosonai/higgs-audio-v3-tts-4b \ --ref-audio path/to/reference.wav \ --ref-text "Transcript of the reference." \ --prompts "Text to clone." ``` ### Offline batch inference ```bash python examples/offline_inference/text_to_speech/higgs_audio_v3/end2end.py \ --texts "Hello world." "The quick brown fox jumps over the lazy dog." \ --output-dir results/higgs_v3_wavs ``` `ref_audio` accepts WAV/FLAC/MP3; `ref_text` is optional but improves fidelity. ## Known limitations - Output is **24 kHz mono** WAV. - The deploy config runs both stages with `enforce_eager: true` (`max_num_seqs=16`). A CUDA graph is available for Stage 0 and helps batch=1 (~-13% RTF) but not batch>1 (the model-owned sampler runs outside the graph); Stage 1 (code2wav) must stay eager (`@torch.inference_mode` is incompatible with graph capture). - Async (chunk-based) streaming is not yet implemented; the pipeline is sync-only. ## References - [Model card](https://huggingface.co/bosonai/higgs-audio-v3-tts-4b) - [vLLM-Omni Higgs Audio V3 online example](https://github.com/vllm-project/vllm-omni/tree/main/examples/online_serving/text_to_speech/higgs_audio_v3) - [vLLM-Omni Higgs Audio V3 offline example](https://github.com/vllm-project/vllm-omni/blob/main/examples/offline_inference/text_to_speech/higgs_audio_v3/end2end.py) --- # fishaudio/s2-pro on vLLM Source: https://recipes.vllm.ai/fishaudio/s2-pro Provider: Fish Audio Released: 2026-03-09T05:48:58.000Z Updated: 2026-06-15 Architecture: dense Parameters: 4.6B Min vLLM: 0.19.0 Tasks: omni Summary: Fish Audio's dual-AR text-to-speech model served via vLLM-Omni, producing 44.1 kHz mono audio with optional voice cloning, through the OpenAI /v1/audio/speech API. ## Variants - default: precision=bf16, vram_minimum_gb=48 BF16 weights — single GPU (model loads ~48.3 GiB; reference run on 1x A800 80 GB) ## Recommended base flags ```bash vllm serve 'fishaudio/s2-pro' \ --omni ``` ## Guide ## Overview [Fish Speech S2 Pro](https://huggingface.co/fishaudio/s2-pro) is Fish Audio's dual-AR text-to-speech model, served through **vLLM-Omni** with the OpenAI-compatible `/v1/audio/speech` API. It produces **44.1 kHz mono** audio and supports zero-shot voice cloning from a reference clip + its transcript. ## Prerequisites - **Hardware**: a single CUDA GPU. Reference run: 1x A800 80 GB — the model loads at ~48.3 GiB and peaks ~48.9 GiB during inference. A two-GPU profile (`fish_qwen3_omni_2gpu.yaml`) is available for higher concurrency. - **vLLM-Omni** targeting vLLM >= 0.19. - **`fish-speech`** for the DAC codec. It depends on `pyaudio`, which needs PortAudio system libraries: ```bash # Ubuntu/Debian sudo apt-get update && sudo apt-get install -y libportaudio2 portaudio19-dev ``` ## Installation ```bash uv venv && source .venv/bin/activate uv pip install git+https://github.com/vllm-project/vllm-omni.git uv pip install fish-speech ``` ### Kvcache attention fast path S2 Pro uses a Triton decode-only kvcache-attention fast path by default on CUDA builds. Toggle it with `VLLM_OMNI_FISH_KVCACHE_ATTN`: ```bash # Verify availability python -c "from vllm_omni.attention import fish_kvcache_attn; print(fish_kvcache_attn.is_available()); print(fish_kvcache_attn.load_error())" export VLLM_OMNI_FISH_KVCACHE_ATTN=0 # disable the fast path # VLLM_OMNI_FISH_KVCACHE_ATTN=required # fail fast if it can't load ``` ## Launch the server The deploy config (`vllm_omni/deploy/fish_qwen3_omni.yaml`) is auto-discovered from `model_type=fish_qwen3_omni`: ```bash # Single GPU vllm serve fishaudio/s2-pro --omni --port 8000 ``` ```bash # Two GPUs (Stage0 Slow/Fast AR on GPU 0, Stage1 DAC decoder on GPU 1) CUDA_VISIBLE_DEVICES=0,1 vllm serve fishaudio/s2-pro --omni --port 8000 \ --deploy-config vllm_omni/deploy/fish_qwen3_omni_2gpu.yaml ``` ## Client usage ### Basic TTS ```bash curl -X POST http://localhost:8000/v1/audio/speech \ -H "Content-Type: application/json" \ -d '{"model": "fishaudio/s2-pro", "input": "Hello, how are you?", "voice": "default", "response_format": "wav"}' \ --output output.wav ``` ### Voice cloning (reference audio + transcript) ```bash curl -X POST http://localhost:8000/v1/audio/speech \ -H "Content-Type: application/json" \ -d '{ "model": "fishaudio/s2-pro", "input": "Hello, this is a cloned voice.", "voice": "default", "ref_audio": "https://example.com/reference.wav", "ref_text": "Transcript of the reference audio." }' --output cloned.wav ``` ## Known limitations - Output is **44.1 kHz mono** WAV. - Voice cloning requires **both** `ref_audio` and `ref_text`. ## References - [Model card](https://huggingface.co/fishaudio/s2-pro) - [vLLM-Omni Fish Speech example](https://github.com/vllm-project/vllm-omni/tree/main/examples/online_serving/text_to_speech/fish_speech) - Benchmarks: vllm-project/vllm-omni [#2515](https://github.com/vllm-project/vllm-omni/pull/2515) (H100x2), [#3323](https://github.com/vllm-project/vllm-omni/pull/3323) (H20x2) --- # inclusionAI/Ming-omni-tts-0.5B on vLLM Source: https://recipes.vllm.ai/inclusionAI/Ming-omni-tts-0.5B Provider: inclusionAI Released: 2026-02-11T03:10:23.000Z Updated: 2026-06-15 Architecture: dense Parameters: 0.5B Min vLLM: 0.22.0 Tasks: omni Summary: inclusionAI's dense 0.5B-LM text-to-speech model served via vLLM-Omni with style, dialect, voice-cloning and multi-speaker controls, through the OpenAI /v1/audio/speech API (44.1 kHz mono). ## Variants - default: precision=bf16, vram_minimum_gb=16 BF16 weights — 0.5B LM backbone (~1.6B total incl. audio encoder + vocoder); fits a small single GPU ## Recommended base flags ```bash vllm serve 'inclusionAI/Ming-omni-tts-0.5B' \ --omni \ --enforce-eager ``` ## Guide ## Overview [Ming-omni-tts-0.5B](https://huggingface.co/inclusionAI/Ming-omni-tts-0.5B) is inclusionAI's dense text-to-speech model served through **vLLM-Omni** with the OpenAI-compatible `/v1/audio/speech` API. It exposes style, dialect, voice cloning, and multi-speaker controls and outputs **44.1 kHz mono** audio. The `0.5B` refers to the LM backbone; the full checkpoint is ~1.6B parameters including the audio encoder and vocoder. ## Prerequisites - **Hardware**: a single GPU. The reference end-to-end run was on **AMD `gfx942` (ROCm 7.x)** using the `vllm/vllm-omni-rocm:v0.22.0` image; the NVIDIA CUDA path uses the same `vllm serve` command shown below. - **vLLM-Omni** targeting vLLM >= 0.22. - Ming's `model_type` falls through deploy-config auto-detection, so the launch command passes `--deploy-config vllm_omni/deploy/ming_tts.yaml` explicitly (the file ships in the vLLM-Omni repo/package). - The tested environment uses `--enforce-eager`. ## Installation ```bash uv venv && source .venv/bin/activate uv pip install git+https://github.com/vllm-project/vllm-omni.git ``` ## Launch the server ### NVIDIA (CUDA) ```bash vllm serve inclusionAI/Ming-omni-tts-0.5B \ --deploy-config vllm_omni/deploy/ming_tts.yaml \ --omni --enforce-eager --port 8000 ``` ### AMD (ROCm) — reference-tested via the official image ```bash docker run --rm \ --group-add=video --ipc=host --cap-add=SYS_PTRACE \ --security-opt seccomp=unconfined \ --device /dev/kfd --device /dev/dri \ -v ~/.cache/huggingface:/root/.cache/huggingface \ -v "$PWD":/app/vllm-omni -w /app/vllm-omni \ -e VLLM_ROCM_USE_AITER=0 \ -p 8000:8000 \ vllm/vllm-omni-rocm:v0.22.0 \ --model inclusionAI/Ming-omni-tts-0.5B \ --deploy-config vllm_omni/deploy/ming_tts.yaml \ --omni --port 8000 --enforce-eager ``` ## Client usage Dialect control with a reference clip (e.g. Cantonese / 广粤话). `--ref-audio` matches upstream `use_spk_emb=True`; do **not** add `--ref-text` for the dialect case: ```bash python examples/online_serving/text_to_speech/ming_tts/openai_speech_client.py \ --text "我觉得社会企业同个人都有责任" \ --instruction-json '{"方言":"广粤话"}' \ --ref-audio /path/to/yue_prompt.wav \ --max-new-tokens 200 \ --output dialect.wav ``` ## Known limitations - Output is **44.1 kHz mono**. - The reference validation in the source recipe was on AMD `gfx942` (ROCm); the CUDA command above is the equivalent path. ## References - [Model card](https://huggingface.co/inclusionAI/Ming-omni-tts-0.5B) - [Ming-omni-tts upstream repo](https://github.com/inclusionAI/Ming-omni-tts) - [vLLM-Omni Ming-TTS online example](https://github.com/vllm-project/vllm-omni/tree/main/examples/online_serving/text_to_speech/ming_tts) - [vLLM-Omni Ming-TTS offline example](https://github.com/vllm-project/vllm-omni/tree/main/examples/offline_inference/text_to_speech/ming_tts) --- # mistralai/Voxtral-4B-TTS-2603 on vLLM Source: https://recipes.vllm.ai/mistralai/Voxtral-4B-TTS-2603 Provider: Mistral AI Released: 2025-11-17T17:06:57.000Z Updated: 2026-06-15 Architecture: dense Parameters: 4B Min vLLM: 0.20.0 Tasks: omni Summary: Mistral's 4B text-to-speech model served via vLLM-Omni with built-in voice presets, exposed through the OpenAI /v1/audio/speech API (24 kHz mono). ## Variants - default: precision=bf16, vram_minimum_gb=24 BF16 weights — fits a single 24 GB GPU (observed server-startup peak ~20.5 GiB on an RTX 4090) ## Recommended base flags ```bash vllm serve 'mistralai/Voxtral-4B-TTS-2603' \ --omni ``` ## Guide ## Overview [Voxtral-4B-TTS-2603](https://huggingface.co/mistralai/Voxtral-4B-TTS-2603) is Mistral AI's 4B text-to-speech model, served through **vLLM-Omni** with the OpenAI-compatible `/v1/audio/speech` API. It generates **24 kHz mono** audio from text using model-provided voice presets. ## Prerequisites - **Hardware**: a single CUDA GPU with **≥ 24 GB** VRAM. Reference run: 1× RTX 4090 24 GB — Stage 0 (`audio_generation`) ~18.95 GiB, Stage 1 (`audio_tokenizer`) ~1.55 GiB, server-startup peak ~20.5 GiB (~85%). Both stages share GPU 0 via the bundled deploy config. - **vLLM-Omni** targeting vLLM >= 0.20, with `mistral_common >= 1.10.0`. ## Installation ```bash uv venv && source .venv/bin/activate uv pip install git+https://github.com/vllm-project/vllm-omni.git uv pip install -U "mistral_common>=1.10.0" ``` ## Launch the server The deploy config (`vllm_omni/deploy/voxtral_tts.yaml`) is auto-discovered: ```bash vllm serve mistralai/Voxtral-4B-TTS-2603 --omni --port 8000 ``` ## Client usage ### Voice preset (curl) ```bash curl -X POST http://127.0.0.1:8000/v1/audio/speech \ -H "Content-Type: application/json" \ -d '{ "model": "mistralai/Voxtral-4B-TTS-2603", "input": "Hello, this is Voxtral TTS running with vLLM-Omni.", "voice": "casual_female", "language": "English", "response_format": "wav" }' --output voxtral.wav ``` ### Streaming PCM ```bash curl -X POST http://127.0.0.1:8000/v1/audio/speech \ -H "Content-Type: application/json" \ -d '{ "model": "mistralai/Voxtral-4B-TTS-2603", "input": "Hello, this is Voxtral TTS streaming PCM.", "voice": "casual_female", "language": "English", "stream": true, "response_format": "pcm" }' --output voxtral_stream.pcm # Convert raw 24 kHz mono PCM to WAV ffmpeg -f s16le -ar 24000 -ac 1 -i voxtral_stream.pcm voxtral_stream.wav -y ``` ### Examples - [Offline end-to-end](https://github.com/vllm-project/vllm-omni/blob/main/examples/offline_inference/text_to_speech/voxtral_tts/end2end.py) - [Online Gradio demo](https://github.com/vllm-project/vllm-omni/blob/main/examples/online_serving/text_to_speech/voxtral_tts/gradio_demo.py) ## Known limitations - Output is **mono 24 kHz**. - **Voice cloning is gated upstream.** With the public `mistralai/Voxtral-4B-TTS-2603` checkpoint, supplying a `ref_audio` (or running the voice-clone path) fails with `RuntimeError: encode_waveforms requires encoder weights which are not available in the open-source checkpoint.` — the public weights omit the encoder needed to turn reference audio into conditioning features. Use the built-in voice presets instead. ## References - [Model card](https://huggingface.co/mistralai/Voxtral-4B-TTS-2603) - [vLLM-Omni text-to-speech examples](https://github.com/vllm-project/vllm-omni/tree/main/examples/online_serving/text_to_speech/voxtral_tts) --- # openbmb/VoxCPM2 on vLLM Source: https://recipes.vllm.ai/openbmb/VoxCPM2 Provider: MiniCPM (OpenBMB) Released: 2026-04-03T05:25:50.000Z Updated: 2026-06-15 Architecture: dense Parameters: 2B Min vLLM: 0.21.0 Tasks: omni Summary: OpenBMB's 2B native-AR text-to-speech model served via vLLM-Omni — 48 kHz mono, 30+ languages, zero-shot synthesis and reference-audio voice cloning — through the OpenAI /v1/audio/speech API. ## Variants - default: precision=bf16, vram_minimum_gb=24 BF16 weights — single 24 GB GPU (~22 GiB resident with the default config on an RTX 4090); no tensor parallelism needed ## Recommended base flags ```bash vllm serve 'openbmb/VoxCPM2' \ --omni ``` ## Guide ## Overview [VoxCPM2](https://huggingface.co/openbmb/VoxCPM2) is OpenBMB's 2B-parameter native-AR text-to-speech model, served through **vLLM-Omni** with the OpenAI-compatible `/v1/audio/speech` API. It emits **48 kHz mono** audio, supports **30+ languages**, and runs both zero-shot synthesis and reference-audio voice cloning. The pipeline is single-stage (`MiniCPM4 base LM → FSQ → MiniCPM4 residual LM → LocDiT CFM solver → AudioVAE`), so a single 24 GB consumer GPU is sufficient. ## Prerequisites - **Hardware**: one CUDA GPU with **≥ 24 GB** VRAM. Reference run: 1x RTX 4090 24 GB — ~4.9 GiB weights + ~15.2 GiB KV cache + ~2 GiB talker buffers ≈ **22 GiB / 24 GiB** with the default `voxcpm2.yaml` (`gpu_memory_utilization: 0.9`, `max_num_seqs: 4`, `enforce_eager: true`). - **vLLM-Omni** targeting vLLM >= 0.21. - **`voxcpm >= 2.0`**, `soundfile`, `httpx`, and **`ninja`** on `PATH`. ## Installation ```bash uv venv && source .venv/bin/activate uv pip install git+https://github.com/vllm-project/vllm-omni.git uv pip install voxcpm soundfile httpx ninja ``` ## Launch the server The deploy config (`vllm_omni/deploy/voxcpm2.yaml`) is auto-discovered from `model_type=voxcpm2` (no `--trust-remote-code` needed): ```bash vllm serve openbmb/VoxCPM2 --omni --host 0.0.0.0 --port 8000 ``` On a shared GPU, pass `--gpu-memory-utilization 0.75` (or lower) if startup fails the free-memory check. Cold start is ~60 s (subprocess fork + vLLM init + model load + flashinfer JIT + torch.compile + the talker's own CUDA-Graph capture over the CFM solver / AudioVAE). ## Client usage ### Zero-shot synthesis (curl, WAV) ```bash curl -X POST http://127.0.0.1:8000/v1/audio/speech \ -H "Content-Type: application/json" \ -d '{"model": "openbmb/VoxCPM2", "input": "Hello, this is VoxCPM2.", "voice": "default", "response_format": "wav"}' \ --output output.wav ``` ### Voice cloning from a reference WAV (Python client) ```bash python examples/online_serving/text_to_speech/voxcpm2/openai_speech_client.py \ --text "This should sound like the reference speaker." \ --ref-audio /path/to/reference.wav \ --api-base http://127.0.0.1:8000 \ --output cloned.wav ``` `--ref-audio` accepts local paths (auto-base64), HTTP(S) URLs, or `data:audio/wav;base64,...` URIs. The OpenAI `voice` field is required by the schema but ignored unless it names an uploaded voice — cloning is driven entirely by `ref_audio`. ### Streaming PCM (48 kHz s16le) ```bash curl -X POST http://127.0.0.1:8000/v1/audio/speech \ -H "Content-Type: application/json" \ -d '{"model": "openbmb/VoxCPM2", "input": "Streaming PCM test.", "voice": "default", "stream": true, "response_format": "pcm"}' \ --no-buffer | play -t raw -r 48000 -e signed -b 16 -c 1 - ``` > The player sample rate is **48 kHz** (`-r 48000`), not 24 kHz. ## Performance After warmup, steady-state inference RTF is **~0.12** (~8x real-time on a single RTX 4090). The first 1–2 requests pay a one-time warmup cost (torch.compile + CUDA-Graph capture) that amortizes to zero; keep a server warm for interactive use. Voice cloning adds only ~+0.017 RTF over zero-shot (the per-request `AudioVAE.encode` over the reference clip). ## Known limitations - Output is **48 kHz mono**. - The default config sets `enforce_eager: true`; vLLM engine-level CUDA graphs are off by design (the talker captures its own CFM/AudioVAE graph instead). ## References - [Model card](https://huggingface.co/openbmb/VoxCPM2) - [vLLM-Omni VoxCPM2 online example](https://github.com/vllm-project/vllm-omni/tree/main/examples/online_serving/text_to_speech/voxcpm2) - [vLLM-Omni VoxCPM2 offline example](https://github.com/vllm-project/vllm-omni/blob/main/examples/offline_inference/text_to_speech/voxcpm2/end2end.py) --- # zai-org/GLM-TTS on vLLM Source: https://recipes.vllm.ai/zai-org/GLM-TTS Provider: GLM (Z-AI) Released: 2025-12-10T09:59:48.000Z Updated: 2026-06-15 Architecture: dense Parameters: 3B Min vLLM: 0.22.0 Tasks: omni Summary: Z-AI's two-stage (AR + DiT flow-matching) zero-shot voice-cloning TTS for Chinese and English, served via vLLM-Omni through the OpenAI /v1/audio/speech API. Every request is conditioned on reference audio + its transcript. ## Variants - default: precision=bf16, vram_minimum_gb=24 BF16 weights — single GPU; ~18-20 GB total (AR ~10 GB + DiT ~8 GB), ~16.6 GiB peak on a 1x A40 48 GB reference run; fits 24 GB cards ## Recommended base flags ```bash vllm serve 'zai-org/GLM-TTS' \ --omni \ --trust-remote-code ``` ## Guide ## Overview [GLM-TTS](https://huggingface.co/zai-org/GLM-TTS) is Z-AI's two-stage text-to-speech system — an LLM that generates speech tokens (AR) followed by a Flow-Matching DiT that produces mel-spectrograms, vocoded by HiFT — served through **vLLM-Omni** with the OpenAI-compatible `/v1/audio/speech` API. It targets **Chinese and English**, and **every request is conditioned on reference audio + its transcript** (zero-shot voice cloning). ## Prerequisites - **Hardware**: a single CUDA GPU. Reference run: 1x A40 48 GB — ~18-20 GB total (AR ~10 GB + DiT ~8 GB), ~16.6 GiB peak; fits 24 GB cards. Both stages share GPU 0 by default; split them across GPUs for higher concurrency. - **vLLM-Omni** targeting vLLM >= 0.22. - `--trust-remote-code` is required for the GLM-TTS phoneme tokenizer. ## Installation ```bash uv venv && source .venv/bin/activate uv pip install git+https://github.com/vllm-project/vllm-omni.git ``` ## Launch the server The deploy config (`vllm_omni/deploy/glm_tts.yaml`) is auto-discovered. Async chunking (streaming) is on by default: ```bash vllm serve zai-org/GLM-TTS --omni --trust-remote-code --port 8000 ``` For the synchronous (non-streaming) path: ```bash vllm serve zai-org/GLM-TTS --omni --trust-remote-code --port 8000 --no-async-chunk ``` ## Client usage ### Zero-shot voice clone (curl) ```bash curl -X POST http://localhost:8000/v1/audio/speech \ -H "Content-Type: application/json" \ -d '{ "model": "zai-org/GLM-TTS", "input": "你好，这是一个语音合成测试。", "response_format": "wav", "ref_audio": "file:///path/to/ref.wav", "ref_text": "这是参考音频的文本内容。" }' --output test.wav ``` ### Python client ```bash python examples/online_serving/text_to_speech/glm_tts/openai_speech_client.py \ --text "你好，这是语音克隆测试。" \ --ref-audio file:///path/to/ref.wav \ --ref-text "这是参考音频的文本内容。" ``` ## Known limitations - Output is **24 kHz mono** WAV via the HiFT vocoder (Vocos2D 32 kHz fallback with resampling). - Voice cloning requires `ref_audio` **and** `ref_text` together; the reference clip should be **3-10 seconds**. Feature extraction (WhisperVQ tokenizer, CampPlus ONNX, mel) runs model-side. - The first request may be slow due to lazy model loading (WhisperVQ, CampPlus ONNX). Warm-cache RTF is ~0.6-0.7x on an A40. ## References - [Model card](https://huggingface.co/zai-org/GLM-TTS) - [GLM-TTS GitHub](https://github.com/zai-org/GLM-TTS) - [vLLM-Omni GLM-TTS example](https://github.com/vllm-project/vllm-omni/tree/main/examples/online_serving/text_to_speech/glm_tts) --- # Qwen/Qwen2.5-32B on vLLM Source: https://recipes.vllm.ai/Qwen/Qwen2.5-32B Provider: Qwen Released: 2024-09-15T12:18:33.000Z Updated: 2026-06-04 Architecture: dense Parameters: 32B Context length: 131,072 Min vLLM: 0.6.2 Tasks: text Verified hardware: TPU v6e (Trillium) Summary: Qwen2.5 32B dense base (pretrained) language model for text completion — verified on TPU v6e (Trillium). ## Variants - default: precision=bf16, vram_minimum_gb=77 Full precision BF16 — TPU v6e 2x2 (4 chips, TP=4) or a single 80GB+ GPU - fp8: precision=fp8, vram_minimum_gb=39, model_id=RedHatAI/Qwen2.5-32B-FP8-dynamic RedHatAI FP8 (E4M3) with dynamic per-token activation quantization ## Guide ## Overview Qwen2.5-32B is the 32B dense **base** (pretrained, non-instruct) model in the Qwen2.5 series. It's a text-completion model — use the `/v1/completions` endpoint rather than chat, or pick `Qwen/Qwen2.5-32B-Instruct` if you need a chat-tuned variant. BF16 is the training precision and gives the best accuracy. ## Prerequisites ### NVIDIA ```bash uv venv source .venv/bin/activate uv pip install -U vllm --torch-backend auto ``` ### TPU (Trillium / v6e) Use the official vLLM TPU image. The 32B model is served on a `v6e-4` slice (`--topology 2x2`, 4 chips, TP=4). ```bash export HF_HOME=/dev/shm export HF_TOKEN= ``` ## Deployment Configurations ### TPU v6e (Trillium, 2x2 slice, TP=4) Verified end-to-end on a 4-chip v6e slice. ```bash export HF_HOME=/dev/shm export HF_TOKEN= export TP=4 export MAX_MODEL_LEN=4096 vllm serve Qwen/Qwen2.5-32B \ --seed 42 \ --gpu-memory-utilization 0.98 \ --max-num-batched-tokens 2048 \ --max-num-seqs 128 \ --tensor-parallel-size $TP \ --max-model-len $MAX_MODEL_LEN ``` ### Single GPU / node (BF16) ```bash vllm serve Qwen/Qwen2.5-32B \ --host 0.0.0.0 \ --port 8000 \ --tensor-parallel-size 2 ``` ## Configuration Tips - `--max-model-len 4096` keeps memory in check on a 4-chip v6e slice; native context is 128K — raise it if you have headroom. - vLLM has been V1-engine-only for a long time, so no `VLLM_USE_V1=1` export is needed even though the upstream TPU recipe still shows it. - vLLM uses 90% of device memory by default; the TPU recipe pushes `--gpu-memory-utilization 0.98` to maximize KV cache. - `--max-num-batched-tokens` and `--max-num-seqs` tune batch throughput vs latency. ## Benchmarking Launch the server with `--no-enable-prefix-caching` for consistent measurements. ```bash vllm bench serve \ --host 0.0.0.0 \ --port 8000 \ --model Qwen/Qwen2.5-32B \ --dataset-name random \ --random-input-len 1024 \ --random-output-len 1024 \ --num-prompts 128 ``` ## References - [Model card](https://huggingface.co/Qwen/Qwen2.5-32B) - [Qwen2.5-32B on Trillium (v6e)](https://github.com/AI-Hypercomputer/tpu-recipes/tree/main/inference/trillium/vLLM/Qwen2.5-32B) --- # JetBrains/Mellum2-12B-A2.5B-Instruct on vLLM Source: https://recipes.vllm.ai/JetBrains/Mellum2-12B-A2.5B-Instruct Provider: JetBrains Released: 2026-05-26T09:09:18.000Z Updated: 2026-06-02 Architecture: moe Parameters: 12B Context length: 131,072 Min vLLM: 0.23.0 Tasks: text Verified hardware: H200, MI300X, MI325X, MI355X Summary: JetBrains' instruction-tuned code MoE (12B total / 2.5B active) that answers directly without an externalized chain of thought — low-latency coding and tool use ## Variants - default: precision=bf16, vram_minimum_gb=29 Native bfloat16 weights; fits comfortably on a single H200/H100/A100 ## Recommended base flags ```bash vllm serve 'JetBrains/Mellum2-12B-A2.5B-Instruct' \ --max-model-len \ 131072 \ --trust-remote-code ``` ## Guide ## Overview [Mellum2-12B-A2.5B-Instruct](https://huggingface.co/JetBrains/Mellum2-12B-A2.5B-Instruct) is JetBrains' instruction-tuned code assistant. It shares the same Mixture-of-Experts backbone as the rest of the Mellum2 family — 64 experts (8 activated per token), 12B total / 2.5B active parameters, sliding-window + full-attention layers, 131,072-token context — but is post-trained (SFT + RLVR on math, coding, tool use, instruction following, reasoning, and knowledge) to **answer directly, without an externalized chain of thought**. For complex debugging, multi-step planning, or math/reasoning-heavy tasks where you want explicit reasoning traces, use the [Thinking](https://huggingface.co/JetBrains/Mellum2-12B-A2.5B-Thinking) variant instead. ## Prerequisites - Hardware: a single H200, H100, or A100 (~29 GB at bf16) is plenty. On AMD, use MI300X / MI325X / MI355X with TP=1. - vLLM **nightly** — `MellumForCausalLM` support landed after v0.22.0 and is not yet in a stable release. Install the nightly wheels until the next tagged release ships. ### Install vLLM (nightly) ```bash uv venv source .venv/bin/activate uv pip install -U vllm --extra-index-url https://wheels.vllm.ai/nightly ``` ## Launch command Unlike the Thinking checkpoint, Instruct does not emit `` blocks, so no `--reasoning-parser` is needed. ### Single GPU ```bash vllm serve JetBrains/Mellum2-12B-A2.5B-Instruct \ --max-model-len 131072 \ --trust-remote-code \ --tensor-parallel-size 1 # Optional: add tool calling. vllm serve JetBrains/Mellum2-12B-A2.5B-Instruct \ --max-model-len 131072 \ --trust-remote-code \ --tensor-parallel-size 1 \ --enable-auto-tool-choice \ --tool-call-parser hermes ``` ### Docker (AMD MI300X / MI325X / MI355X) MI300X / MI325X / MI355X GPUs have larger per-GPU HBM, so you can use `--max-model-len auto`. ```bash # Plain serving vllm serve JetBrains/Mellum2-12B-A2.5B-Instruct \ --max-model-len auto \ --trust-remote-code \ --tensor-parallel-size 1 # Add tool calling. vllm serve JetBrains/Mellum2-12B-A2.5B-Instruct \ --max-model-len auto \ --trust-remote-code \ --tensor-parallel-size 1 \ --enable-auto-tool-choice \ --tool-call-parser hermes ``` ## Client usage JetBrains recommends sampling at `temperature=0.6`, `top_p=0.95`, `top_k=20`. ```python from openai import OpenAI client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY") resp = client.chat.completions.create( model="JetBrains/Mellum2-12B-A2.5B-Instruct", messages=[{"role": "user", "content": "Write a Python function to reverse a string."}], max_tokens=81920, temperature=0.6, top_p=0.95, extra_body={"top_k": 20}, ) print(resp.choices[0].message.content) ``` ## References - [Model card](https://huggingface.co/JetBrains/Mellum2-12B-A2.5B-Instruct) - [vLLM support PR #43992](https://github.com/vllm-project/vllm/pull/43992) - [Mellum2 Technical Report](https://arxiv.org/abs/2605.31268) --- # JetBrains/Mellum2-12B-A2.5B-Thinking on vLLM Source: https://recipes.vllm.ai/JetBrains/Mellum2-12B-A2.5B-Thinking Provider: JetBrains Released: 2026-05-26T09:12:25.000Z Updated: 2026-06-02 Architecture: moe Parameters: 12B Context length: 131,072 Min vLLM: 0.23.0 Tasks: text Verified hardware: H200 Summary: JetBrains' reasoning-augmented code MoE (12B total / 2.5B active) that emits explicit chains for debugging, planning, and agentic coding ## Variants - default: precision=bf16, vram_minimum_gb=29 Native bfloat16 weights; fits comfortably on a single H200/H100/A100 - fp8: precision=fp8, vram_minimum_gb=15, model_id=RedHatAI/Mellum2-12B-A2.5B-Thinking-FP8-Dynamic RedHatAI FP8 (E4M3) with dynamic per-token activation quantization - nvfp4: precision=nvfp4, vram_minimum_gb=8, model_id=RedHatAI/Mellum2-12B-A2.5B-Thinking-NVFP4 RedHatAI NVFP4 quantization ## Recommended base flags ```bash vllm serve 'JetBrains/Mellum2-12B-A2.5B-Thinking' \ --max-model-len \ 131072 ``` ## Guide ## Overview [Mellum2-12B-A2.5B-Thinking](https://huggingface.co/JetBrains/Mellum2-12B-A2.5B-Thinking) is JetBrains' reasoning-augmented code assistant. It uses a Mixture-of-Experts architecture with 64 experts (8 activated per token) — 12B total parameters, 2.5B active — combining sliding-window and full-attention layers for a 131,072-token context. The model emits its chain-of-thought inside `...` blocks before the final answer, making it suited to complex debugging, multi-step planning, and agentic workflows. For direct, low-latency answers without reasoning traces, use the [Instruct](https://huggingface.co/JetBrains/Mellum2-12B-A2.5B-Instruct) variant instead. ## Prerequisites - Hardware: a single H200, H100, or A100 (~29 GB at bf16) is plenty - vLLM **nightly** — `MellumForCausalLM` support landed after v0.22.0 and is not yet in a stable release. Install the nightly wheels until the next tagged release ships. ### Install vLLM (nightly) ```bash uv venv source .venv/bin/activate uv pip install -U vllm --extra-index-url https://wheels.vllm.ai/nightly ``` ## Launch command ```bash # With reasoning (recommended for the Thinking checkpoint) vllm serve JetBrains/Mellum2-12B-A2.5B-Thinking \ --max-model-len 131072 \ --reasoning-parser qwen3 # Add tool calling vllm serve JetBrains/Mellum2-12B-A2.5B-Thinking \ --max-model-len 131072 \ --reasoning-parser qwen3 \ --enable-auto-tool-choice \ --tool-call-parser hermes ``` ## Client usage JetBrains recommends sampling at `temperature=0.6`, `top_p=0.95`, `top_k=20` for the Thinking checkpoint. ```python from openai import OpenAI client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY") resp = client.chat.completions.create( model="JetBrains/Mellum2-12B-A2.5B-Thinking", messages=[{"role": "user", "content": "Is 1024 a power of 2? Explain your reasoning."}], max_tokens=81920, temperature=0.6, top_p=0.95, extra_body={"top_k": 20}, ) print(resp.choices[0].message.content) ``` ## References - [Model card](https://huggingface.co/JetBrains/Mellum2-12B-A2.5B-Thinking) - [vLLM support PR #43992](https://github.com/vllm-project/vllm/pull/43992) - [Mellum2 Technical Report](https://arxiv.org/abs/2605.31268) --- # nvidia/Cosmos3-Nano on vLLM Source: https://recipes.vllm.ai/nvidia/Cosmos3-Nano Provider: NVIDIA Released: 2026-03-10T23:01:40.000Z Updated: 2026-06-02 Architecture: dense Parameters: 16B Context length: 262,144 Min vLLM: 0.21.0 Tasks: omni Verified hardware: H200, H100 Summary: Compact 16B omnimodal world model (Mixture-of-Transformers) for multimodal understanding, world simulation, future prediction, action reasoning, and Physical AI ## Variants - default: precision=bf16, vram_minimum_gb=39 BF16 weights — single-GPU H200 omni generation ## Recommended base flags ```bash vllm serve 'nvidia/Cosmos3-Nano' \ --host \ 0.0.0.0 \ --port \ 8000 \ --init-timeout \ 1800 ``` ## Guide ## Overview **Cosmos3-Nano** is the compact (16B) member of NVIDIA's Cosmos3 family of **omnimodal world models**. Built on a Mixture-of-Transformers (MoT) architecture — an autoregressive tower for discrete-token generation paired with a diffusion tower for continuous modalities — it generates coherent text, images, video, audio, and action commands from combinations of text, image, video, and action-trajectory inputs. It targets Physical AI: world understanding, world simulation, future prediction, and action reasoning. This recipe covers the **vLLM-Omni** serving path (video + muxed-audio generation). The same checkpoint also serves a text/multimodal **Reasoner** endpoint via standard vLLM — see [Reasoner mode](#reasoner-mode-multimodal-understanding) below. ## Prerequisites - 1× H200 (or H100 / A100) for the documented single-GPU profile - The release-tested `vllm/vllm-omni:cosmos3` container, or vLLM-Omni installed on top of `vllm==0.21.0` - Prompts should be JSON-upsampled for best quality (see [cosmos-framework prompt upsampling](https://github.com/NVIDIA/cosmos-framework/blob/main/docs/prompt_upsampling.md)) ## Launch command (vLLM-Omni) ```bash docker pull vllm/vllm-omni:cosmos3 vllm serve nvidia/Cosmos3-Nano \ --omni \ --host 0.0.0.0 \ --port 8000 \ --init-timeout 1800 ``` To speed up inference with additional GPUs, enable context parallelism with `--ulysses-degree` or switch to tensor parallelism with `--tensor-parallel-size`. On memory-constrained GPUs, `--enable-layerwise-offload` reduces VRAM usage at a performance cost. ### Example: text-to-video ```bash curl -X POST http://localhost:8000/v1/videos/sync \ -H "Accept: video/mp4" \ -F "prompt=$(cat assets/example_t2v_prompt.json)" \ -F "negative_prompt=$(cat assets/negative_prompt.json)" \ -F "size=1280x720" \ -F "num_frames=189" \ -F "fps=24" \ -F "num_inference_steps=35" \ -F "guidance_scale=6.0" \ -F "max_sequence_length=4096" \ -F "flow_shift=10.0" \ -F "seed=123" \ --output output.mp4 ``` Image-to-video uses the same endpoint with an `input_reference` file part. Audio is generated jointly and muxed into the output MP4 (48 kHz stereo AAC) — there is no separate audio endpoint. ## Reasoner mode (multimodal understanding) The same checkpoint serves a text/multimodal **Reasoner** for world understanding, future prediction, and action reasoning via the standard `vllm` package (not vLLM-Omni): ```bash # CUDA 13 drivers: uv pip install --torch-backend=cu130 "vllm==0.21.0" \ "vllm-cosmos3 @ git+https://github.com/NVIDIA/cosmos-framework.git#subdirectory=packages/vllm-cosmos3" \ openai # CUDA 12.8 drivers: use --torch-backend=cu128 "vllm==0.19.1" CUDA_VISIBLE_DEVICES=0 \ vllm serve nvidia/Cosmos3-Nano \ --hf-overrides '{"architectures": ["Cosmos3ReasonerForConditionalGeneration"]}' \ --tensor-parallel-size 1 \ --mm-encoder-tp-mode data \ --async-scheduling \ --allowed-local-media-path / \ --media-io-kwargs '{"video": {"num_frames": -1}}' \ --port 8000 ``` Then query the OpenAI-compatible `/v1/chat/completions` endpoint with text + image/video content. ## References - [Model card](https://huggingface.co/nvidia/Cosmos3-Nano) - [vLLM-Omni](https://github.com/vllm-project/vllm-omni) - [cosmos-framework](https://github.com/NVIDIA/cosmos-framework) - [Cosmos](https://github.com/nvidia/cosmos) --- # nvidia/Cosmos3-Super-Image2Video on vLLM Source: https://recipes.vllm.ai/nvidia/Cosmos3-Super-Image2Video Provider: NVIDIA Released: 2026-05-21T19:21:28.000Z Updated: 2026-06-02 Architecture: dense Parameters: 64B Context length: 262,144 Min vLLM: 0.21.0 Tasks: omni Verified hardware: H200, H100 Summary: 64B Cosmos3-Super specialization for temporally coherent image-to-video generation ## Variants - default: precision=bf16, vram_minimum_gb=154 BF16 weights — 8×H200/H100/A100 node recommended ## Recommended base flags ```bash vllm serve 'nvidia/Cosmos3-Super-Image2Video' \ --host \ 0.0.0.0 \ --port \ 8000 \ --cfg-parallel-size \ 2 \ --ulysses-degree \ 4 \ --use-hsdp \ --hsdp-shard-size \ 8 \ --init-timeout \ 1800 ``` ## Guide ## Overview **Cosmos3-Super-Image2Video** is a 64B specialization of NVIDIA's Cosmos3-Super omnimodal world model, tuned for **temporally coherent image-to-video generation**: given one input image and text instructions, it produces video sequences consistent with the provided visual content. It is served via **vLLM-Omni**. For best quality, text prompts should be JSON-upsampled (see the `scripts/upsample_prompt.py` helper in the model repo). ## Prerequisites - 8× H200, H100, or A100 for the documented full-node profile - The release-tested `vllm/vllm-omni:cosmos3` container, or vLLM-Omni installed on top of `vllm==0.21.0` ## Launch command (vLLM-Omni) Recommended configuration on 8×H200, 8×H100, or 8×A100: ```bash docker pull vllm/vllm-omni:cosmos3 vllm serve nvidia/Cosmos3-Super-Image2Video \ --omni \ --host 0.0.0.0 \ --port 8000 \ --cfg-parallel-size 2 \ --ulysses-degree 4 \ --use-hsdp \ --hsdp-shard-size 8 \ --init-timeout 1800 ``` With this configuration, 50-step video generation takes ~55 seconds on H200. For **2×H200**, use `--cfg-parallel-size 2 --use-hsdp --hsdp-shard-size 2` (a video takes ~3 minutes). Tensor parallelism is also supported via `--tensor-parallel-size`. On memory-constrained GPUs, `--enable-layerwise-offload` reduces VRAM usage at a performance cost. ### Example: image-to-video ```bash curl -X POST http://localhost:8000/v1/videos/sync \ -H "Accept: video/mp4" \ -F "input_reference=@assets/example_first_frame.png" \ -F "prompt=$(cat assets/example_prompt.json)" \ -F "size=1280x720" \ -F "num_frames=189" \ -F "fps=24" \ -F "num_inference_steps=35" \ -F "guidance_scale=6.0" \ -F "max_sequence_length=4096" \ -F "flow_shift=10.0" \ -F "seed=1111" \ --output output.mp4 ``` ## References - [Model card](https://huggingface.co/nvidia/Cosmos3-Super-Image2Video) - [vLLM-Omni](https://github.com/vllm-project/vllm-omni) - [Cosmos](https://github.com/nvidia/cosmos) --- # nvidia/Cosmos3-Super-Text2Image on vLLM Source: https://recipes.vllm.ai/nvidia/Cosmos3-Super-Text2Image Provider: NVIDIA Released: 2026-05-28T03:49:21.000Z Updated: 2026-06-02 Architecture: dense Parameters: 64B Context length: 262,144 Min vLLM: 0.21.0 Tasks: omni Verified hardware: H200, H100, GB200 NVL4 Summary: 64B Cosmos3-Super specialization for high-fidelity text-to-image generation ## Variants - default: precision=bf16, vram_minimum_gb=154 BF16 weights — 8×H100 node recommended ## Recommended base flags ```bash vllm serve 'nvidia/Cosmos3-Super-Text2Image' \ --host \ 0.0.0.0 \ --port \ 8000 \ --cfg-parallel-size \ 2 \ --ulysses-degree \ 4 \ --tensor-parallel-size \ 1 \ --use-hsdp \ --hsdp-shard-size \ 8 \ --init-timeout \ 1800 ``` ## Guide ## Overview **Cosmos3-Super-Text2Image** is a 64B specialization of NVIDIA's Cosmos3-Super omnimodal world model, tuned for **high-fidelity text-to-image generation**. It is served via **vLLM-Omni**. For best quality, text prompts should be JSON-upsampled — the model repo ships an `agentic_upsampling/` package and an `AGENTIC_UPSAMPLING.md` guide. ## Prerequisites - 8× H100 for the documented full-node profile (4×H200 / 4×GB200 alternative below) - The release-tested `vllm/vllm-omni:cosmos3` container, or vLLM-Omni installed on top of `vllm==0.21.0` ## Launch command (vLLM-Omni) Recommended configuration on an 8×H100 node: ```bash docker pull vllm/vllm-omni:cosmos3 vllm serve nvidia/Cosmos3-Super-Text2Image \ --omni \ --host 0.0.0.0 \ --port 8000 \ --cfg-parallel-size 2 \ --ulysses-degree 4 \ --tensor-parallel-size 1 \ --use-hsdp \ --hsdp-shard-size 8 \ --init-timeout 1800 ``` For **4×H200 or 4×GB200**, use `--cfg-parallel-size 2 --ulysses-degree 2 --tensor-parallel-size 1`. `--enable-layerwise-offload` can reduce VRAM usage on smaller GPUs, but for text-to-image it incurs a significant performance penalty. ### Example: text-to-image ```bash curl -X POST http://localhost:8000/v1/images/generations \ -H "Content-Type: application/json" \ -d '{ "prompt": "a dragon flying over the Green Mountains at sunset", "size": "1024x1024", "seed": 42 }' | jq -r '.data[0].b64_json' | base64 -d > output.png ``` ## References - [Model card](https://huggingface.co/nvidia/Cosmos3-Super-Text2Image) - [Agentic prompt upsampling](https://huggingface.co/nvidia/Cosmos3-Super-Text2Image/blob/main/AGENTIC_UPSAMPLING.md) - [vLLM-Omni](https://github.com/vllm-project/vllm-omni) - [Cosmos](https://github.com/nvidia/cosmos) --- # nvidia/Cosmos3-Super on vLLM Source: https://recipes.vllm.ai/nvidia/Cosmos3-Super Provider: NVIDIA Released: 2026-03-10T22:59:21.000Z Updated: 2026-06-02 Architecture: dense Parameters: 64B Context length: 262,144 Min vLLM: 0.21.0 Tasks: omni Verified hardware: H200, H100 Summary: Frontier-scale 64B omnimodal world model (Mixture-of-Transformers) for advanced multimodal understanding, world simulation, future prediction, action reasoning, and Physical AI ## Variants - default: precision=bf16, vram_minimum_gb=154 BF16 weights — frontier-scale; 8×H200/H100/A100 node recommended ## Recommended base flags ```bash vllm serve 'nvidia/Cosmos3-Super' \ --host \ 0.0.0.0 \ --port \ 8000 \ --cfg-parallel-size \ 2 \ --ulysses-degree \ 4 \ --use-hsdp \ --hsdp-shard-size \ 8 \ --init-timeout \ 1800 ``` ## Guide ## Overview **Cosmos3-Super** is the frontier-scale (64B) member of NVIDIA's Cosmos3 family of **omnimodal world models**. Built on a Mixture-of-Transformers (MoT) architecture — an autoregressive tower for discrete-token generation paired with a diffusion tower for continuous modalities — it generates coherent text, images, video, audio, and action commands from combinations of text, image, video, and action-trajectory inputs, for advanced world understanding, simulation, future prediction, and action reasoning. This recipe covers the **vLLM-Omni** serving path (video + muxed-audio generation). The same checkpoint also serves a text/multimodal **Reasoner** endpoint via standard vLLM — see [Reasoner mode](#reasoner-mode-multimodal-understanding) below. ## Prerequisites - 8× H200, H100, or A100 for the documented full-node profile - The release-tested `vllm/vllm-omni:cosmos3` container, or vLLM-Omni installed on top of `vllm==0.21.0` - Prompts should be JSON-upsampled for best quality (see [cosmos-framework prompt upsampling](https://github.com/NVIDIA/cosmos-framework/blob/main/docs/prompt_upsampling.md)) ## Launch command (vLLM-Omni) Recommended configuration on 8×H200, 8×H100, or 8×A100: ```bash docker pull vllm/vllm-omni:cosmos3 vllm serve nvidia/Cosmos3-Super \ --omni \ --host 0.0.0.0 \ --port 8000 \ --cfg-parallel-size 2 \ --ulysses-degree 4 \ --use-hsdp \ --hsdp-shard-size 8 \ --init-timeout 1800 ``` With this configuration, 50-step video generation takes ~55 seconds on H200. For **2×H200**, use `--cfg-parallel-size 2 --use-hsdp --hsdp-shard-size 2` (a video takes ~3 minutes). Tensor parallelism is also supported via `--tensor-parallel-size`. On memory-constrained GPUs, `--enable-layerwise-offload` reduces VRAM usage at a performance cost. ### Example: text-to-video ```bash curl -X POST http://localhost:8000/v1/videos/sync \ -H "Accept: video/mp4" \ -F "prompt=$(cat assets/example_t2v_prompt.json)" \ -F "negative_prompt=$(cat assets/negative_prompt.json)" \ -F "size=1280x720" \ -F "num_frames=189" \ -F "fps=24" \ -F "num_inference_steps=35" \ -F "guidance_scale=6.0" \ -F "max_sequence_length=4096" \ -F "flow_shift=10.0" \ -F "seed=123" \ --output output.mp4 ``` Image-to-video uses the same endpoint with an `input_reference` file part. Audio is generated jointly and muxed into the output MP4 (48 kHz stereo AAC). ## Reasoner mode (multimodal understanding) The same checkpoint serves a text/multimodal **Reasoner** via the standard `vllm` package (not vLLM-Omni): ```bash # CUDA 13 drivers: uv pip install --torch-backend=cu130 "vllm==0.21.0" \ "vllm-cosmos3 @ git+https://github.com/NVIDIA/cosmos-framework.git#subdirectory=packages/vllm-cosmos3" \ openai # CUDA 12.8 drivers: use --torch-backend=cu128 "vllm==0.19.1" vllm serve nvidia/Cosmos3-Super \ --hf-overrides '{"architectures": ["Cosmos3ReasonerForConditionalGeneration"]}' \ --tensor-parallel-size 4 \ --mm-encoder-tp-mode data \ --async-scheduling \ --allowed-local-media-path / \ --media-io-kwargs '{"video": {"num_frames": -1}}' \ --port 8000 ``` Then query the OpenAI-compatible `/v1/chat/completions` endpoint with text + image/video content. ## References - [Model card](https://huggingface.co/nvidia/Cosmos3-Super) - [vLLM-Omni](https://github.com/vllm-project/vllm-omni) - [cosmos-framework](https://github.com/NVIDIA/cosmos-framework) - [Cosmos](https://github.com/nvidia/cosmos) --- # openbmb/MiniCPM-V-4.6 on vLLM Source: https://recipes.vllm.ai/openbmb/MiniCPM-V-4.6 Provider: MiniCPM (OpenBMB) Released: 2026-04-13T09:12:47.000Z Updated: 2026-06-02 Architecture: dense Parameters: 1.3B Context length: 262,144 Min vLLM: 0.22.0 Tasks: multimodal Summary: MiniCPM-V 4.6 (1.3B) — pocket-sized multimodal LLM for ultra-efficient single-image, multi-image, and video understanding, built on SigLIP2-400M + a Qwen3.5-0.8B hybrid-attention backbone ## Variants - default: precision=bf16, vram_minimum_gb=4 BF16 weights (~1.3B params) — tiny footprint, runs comfortably on a single consumer GPU ## Recommended base flags ```bash vllm serve 'openbmb/MiniCPM-V-4.6' \ --trust-remote-code ``` ## Guide ## Overview **MiniCPM-V 4.6** is OpenBMB's most edge-friendly multimodal model to date. It pairs a **SigLIP2-400M** vision encoder with a **Qwen3.5-0.8B** hybrid linear/full-attention language backbone (~1.3B params total). It handles single-image, multi-image, and video understanding, and introduces **mixed 4×/16× visual token compression** — switch the `downsample_mode` per request to trade accuracy for speed. > MiniCPM-V 4.6 ships as **two independent checkpoints** — this recipe covers > the Instruct model (`openbmb/MiniCPM-V-4.6`). For chain-of-thought output, > use the separate `openbmb/MiniCPM-V-4.6-Thinking` checkpoint instead (unlike > v4.5, the mode is no longer toggled at runtime). ## Prerequisites - **vLLM ≥ 0.22.0** — the `MiniCPMV4_6ForConditionalGeneration` architecture landed via [PR #43213](https://github.com/vllm-project/vllm/pull/43213) (merged 2026-05-22) and first shipped in the v0.22.0 release. No fork is required. - **transformers ≥ 5.7.0** — the model is merged as a standalone architecture only in this version. - For video inputs, install the `vllm[video]` extra. ## Launch command Use the command builder above. The baseline is simply: ```bash vllm serve openbmb/MiniCPM-V-4.6 --trust-remote-code --port 8000 ``` Enable the **Tool calling** feature to add `--enable-auto-tool-choice --tool-call-parser qwen3_coder`. The model emits a Qwen3-Coder-style `` block inside the message content. At 1.3B params the model fits on a single GPU (TP=1). v4.6 supports context up to 256K, but start with a smaller `--max-model-len` (e.g. `8192`) and raise it to fit your GPU memory and workload. ## Client usage **Image:** ```bash curl -s http://localhost:8000/v1/chat/completions -H 'Content-Type: application/json' -d '{ "model": "openbmb/MiniCPM-V-4.6", "messages": [{"role": "user", "content": [ {"type": "image_url", "image_url": {"url": "https://huggingface.co/datasets/openbmb/DemoCase/resolve/main/refract.png"}}, {"type": "text", "text": "What causes this phenomenon?"} ]}] }' ``` > **Stop tokens:** v4.6 uses the Qwen3.5 backbone with a new vocab. If you see > runaway generations, pass `"stop_token_ids": [248044, 248046]` in your > request body. ## References - [Model card](https://huggingface.co/openbmb/MiniCPM-V-4.6) - [Thinking checkpoint](https://huggingface.co/openbmb/MiniCPM-V-4.6-Thinking) - [vLLM deployment guide (Cookbook)](https://github.com/OpenSQZ/MiniCPM-V-CookBook/blob/main/deployment/vllm/minicpm-v4_6_vllm.md) - [vLLM support PR #43213](https://github.com/vllm-project/vllm/pull/43213) --- # openbmb/MiniCPM5-1B on vLLM Source: https://recipes.vllm.ai/openbmb/MiniCPM5-1B Provider: MiniCPM (OpenBMB) Released: 2026-05-21T07:27:59.000Z Updated: 2026-06-02 Architecture: dense Parameters: 1.1B Context length: 131,072 Min vLLM: 0.21.0 Tasks: text Summary: MiniCPM5-1B — dense 1B on-device LLM with hybrid Think/No-Think reasoning, native 128K context, and strong agentic tool use, built on the standard Llama architecture ## Variants - default: precision=bf16, vram_minimum_gb=3 BF16 weights (~1.1B params) — tiny footprint, runs on a single consumer GPU ## Guide ## Overview **MiniCPM5-1B** is the first checkpoint in OpenBMB's MiniCPM5 series — a dense 1B model built for on-device and resource-constrained deployment, reaching 1B-class open-source SOTA on agentic tool use, code generation, and difficult reasoning. It uses the **standard `LlamaForCausalLM` architecture**, so vLLM loads it natively with no custom kernels or model-code fork. Its headline feature is **hybrid reasoning**: a single checkpoint serves as both a fast assistant (No-Think) and a deliberate reasoner (Think), toggled by the chat template's `enable_thinking` flag. ## Prerequisites - **vLLM ≥ 0.21.0** — MiniCPM5-1B is supported natively as of the v0.21.0 release. (For CUDA 12.x driver hosts, the cookbook suggests `vllm==0.10.1.1` as a fallback.) ## Launch command Use the command builder above. The baseline is simply: ```bash vllm serve openbmb/MiniCPM5-1B --port 8000 ``` At ~1.1B params the model fits on a single GPU (TP=1). It supports the full native 128K context; drop `--max-model-len` to `8192` / `32768` to free KV cache on small GPUs, and set `--enforce-eager` if CUDA graphs OOM on a tiny VRAM budget. ## Reasoning modes Toggle the **Reasoning** feature to serve with deep-thinking on by default (`--default-chat-template-kwargs '{"enable_thinking": true}'`). You can also flip it per request via `chat_template_kwargs`: ```bash curl http://localhost:8000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "openbmb/MiniCPM5-1B", "messages": [{"role": "user", "content": "Explain GQA in one sentence."}], "temperature": 0.9, "top_p": 0.95, "max_tokens": 1024, "chat_template_kwargs": {"enable_thinking": true} }' ``` | Mode | `enable_thinking` | `temperature` | `top_p` | | --- | --- | --- | --- | | Think | `true` | 0.9 | 0.95 | | No-Think | `false` | 0.7 | 0.95 | ## Tool calling MiniCPM5-1B emits **XML-style** tool calls. The vLLM-side `minicpm5` parser ([PR #43175](https://github.com/vllm-project/vllm/pull/43175)) merged to `main` on 2026-05-27 but is **not in v0.21.0 or v0.22.0** — those releases were cut before the merge. Until a release bakes it in (v0.23+), load it as a plugin from the [MiniCPM repo](https://github.com/OpenBMB/MiniCPM): ```bash vllm serve openbmb/MiniCPM5-1B --port 8000 \ --enable-auto-tool-choice \ --tool-parser-plugin /path/to/MiniCPM/tool_parsers/minicpm5xml_tool_parser.py \ --tool-call-parser minicpm5 ``` SGLang ships the `minicpm5` parser built-in and is the author-recommended backend for tool calling. ## References - [Model card](https://huggingface.co/openbmb/MiniCPM5-1B) - [vLLM deployment cookbook](https://github.com/OpenBMB/MiniCPM/blob/main/docs/deployment/vllm.md) - [MiniCPM Tech Report](https://arxiv.org/pdf/2506.07900) --- # stepfun-ai/Step-3.7-Flash on vLLM Source: https://recipes.vllm.ai/stepfun-ai/Step-3.7-Flash Provider: StepFun Released: 2026-05-23T02:13:46.000Z Updated: 2026-05-30 Architecture: moe Parameters: 198B Context length: 262,144 Min vLLM: 0.23.0 Tasks: multimodal Verified hardware: H200, B200, DGX Station (GB300) Summary: Production-grade vision-language MoE (~198B total / 11B active parameters) combining a 196B sparse language backbone with a 1.8B perception encoder, hybrid SWA/Global attention, and 3-way Multi-Token Prediction ## Variants - default: precision=bf16, vram_minimum_gb=475 Full precision BF16 — recommended on 8xH200/B200 with TP8+EP - fp8: precision=fp8, vram_minimum_gb=238, model_id=stepfun-ai/Step-3.7-Flash-FP8 Native FP8 checkpoint — runs on 8xH200/B200 with TP8+EP - nvfp4: precision=nvfp4, vram_minimum_gb=119, model_id=stepfun-ai/Step-3.7-Flash-NVFP4 NVFP4 quantized — Blackwell only; TP4+EP with FP8 KV cache ## Recommended base flags ```bash vllm serve 'stepfun-ai/Step-3.7-Flash' \ --trust-remote-code \ --enable-expert-parallel \ --disable-cascade-attn ``` ## Guide ## Overview [Step-3.7-Flash](https://huggingface.co/stepfun-ai/Step-3.7-Flash) is a 198B-parameter sparse Mixture-of-Experts vision-language model from [StepFun](https://www.stepfun.com/company), pairing a 196B language backbone with a 1.8B perception encoder. It activates ~11B parameters per token and supports a 256k context window with three selectable reasoning levels (low / medium / high). Key highlights: - **Multimodal Understanding**: Native vision encoder for single and multi-image inputs alongside text - **Hybrid Attention Architecture**: Interleaves Sliding Window Attention (512-token window) and Global Attention at a 3:1 ratio - **Sparse MoE**: 11B active parameters out of 198B total - **Multi-Layer MTP**: 3-way Multi-Token Prediction (MTP-3) for low-latency reasoning chains Available precisions: - [stepfun-ai/Step-3.7-Flash](https://huggingface.co/stepfun-ai/Step-3.7-Flash) (BF16) - [stepfun-ai/Step-3.7-Flash-FP8](https://huggingface.co/stepfun-ai/Step-3.7-Flash-FP8) - [stepfun-ai/Step-3.7-Flash-NVFP4](https://huggingface.co/stepfun-ai/Step-3.7-Flash-NVFP4) (Blackwell only) ## Prerequisites - **vLLM version:** nightly (the model registry hasn't shipped in a stable release yet) - **Hardware:** 8xH200/B200 for BF16 and FP8; 4xB200 for NVFP4 ### Install vLLM (nightly) ```bash uv venv source .venv/bin/activate uv pip install -U vllm --pre \ --extra-index-url https://wheels.vllm.ai/nightly ``` Or via Docker: ```bash docker pull vllm/vllm-openai:stepfun37 ``` ## Launching the Server ### BF16 ```bash vllm serve stepfun-ai/Step-3.7-Flash \ --served-model-name step3p7-flash \ --tensor-parallel-size 8 \ --enable-expert-parallel \ --disable-cascade-attn \ --reasoning-parser step3p5 \ --tool-call-parser step3p5 \ --enable-auto-tool-choice \ --speculative-config '{"method": "mtp", "num_speculative_tokens": 3}' \ --trust-remote-code ``` ### FP8 ```bash vllm serve stepfun-ai/Step-3.7-Flash-FP8 \ --served-model-name step3p7-flash \ --tensor-parallel-size 8 \ --enable-expert-parallel \ --disable-cascade-attn \ --reasoning-parser step3p5 \ --tool-call-parser step3p5 \ --enable-auto-tool-choice \ --speculative-config '{"method": "mtp", "num_speculative_tokens": 3}' \ --trust-remote-code ``` ### NVFP4 (Blackwell only) Requires modelopt quantization and FP8 KV cache alignment. ```bash vllm serve stepfun-ai/Step-3.7-Flash-NVFP4 \ --served-model-name step3p7 \ --tensor-parallel-size 4 \ --gpu-memory-utilization 0.9 \ --enable-expert-parallel \ --quantization modelopt \ --kv-cache-dtype fp8 \ --reasoning-parser step3p5 \ --tool-call-parser step3p5 \ --enable-auto-tool-choice \ --async-scheduling \ --trust-remote-code ``` ### DGX Station Single-GPU The [DGX Station](https://www.nvidia.com/en-us/products/workstations/dgx-station/) ships a single GB300 Grace-Blackwell Ultra Superchip with 252 GB of HBM3e, so the FP8 and NVFP4 checkpoints both fit entirely in VRAM on one GPU (BF16 at ~475 GB does not). Use the dedicated `vllm/vllm-openai:stepfun37` image and serve on a single GPU: ```bash vllm serve stepfun-ai/Step-3.7-Flash-FP8 \ --served-model-name step3p7-flash \ --tensor-parallel-size 1 \ --gpu-memory-utilization 0.95 \ --kv-cache-dtype fp8 \ --reasoning-parser step3p5 \ --tool-call-parser step3p5 \ --enable-auto-tool-choice \ --trust-remote-code ``` Swap in `stepfun-ai/Step-3.7-Flash-NVFP4` to run the NVFP4 checkpoint instead. Or launch the prebuilt container directly: ```bash docker run -d --name vllm-server \ --gpus all --ipc host \ --ulimit memlock=-1 --ulimit stack=67108864 \ -p 8000:8000 \ -e HF_TOKEN="$HF_TOKEN" \ -v "$HOME/.cache/huggingface/hub:/root/.cache/huggingface/hub" \ vllm/vllm-openai:stepfun37 \ stepfun-ai/Step-3.7-Flash-FP8 \ --gpu-memory-utilization 0.95 \ --trust-remote-code \ --reasoning-parser step3p5 \ --enable-auto-tool-choice \ --tool-call-parser step3p5 \ --kv-cache-dtype fp8 ``` See the [NVIDIA build DGX Station instructions](https://build.nvidia.com/station/vllm/instructions) for the full container setup. ## Benchmarking ```bash vllm bench serve \ --backend vllm \ --model stepfun-ai/Step-3.7-Flash \ --endpoint /v1/completions \ --dataset-name random \ --random-input 2048 \ --random-output 1024 \ --max-concurrency 10 \ --num-prompt 100 ``` ## Troubleshooting - **MoE kernel tuning:** See [tune-moe-kernel](https://github.com/vllm-project/recipes/blob/main/Qwen/Qwen3-Next.md#tune-moe-kernel) to tune Triton kernels for your hardware. - **NVFP4 + TP > 4:** The author recommends TP4+EP for NVFP4. Higher TP isn't validated. - **Cascade attention:** Always pass `--disable-cascade-attn` — the hybrid SWA/GA schedule is not compatible with cascade attention in vLLM. ## References - [Model card](https://huggingface.co/stepfun-ai/Step-3.7-Flash) - [FP8 checkpoint](https://huggingface.co/stepfun-ai/Step-3.7-Flash-FP8) - [NVFP4 checkpoint](https://huggingface.co/stepfun-ai/Step-3.7-Flash-NVFP4) - [StepFun](https://www.stepfun.com/company) --- # zai-org/GLM-GA on vLLM Source: https://recipes.vllm.ai/zai-org/GLM-GA Provider: GLM (Z-AI) Updated: 2026-05-27 Architecture: dense Parameters: 10B Context length: 131,072 Min vLLM: 0.21.0 Tasks: multimodal Summary: GLM-GA dense vision-language model (~10B) — image and video understanding with 128K context and dedicated Glmga video processor (fps=2, up to 640 frames) ## Variants - default: precision=bf16, vram_minimum_gb=29 Full precision BF16 — runs on a single H100/H200 ## Recommended base flags ```bash vllm serve 'zai-org/GLM-GA' \ --trust-remote-code \ --allowed-local-media-path \ / \ --mm-processor-cache-type \ shm \ --reasoning-parser \ glm45 ``` Environment: ```bash VLLM_VIDEO_LOADER_BACKEND='glm4_6v' ``` ## Guide ## Overview GLM-GA is a dense vision-language model (~10B parameters) based on the GLM-4.6V-Flash architecture. It shares the same `Glm4vForConditionalGeneration` model class as GLM-4.6V but uses dedicated `GlmgaImageProcessor` and `GlmgaVideoProcessor` sub-processors. The key difference is in video processing: GLM-GA samples at a fixed 2 fps and supports up to 640 frames, enabling long-video understanding. ## Prerequisites - **vLLM version:** >= 0.21.0 - **Transformers:** latest (required for Glmga processor support) - **Hardware:** 1x H100/H200 or equivalent (BF16) ### Install vLLM ```bash uv venv source .venv/bin/activate uv pip install -U vllm --torch-backend auto uv pip install git+https://github.com/huggingface/transformers.git ``` ## Launching the Server ### Single GPU ```bash export VLLM_VIDEO_LOADER_BACKEND=glm4_6v vllm serve zai-org/GLM-GA \ --tool-call-parser glm45 \ --reasoning-parser glm45 \ --enable-auto-tool-choice \ --allowed-local-media-path / \ --mm-processor-cache-type shm ``` ### Tuning Tips - `--max-model-len=65536` is a common default; you can push to 131072. - `--max-num-batched-tokens=32768` for prompt-heavy workloads. - `--mm-processor-cache-type shm` for efficient vision processing. ## Client Usage ### Image Understanding ```python from openai import OpenAI client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1") resp = client.chat.completions.create( model="zai-org/GLM-GA", messages=[{ "role": "user", "content": [ {"type": "image_url", "image_url": {"url": "https://example.com/image.png"}}, {"type": "text", "text": "Describe the image."} ] }], max_tokens=512, ) print(resp.choices[0].message.content) ``` ### Video Understanding ```python from openai import OpenAI client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1") resp = client.chat.completions.create( model="zai-org/GLM-GA", messages=[{ "role": "user", "content": [ {"type": "video_url", "video_url": {"url": "https://example.com/video.mp4"}}, {"type": "text", "text": "Summarize what happens in this video."} ] }], max_tokens=1024, ) print(resp.choices[0].message.content) ``` ## Video Processing Details GLM-GA uses a dedicated `GlmgaVideoProcessor` that differs from GLM-4.6V: - Fixed 2 fps sampling (GLM-4.6V uses dynamic fps based on duration) - Up to 640 frames per video - Frame upsampling uses `math.floor` alignment matching the HuggingFace reference ## Troubleshooting - **Long-context memory:** At 128K context, tune `--max-num-batched-tokens` and `--gpu-memory-utilization` to prevent OOM. - **Video loading errors:** Ensure OpenCV or PyAV is installed for video decoding. ## References - [Model card](https://huggingface.co/zai-org/GLM-GA) - [GLM-4.6V-Flash](https://huggingface.co/zai-org/GLM-4.6V-Flash) (base architecture) - [vLLM multimodal inputs guide](https://docs.vllm.ai/en/latest/features/multimodal_inputs.html) --- # mistralai/Mistral-Medium-3.5-128B on vLLM Source: https://recipes.vllm.ai/mistralai/Mistral-Medium-3.5-128B Provider: Mistral AI Released: 2026-03-31T09:50:20.000Z Updated: 2026-05-26 Architecture: dense Parameters: 128B Context length: 262,144 Min vLLM: 0.18.0 Tasks: multimodal Verified hardware: MI300X Summary: Mistral Medium 3.5 (128B) dense vision-language model with native FP8 weights and 256K context ## Variants - default: precision=fp8, vram_minimum_gb=154 Native FP8 weights (vision tower / projector / lm_head kept in BF16); recommended on 8xH200, 8xMI300X, or 4xB200 ## Recommended base flags ```bash vllm serve 'mistralai/Mistral-Medium-3.5-128B' \ --tokenizer_mode \ mistral \ --config_format \ mistral \ --load_format \ mistral ``` ## Guide ## Overview Mistral-Medium-3.5 is a 128B dense vision-language model from Mistral AI. The weights ship pre-quantized to FP8 (E4M3) with the vision tower, multimodal projector, and `lm_head` retained in BF16. Image input is supported up to 1540x1540 (Pixtral-style encoder, patch size 14). Context length is 256K via YaRN scaling (factor 64x over the 4K base). Reasoning is opt-in per request via `reasoning_effort: "high"` — when set, the model emits `[THINK]...[/THINK]` blocks that the Mistral reasoning parser surfaces as `message.reasoning`. Tool calling uses the `[AVAILABLE_TOOLS]` / `[TOOL_CALLS]` chat-template tokens. ## Prerequisites - Hardware: 8xH200 (recommended), 4xB200 or 2-8xMI300X; single B200 / MI300X also fits the weights (~134 GB raw) but leaves little room for the 256K KV cache - see below. - vLLM **nightly** (Mistral 3.5 architecture support has not yet shipped in a stable release). ### Install vLLM ```bash uv venv source .venv/bin/activate uv pip install -U vllm --torch-backend=auto \ --extra-index-url https://wheels.vllm.ai/nightly ``` This pulls in `mistral_common >= 1.11.1` and `transformers >= 5.4.0` automatically. ## Launch command 8xH200 (or 8xB200): ```bash vllm serve mistralai/Mistral-Medium-3.5-128B \ --tensor-parallel-size 8 \ --tokenizer_mode mistral --config_format mistral --load_format mistral \ --enable-auto-tool-choice --tool-call-parser mistral \ --reasoning-parser mistral ``` Useful flags: - `--max-model-len`: default 262144; lower it (e.g. 65536) to free VRAM for larger batch sizes on tighter GPU pools. - `--language-model-only`: skip the vision encoder entirely for text-only workloads. - `--mm-encoder-tp-mode data`: run the small vision encoder data-parallel instead of tensor-parallel — avoids the all-reduce overhead. - `--limit-mm-per-prompt.image N`: cap images per request. ## EAGLE speculative decoding Mistral ships a dedicated EAGLE draft head at [`mistralai/Mistral-Medium-3.5-128B-EAGLE`](https://huggingface.co/mistralai/Mistral-Medium-3.5-128B-EAGLE). It is **not** included in the default config — toggle the `spec_decoding` feature. Mistral's recommended serve command (from the EAGLE model card): ```bash vllm serve mistralai/Mistral-Medium-3.5-128B --tensor-parallel-size 8 \ --tool-call-parser mistral --enable-auto-tool-choice --reasoning-parser mistral \ --max-num-batched-tokens 16384 --max-num-seqs 128 --gpu-memory-utilization 0.8 \ --speculative-config '{"model":"mistralai/Mistral-Medium-3.5-128B-EAGLE","num_speculative_tokens":3,"method":"eagle","max_model_len":"65536"}' ``` The draft model is a 2-layer Mistral-style head trained on the 128B target; it shares the tokenizer and runs at TP=8 alongside the target. ## Client usage Reasoning + tool calling against the OpenAI-compatible endpoint: ```python from openai import OpenAI client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1") resp = client.chat.completions.create( model="mistralai/Mistral-Medium-3.5-128B", messages=[{"role": "user", "content": "Plan a 3-day Paris trip."}], extra_body={"reasoning_effort": "high"}, temperature=0.7, max_tokens=4096, ) msg = resp.choices[0].message print("reasoning:", msg.reasoning) print("answer:", msg.content) ``` Image input (vision): ```python resp = client.chat.completions.create( model="mistralai/Mistral-Medium-3.5-128B", messages=[{ "role": "user", "content": [ {"type": "image_url", "image_url": {"url": "https://..."}}, {"type": "text", "text": "Describe this image."}, ], }], max_tokens=512, ) ``` ### AMD MI300X (ROCm) The command builder fits the model on one GPU on MI300X. This is prone to fail due to OOM. Use the following instead. **Single GPU** - Use `--tensor-parallel-size 1` and limit context, e.g. `--max-model-len 131072`. Without that limit, KV-cache allocation fails at the default **262144** context even in text-only mode. You can also reduce `--max-num-batched-tokens` or change `--gpu-memory-utilization` if you still hit OOM. Here is a text-only example: ```bash docker run --device=/dev/kfd --device=/dev/dri \ --security-opt seccomp=unconfined --group-add video \ --privileged --ipc=host -p 8000:8000 \ -e VLLM_ROCM_USE_AITER=1 \ -e SAFETENSORS_FAST_GPU=1 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ vllm/vllm-openai-rocm:nightly mistralai/Mistral-Medium-3.5-128B \ --tokenizer_mode mistral \ --config_format mistral \ --load_format mistral \ --tensor-parallel-size 1 \ --no-enable-prefix-caching \ --enable-auto-tool-choice \ --tool-call-parser mistral \ --reasoning-parser mistral \ --max-model-len 131072 \ --language-model-only ``` **Text-only** - Set `--tensor-parallel-size` to the number of GPUs you wish to use: ```bash docker run --device=/dev/kfd --device=/dev/dri \ --security-opt seccomp=unconfined --group-add video \ --privileged --ipc=host -p 8000:8000 \ -e VLLM_ROCM_USE_AITER=1 \ -e SAFETENSORS_FAST_GPU=1 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ vllm/vllm-openai-rocm:nightly mistralai/Mistral-Medium-3.5-128B \ --tokenizer_mode mistral \ --config_format mistral \ --load_format mistral \ --tensor-parallel-size 8 \ --no-enable-prefix-caching \ --enable-auto-tool-choice \ --tool-call-parser mistral \ --reasoning-parser mistral \ --language-model-only ``` For `lm_eval` against a text-only server, pass `fix_mistral_regex=True` in `--model_args` (Mistral 3.5 tokenizer quirk). **Multimodal (text + image)** - Enable the image inputs and set the limit for images per prompt, here 1, as necessary: ```bash docker run --device=/dev/kfd --device=/dev/dri \ --security-opt seccomp=unconfined --group-add video \ --privileged --ipc=host -p 8000:8000 \ -e SAFETENSORS_FAST_GPU=1 \ -e VLLM_ROCM_USE_AITER=1 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ vllm/vllm-openai-rocm:nightly mistralai/Mistral-Medium-3.5-128B \ --tokenizer_mode mistral --config_format mistral --load_format mistral \ --tensor-parallel-size 8 \ --no-enable-prefix-caching \ --enable-mm \ --limit-mm-per-prompt '{"image": 1}' \ --enable-auto-tool-choice --tool-call-parser mistral \ --reasoning-parser mistral ``` ### MI300X benchmarking (text-only) Serving benchmark: `vllm bench serve` with 100 requests, max concurrency 32, 1024 input + 1024 output tokens per request. Accuracy: `lm_eval` GSM8k (5-shot), `local-completions` backend. | TP | Output tok/s | Mean TTFT (ms) | Mean TPOT (ms) | GSM8k flexible | GSM8k strict | |----|-------------:|---------------:|---------------:|---------------:|-------------:| | 1 | 438 | 5719 | 55.2 | 0.928 | 0.876 | | 2 | 530 | 5310 | 45.0 | 0.923 | 0.867 | | 4 | 729 | 3317 | 33.0 | 0.928 | 0.873 | | 8 | 915 | 2522 | 26.1 | 0.929 | 0.876 | ## Troubleshooting - **OOM at full 256K context** on H200 or MI300X: drop `--max-model-len` to **131072** or **65536**, or set `--language-model-only` if you don't need vision. - `reasoning_effort` rejected: only `"none"` and `"high"` are accepted by the chat template — anything else raises an exception. ## References - [Model card](https://huggingface.co/mistralai/Mistral-Medium-3.5-128B) --- # mistralai/Ministral-3-14B-Instruct-2512 on vLLM Source: https://recipes.vllm.ai/mistralai/Ministral-3-14B-Instruct-2512 Provider: Mistral AI Released: 2025-10-31T08:43:24.000Z Updated: 2026-05-25 Architecture: dense Parameters: 14B Context length: 262,144 Min vLLM: 0.11.0 Tasks: multimodal Verified hardware: H200, MI300X, MI325X, MI355X Summary: Ministral 3 Instruct family (3B/8B/14B) with FP8 weights, vision support, and 256K context ## Variants - default: precision=fp8, vram_minimum_gb=17 Native FP8 weights (14B), fits on 1x H200 - 8b: precision=fp8, vram_minimum_gb=12, model_id=mistralai/Ministral-3-8B-Instruct-2512 Smaller 8B variant with independent embedding/output layers - 3b: precision=fp8, vram_minimum_gb=6, model_id=mistralai/Ministral-3-3B-Instruct-2512 Smallest 3B variant with tied embeddings - fp8: precision=fp8, vram_minimum_gb=17, model_id=mistralai/Ministral-3-14B-Instruct-2512-FP8 FP8 quantized weights for Hopper/Blackwell GPUs - redhat_fp8: precision=fp8, vram_minimum_gb=17, model_id=RedHatAI/Ministral-3-14B-Instruct-2512-FP8-dynamic RedHatAI FP8 (E4M3) with dynamic per-token activation quantization - nvfp4: precision=nvfp4, vram_minimum_gb=9, model_id=RedHatAI/Ministral-3-14B-Instruct-2512-NVFP4 RedHatAI NVFP4 quantization ## Recommended base flags ```bash vllm serve 'mistralai/Ministral-3-14B-Instruct-2512' \ --tokenizer_mode \ mistral \ --config_format \ mistral \ --load_format \ mistral ``` ## Guide ## Overview Ministral-3 Instruct comes with FP8 weights in 3 different sizes: - 3B: tied embeddings (shares embedding and output layers) - 8B and 14B: independent embedding and output layers Each variant has vision support and a 256K context length. Smaller models offer faster inference at the cost of lower quality; pick the best trade-off for your use case. ## Prerequisites - Hardware: 1x H200 (sufficient for all three sizes thanks to FP8 weights); 1x MI300X (verified) / MI325X / MI355X - vLLM >= 0.11.0 ### Install vLLM ```bash uv venv source .venv/bin/activate uv pip install -U vllm --torch-backend auto ``` ## Launch command ```bash vllm serve mistralai/Ministral-3-14B-Instruct-2512 \ --tokenizer_mode mistral --config_format mistral --load_format mistral \ --enable-auto-tool-choice --tool-call-parser mistral ``` For 8B: `mistralai/Ministral-3-8B-Instruct-2512` For 3B: `mistralai/Ministral-3-3B-Instruct-2512` - `enable-auto-tool-choice`: required for tool usage - `tool-call-parser mistral`: required for tool usage - `--max-model-len` defaults to `262144`; reduce to save memory - `--max-num-batched-tokens` balances throughput and latency ### AMD (MI300X / MI325X / MI355X) Verified on an 8-GPU MI300X node with **TP=1** per variant. **3B** ```bash docker run --device=/dev/kfd --device=/dev/dri \ --security-opt seccomp=unconfined --group-add video \ --privileged --ipc=host -p 8000:8000 \ -e VLLM_ROCM_USE_AITER=1 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ vllm/vllm-openai-rocm:latest mistralai/Ministral-3-3B-Instruct-2512 \ --tokenizer_mode mistral \ --tensor-parallel-size 1 \ --config_format mistral \ --load_format mistral \ --enable-auto-tool-choice \ --tool-call-parser mistral ``` **8B** ```bash docker run --device=/dev/kfd --device=/dev/dri \ --security-opt seccomp=unconfined --group-add video \ --privileged --ipc=host -p 8000:8000 \ -e VLLM_ROCM_USE_AITER=1 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ vllm/vllm-openai-rocm:latest mistralai/Ministral-3-8B-Instruct-2512 \ --tokenizer_mode mistral \ --tensor-parallel-size 1 \ --config_format mistral \ --load_format mistral \ --max-model-len auto \ --max-num-batched-tokens 8192 \ --enable-auto-tool-choice \ --tool-call-parser mistral ``` **14B** ```bash docker run --device=/dev/kfd --device=/dev/dri \ --security-opt seccomp=unconfined --group-add video \ --privileged --ipc=host -p 8000:8000 \ -e VLLM_ROCM_USE_AITER=1 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ vllm/vllm-openai-rocm:latest mistralai/Ministral-3-14B-Instruct-2512 \ --tokenizer_mode mistral \ --tensor-parallel-size 1 \ --config_format mistral \ --load_format mistral \ --max-num-seqs 256 \ --max-model-len auto \ --gpu-memory-utilization 0.95 \ --max-num-batched-tokens 8192 \ --enable-auto-tool-choice \ --tool-call-parser mistral ``` ## Client Usage Vision reasoning example: ```python from datetime import datetime, timedelta from openai import OpenAI from huggingface_hub import hf_hub_download client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1") model = client.models.list().data[0].id def load_system_prompt(repo_id, filename): path = hf_hub_download(repo_id=repo_id, filename=filename) with open(path) as f: prompt = f.read() today = datetime.today().strftime("%Y-%m-%d") yesterday = (datetime.today() - timedelta(days=1)).strftime("%Y-%m-%d") return prompt.format(name=repo_id.split("/")[-1], today=today, yesterday=yesterday) SYSTEM_PROMPT = load_system_prompt(model, "SYSTEM_PROMPT.txt") image_url = "https://static.wikia.nocookie.net/essentialsdocs/images/7/70/Battle.png/revision/latest?cb=20220523172438" response = client.chat.completions.create( model=model, messages=[ {"role": "system", "content": SYSTEM_PROMPT}, {"role": "user", "content": [ {"type": "text", "text": "What action should I take here?"}, {"type": "image_url", "image_url": {"url": image_url}}, ]}, ], temperature=0.15, max_tokens=262144, ) print(response.choices[0].message.content) ``` Function calling and text-only examples follow a similar OpenAI-compatible pattern. ## Benchmarking (MI300X verification) Serving benchmarks used `vllm bench serve` with random 1024-token input/output, `--max-concurrency 32`, and `--num-prompts 100` against each variant above. Accuracy used `lm-eval` GSM8K (5-shot, `flexible-extract` / `strict-match` filters). ### Throughput | Variant | Output tok/s | Mean TTFT (ms) | Mean TPOT (ms) | |---------|-------------:|---------------:|---------------:| | 3B | 3842 | 288 | 6.42 | | 8B | 2468 | 1117 | 9.48 | | 14B | 1941 | 1229 | 12.22 | ### GSM8K (5-shot `exact_match`) | Variant | flexible-extract | strict-match | |---------|-----------------:|-------------:| | 3B | 0.7786 ± 0.0114 | 0.7445 ± 0.0120 | | 8B | 0.8560 ± 0.0097 | 0.8491 ± 0.0099 | | 14B | 0.8795 ± 0.0090 | 0.8764 ± 0.0091 | **14B** full `vllm bench serve` output (TP=1, MI300X): ```shell ============ Serving Benchmark Result ============ Successful requests: 100 Failed requests: 0 Maximum request concurrency: 32 Benchmark duration (s): 52.76 Total input tokens: 102400 Total generated tokens: 102400 Request throughput (req/s): 1.90 Output token throughput (tok/s): 1940.79 Peak output token throughput (tok/s): 3126.00 Peak concurrent requests: 64.00 Total token throughput (tok/s): 3881.58 ---------------Time to First Token---------------- Mean TTFT (ms): 1228.72 Median TTFT (ms): 952.25 P99 TTFT (ms): 2925.25 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 12.22 Median TPOT (ms): 12.25 P99 TPOT (ms): 12.83 ---------------Inter-token Latency---------------- Mean ITL (ms): 12.22 Median ITL (ms): 11.78 P99 ITL (ms): 13.66 ================================================== ``` ## Troubleshooting - OOM: lower `--max-model-len` (e.g. 32768) or use the 3B/8B variant. ## References - [Ministral-3-14B-Instruct](https://huggingface.co/mistralai/Ministral-3-14B-Instruct-2512) - [Ministral-3-8B-Instruct](https://huggingface.co/mistralai/Ministral-3-8B-Instruct-2512) - [Ministral-3-3B-Instruct](https://huggingface.co/mistralai/Ministral-3-3B-Instruct-2512) --- # inclusionAI/Ring-2.6-1T on vLLM Source: https://recipes.vllm.ai/inclusionAI/Ring-2.6-1T Provider: inclusionAI Released: 2026-05-14T08:10:02.000Z Updated: 2026-05-15 Architecture: moe Parameters: 1T Context length: 131,072 Min vLLM: 0.20.2 Tasks: text Summary: Ring-2.6-1T (BailingMoeV2_5) FP8 thinking model with 1T total / 50B active params, hybrid linear + MLA attention, 128K context ## Variants - default: precision=fp8, vram_minimum_gb=1200 FP8 weights with TP=8 on B300 or 8x MI300X/MI355X-class nodes ## Recommended base flags ```bash vllm serve 'inclusionAI/Ring-2.6-1T' \ --trust-remote-code \ --tensor-parallel-size \ 8 ``` ## Guide ## Overview [Ring-2.6-1T](https://huggingface.co/inclusionAI/Ring-2.6-1T) is inclusionAI's BailingMoeV2_5 FP8 trillion-scale **thinking** model (1T total / 50B active parameters) with hybrid linear + MLA attention and a 128K context window. It is the reasoning-focused counterpart to [Ling-2.6-1T](https://huggingface.co/inclusionAI/Ling-2.6-1T) in the Ring 2.6 series and emits explicit `...` traces; the chat template accepts a `reasoning_effort` field (default `high`). vLLM 0.20.1 shipped the BailingMoeV2.5 MLA RoPE fix (vllm-project/vllm#41185) that is load-bearing for this architecture — pin v0.20.2 or newer. ## Deployment Configurations ### Docker (AMD MI300X / MI325X / MI355X, TP=8) TP=8 fits the model-derived 128K context on an MI300X-class node; MI325X and MI355X have larger per-GPU HBM and more headroom. ```bash docker run --rm -it \ --cap-add=SYS_PTRACE \ --ipc=host \ --privileged=true \ --shm-size=128GB \ --network=host \ --device=/dev/kfd \ --device=/dev/dri \ --group-add video \ -e VLLM_ROCM_USE_AITER=1 \ vllm/vllm-openai-rocm:v0.28.0 \ inclusionAI/Ring-2.6-1T \ --tensor-parallel-size 8 \ --trust-remote-code ``` ### Pip (NVIDIA B300, TP=8) ```bash uv venv && source .venv/bin/activate uv pip install -U vllm --torch-backend auto vllm serve inclusionAI/Ring-2.6-1T \ --trust-remote-code \ --tensor-parallel-size 8 ``` ## Client Usage ### Text Generation ```python from openai import OpenAI client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY") response = client.chat.completions.create( model="inclusionAI/Ring-2.6-1T", messages=[{"role": "user", "content": "Prove there are infinitely many primes."}], max_tokens=4096, temperature=0.6, extra_body={"chat_template_kwargs": {"reasoning_effort": "high"}}, ) print(response.choices[0].message.content) ``` Set `reasoning_effort` to `low` / `medium` / `high` to control the depth of thinking. The model wraps its reasoning trace in `...` before emitting the final answer. ## References - [Ring-2.6-1T on Hugging Face](https://huggingface.co/inclusionAI/Ring-2.6-1T) - [vLLM PR #41185 — BailingMoeV2.5 MLA RoPE fix](https://github.com/vllm-project/vllm/pull/41185) - [vLLM ROCm GPU install docs](https://docs.vllm.ai/en/latest/getting_started/installation/gpu/?device=rocm) --- # internlm/Intern-S2-Preview on vLLM Source: https://recipes.vllm.ai/internlm/Intern-S2-Preview Provider: InternLM Released: 2026-05-15T03:49:25.000Z Updated: 2026-05-15 Architecture: moe Parameters: 36B Context length: 262,144 Min vLLM: 0.22.0 Tasks: multimodal, text Verified hardware: H200, GB200 NVL4 Summary: Scientific multimodal MoE (36B total / 3B active) continued pre-trained from Qwen3.5 — hybrid linear/full attention, 262K context, MTP-accelerated reasoning. BF16 and FP8 checkpoints. ## Variants - default: precision=bf16, vram_minimum_gb=87 BF16 on 1x H200 or 2x H100/H800 (deployment guide uses TP=2) - fp8: precision=fp8, vram_minimum_gb=44, model_id=internlm/Intern-S2-Preview-FP8 Official DeepSeek-style block FP8 (128x128, ue8m0 scales) — fits on a single H100/H200 ## Recommended base flags ```bash vllm serve 'internlm/Intern-S2-Preview' \ --trust-remote-code ``` ## Guide ## Overview [Intern-S2-Preview](https://huggingface.co/internlm/Intern-S2-Preview) is a scientific multimodal foundation model from Shanghai AI Laboratory, continued pre-trained from Qwen3.5. It packs 36B total / 3B active parameters across 256 experts with hybrid linear/full attention, supports a 262K context, and ships with built-in shared-weight MTP for fast reasoning. Beyond chat and reasoning, it adds vision and time-series modalities and improves agent capabilities for scientific workflows. ## Prerequisites - **vLLM version:** nightly build with `InternS2PreviewForConditionalGeneration` support ([PR #42705](https://github.com/vllm-project/vllm/pull/42705)). The architecture is not yet in any stable release. - **Hardware (BF16):** 1x H200 (141 GB) or 2x H100/H800 (80 GB) - **Hardware (FP8):** single H100/H200 - **Trust remote code:** required (custom modeling files ship in the repo) ### Install vLLM (nightly) ```bash uv venv source .venv/bin/activate uv pip install -U vllm --pre --extra-index-url https://wheels.vllm.ai/nightly ``` ## Launch commands ### Recommended — with MTP speculative decoding ```bash vllm serve internlm/Intern-S2-Preview \ --trust-remote-code \ --tensor-parallel-size 2 \ --reasoning-parser qwen3 \ --enable-auto-tool-choice \ --tool-call-parser qwen3_coder \ --speculative-config '{"method":"mtp","num_speculative_tokens":4}' ``` ### Basic serving (no MTP) ```bash vllm serve internlm/Intern-S2-Preview \ --trust-remote-code \ --tensor-parallel-size 2 \ --reasoning-parser qwen3 \ --enable-auto-tool-choice \ --tool-call-parser qwen3_coder ``` ### Long-context (YaRN, up to 512K) The base config sets `max_position_embeddings = 262144`. For longer contexts, override the RoPE config to enable YaRN: ```bash vllm serve internlm/Intern-S2-Preview \ --trust-remote-code \ --tensor-parallel-size 2 \ --max-model-len 512000 \ --reasoning-parser qwen3 \ --enable-auto-tool-choice \ --tool-call-parser qwen3_coder \ --hf-overrides '{"text_config": {"rope_parameters": {"mrope_interleaved": true, "mrope_section": [11, 11, 10], "rope_type": "yarn", "rope_theta": 10000000, "partial_rotary_factor": 0.25, "factor": 4.0, "original_max_position_embeddings": 262144}}}' ``` ## Client Usage Recommended sampling parameters from the model card: ```python from openai import OpenAI client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1") resp = client.chat.completions.create( model="internlm/Intern-S2-Preview", messages=[{"role": "user", "content": "Design a synthesis route for paracetamol."}], temperature=0.8, top_p=0.95, max_tokens=32768, extra_body={ "top_k": 50, "min_p": 0.0, "spaces_between_special_tokens": False, }, ) print(resp.choices[0].message.content) ``` ### Toggle thinking mode Thinking is enabled by default. Disable it per request: ```python resp = client.chat.completions.create( model="internlm/Intern-S2-Preview", messages=[{"role": "user", "content": "What is AGI?"}], temperature=0.8, top_p=0.95, extra_body={"chat_template_kwargs": {"enable_thinking": False}}, ) ``` > The model card notes: do **not** disable thinking mode for agentic tasks. ## Troubleshooting - **Unknown architecture `InternS2PreviewForConditionalGeneration`:** the handler landed via vLLM PR #42705 — use a nightly wheel or build from main once the PR merges. - **OOM at full 262K context:** drop `--max-model-len` to 65536 or 131072, or lower `--gpu-memory-utilization` headroom. ## References - [Model card](https://huggingface.co/internlm/Intern-S2-Preview) - [Deployment guide](https://huggingface.co/internlm/Intern-S2-Preview/blob/main/deployment_guide.md) - [Intern-S1 GitHub](https://github.com/InternLM/Intern-S1) - [vLLM support PR #42705](https://github.com/vllm-project/vllm/pull/42705) --- # inclusionAI/Ling-2.6-1T on vLLM Source: https://recipes.vllm.ai/inclusionAI/Ling-2.6-1T Provider: inclusionAI Released: 2026-04-29T03:19:36.000Z Updated: 2026-05-13 Architecture: moe Parameters: 1T Context length: 262,144 Min vLLM: 0.20.2 Tasks: text Verified hardware: B300, MI300X Summary: Ling-2.6-1T (BailingMoeV2_5) FP8 instruct model with 1T total / 50B active params, hybrid linear + MLA attention, 262K context ## Variants - default: precision=fp8, vram_minimum_gb=1200 FP8 weights with TP=8 on B300 or 8x MI300X/MI355X-class nodes ## Recommended base flags ```bash vllm serve 'inclusionAI/Ling-2.6-1T' \ --trust-remote-code \ --tensor-parallel-size \ 8 ``` ## Guide ## Overview [Ling-2.6-1T](https://huggingface.co/inclusionAI/Ling-2.6-1T) is inclusionAI's BailingMoeV2_5 FP8 flagship model with 1T total / 50B active parameters, hybrid linear + MLA attention, and a 262K context window. ## Deployment Configurations ### Docker (AMD MI300X / MI325X / MI355X, TP=8) TP=8 has been verified on an MI300X-class node at the model-derived 262K context. MI325X and MI355X have larger per-GPU HBM. ```bash docker run --rm -it \ --cap-add=SYS_PTRACE \ --ipc=host \ --privileged=true \ --shm-size=128GB \ --network=host \ --device=/dev/kfd \ --device=/dev/dri \ --group-add video \ -e VLLM_ROCM_USE_AITER=1 \ vllm/vllm-openai-rocm:v0.28.0 \ inclusionAI/Ling-2.6-1T \ --tensor-parallel-size 8 \ --trust-remote-code ``` ## Client Usage ### Text Generation ```python from openai import OpenAI client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY") response = client.chat.completions.create( model="inclusionAI/Ling-2.6-1T", messages=[{"role": "user", "content": "Write a poem about the ocean."}], max_tokens=512, temperature=0.7, ) print(response.choices[0].message.content) ``` ## References - [Ling-2.6-1T on Hugging Face](https://huggingface.co/inclusionAI/Ling-2.6-1T) - [vLLM ROCm GPU install docs](https://docs.vllm.ai/en/latest/getting_started/installation/gpu/?device=rocm) - [ROCm vLLM optimization guide](https://rocm.docs.amd.com/en/latest/how-to/rocm-for-ai/inference-optimization/vllm-optimization.html) --- # inclusionAI/Ling-2.6-flash on vLLM Source: https://recipes.vllm.ai/inclusionAI/Ling-2.6-flash Provider: inclusionAI Released: 2026-04-28T03:27:56.000Z Updated: 2026-05-13 Architecture: moe Parameters: 104B Context length: 131,072 Min vLLM: 0.20.2 Tasks: text Verified hardware: H200, MI300X Summary: Ling-2.6-flash (BailingMoeV2_5) instruct model with 104B total / 7.4B active params, hybrid linear + MLA attention, 128K context, optimized for agent workloads ## Variants - default: precision=bf16, vram_minimum_gb=250 BF16 weights with base TP=4; the guide shows a tested 2-GPU AMD command ## Recommended base flags ```bash vllm serve 'inclusionAI/Ling-2.6-flash' \ --trust-remote-code \ --tensor-parallel-size \ 4 ``` ## Guide ## Overview [Ling-2.6-flash](https://huggingface.co/inclusionAI/Ling-2.6-flash) is a BailingMoeV2_5 MoE instruct model with 104B total / 7.4B active parameters, hybrid linear + MLA attention, and a 131K context window. ## Deployment Configurations ### Docker (AMD MI300X / MI325X / MI355X, TP=2) MI300X / MI325X / MI355X GPUs have larger per-GPU HBM, so TP=2 fits the full 131K context. ```bash docker run --rm -it \ --cap-add=SYS_PTRACE \ --ipc=host \ --privileged=true \ --shm-size=128GB \ --network=host \ --device=/dev/kfd \ --device=/dev/dri \ --group-add video \ -e VLLM_ROCM_USE_AITER=1 \ vllm/vllm-openai-rocm:v0.28.0 \ inclusionAI/Ling-2.6-flash \ --tensor-parallel-size 2 \ --trust-remote-code ``` ## Client Usage ### Text Generation ```python from openai import OpenAI client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY") response = client.chat.completions.create( model="inclusionAI/Ling-2.6-flash", messages=[{"role": "user", "content": "Write a poem about the ocean."}], max_tokens=512, temperature=0.7, ) print(response.choices[0].message.content) ``` ## References - [Ling-2.6-flash on Hugging Face](https://huggingface.co/inclusionAI/Ling-2.6-flash) - [vLLM ROCm GPU install docs](https://docs.vllm.ai/en/latest/getting_started/installation/gpu/?device=rocm) - [ROCm vLLM optimization guide](https://rocm.docs.amd.com/en/latest/how-to/rocm-for-ai/inference-optimization/vllm-optimization.html) --- # mistralai/Mistral-Small-4-119B-2603 on vLLM Source: https://recipes.vllm.ai/mistralai/Mistral-Small-4-119B-2603 Provider: Mistral AI Released: 2026-01-23T13:14:14.000Z Updated: 2026-05-13 Architecture: moe Parameters: 119B Context length: 262,144 Min vLLM: 0.20.0 Tasks: multimodal Summary: Mistral Small 4 (119B MoE, 6.5B active) — multimodal hybrid instruct + reasoning model with native FP8 weights and 256K context ## Variants - default: precision=fp8, vram_minimum_gb=143 Native FP8 E4M3 weights (vision tower / projector / lm_head kept in BF16); recommended on 2xB200/H200 or MI300X - nvfp4: precision=nvfp4, vram_minimum_gb=72, model_id=mistralai/Mistral-Small-4-119B-2603-NVFP4 NVFP4 4-bit weights for B200-class GPUs (Marlin fallback on Hopper) - redhat_nvfp4: precision=nvfp4, vram_minimum_gb=72, model_id=RedHatAI/Mistral-Small-4-119B-2603-NVFP4 RedHatAI NVFP4 quantization ## Recommended base flags ```bash vllm serve 'mistralai/Mistral-Small-4-119B-2603' \ --max-model-len \ 262144 ``` ## Guide ## Overview Mistral Small 4 119B is a hybrid Mixture-of-Experts model from Mistral AI: **128 experts, 4 active per token** (plus 1 shared expert), 119B total parameters with **6.5B activated per token**. It unifies the capabilities of three earlier Mistral families — **Instruct**, **Reasoning** (formerly Magistral), and **Devstral** — into a single checkpoint, with per-request toggling between fast instant-reply and step-by-step reasoning via `reasoning_effort`. The weights ship pre-quantized to FP8 E4M3 (the vision tower, multimodal projector, and `lm_head` are kept in BF16). Multimodal input accepts text and image (Pixtral-style encoder, 1540x1540, patch size 14). Context length is **256K** via YaRN scaling (factor 128x over the 8K base) — the config advertises 1M positions but Mistral recommends serving at 256K. Mistral Small 4 also ships two companion checkpoints: - **NVFP4** ([`mistralai/Mistral-Small-4-119B-2603-NVFP4`](https://huggingface.co/mistralai/Mistral-Small-4-119B-2603-NVFP4)) — 4-bit `compressed-tensors` weights (~72 GB raw). - **EAGLE draft head** ([`mistralai/Mistral-Small-4-119B-2603-eagle`](https://huggingface.co/mistralai/Mistral-Small-4-119B-2603-eagle)) — a 2-layer Mistral-style draft trained on the 119B target; enable via the `spec_decoding` feature. ## Prerequisites - Hardware: 2xB200 / 2xH200 / 2xMI300X (FP8 default) or 1xB200 with reduced context (NVFP4). - vLLM **>= 0.20.0** — earlier releases load the model but trip on the Mistral tool-call / reasoning parsers fixed in PR #39217 (in 0.19.1+) and the grammar factory landed in 0.20.0. ### Install vLLM ```bash uv venv && source .venv/bin/activate uv pip install -U vllm --torch-backend=auto uv pip install -U "mistral_common>=1.11.0" transformers ``` Verify with `python -c "import mistral_common; print(mistral_common.__version__)"`. ## Launch command FP8 default (2xB200 / 2xH200): ```bash vllm serve mistralai/Mistral-Small-4-119B-2603 \ --max-model-len 262144 \ --tensor-parallel-size 2 \ --tool-call-parser mistral --enable-auto-tool-choice \ --reasoning-parser mistral \ --max-num-batched-tokens 16384 --max-num-seqs 128 \ --gpu-memory-utilization 0.8 ``` NVFP4 variant (single B200, or 2xB200 for full 256K context): ```bash vllm serve mistralai/Mistral-Small-4-119B-2603-NVFP4 \ --max-model-len 262144 \ --tensor-parallel-size 2 \ --tool-call-parser mistral --enable-auto-tool-choice \ --reasoning-parser mistral \ --max-num-batched-tokens 16384 --max-num-seqs 128 \ --gpu-memory-utilization 0.8 ``` Mistral publishes a custom Docker image [`mistralllm/vllm-ms4:latest`](https://hub.docker.com/r/mistralllm/vllm-ms4) with patched tool-call / reasoning parsing — use it if you're pinned to a vLLM version below 0.20.0 and hit either of those issues. ## EAGLE speculative decoding The EAGLE draft head is **not** included in the default config — toggle the `spec_decoding` feature (or pass `--speculative-config` directly): ```bash vllm serve mistralai/Mistral-Small-4-119B-2603 \ --max-model-len 262144 \ --tensor-parallel-size 2 \ --tool-call-parser mistral --enable-auto-tool-choice \ --reasoning-parser mistral \ --max-num-batched-tokens 16384 --max-num-seqs 128 \ --gpu-memory-utilization 0.8 \ --speculative-config '{ "model": "mistralai/Mistral-Small-4-119B-2603-eagle", "num_speculative_tokens": 3, "method": "eagle", "max_model_len": "65536" }' ``` ## Client usage Reasoning is opt-in per request via `reasoning_effort`. Only `"none"` and `"high"` are accepted — anything else raises an exception in the chat template. Recommended sampling: - `reasoning_effort="none"`: `temperature` 0.0–0.7 depending on task. - `reasoning_effort="high"`: `temperature=0.7`. ```python from openai import OpenAI client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1") resp = client.chat.completions.create( model="mistralai/Mistral-Small-4-119B-2603", messages=[{"role": "user", "content": "Plan a 3-day Paris trip."}], extra_body={"reasoning_effort": "high"}, temperature=0.7, max_tokens=4096, ) msg = resp.choices[0].message print("reasoning:", msg.reasoning) print("answer:", msg.content) ``` Image input (vision): ```python resp = client.chat.completions.create( model="mistralai/Mistral-Small-4-119B-2603", messages=[{ "role": "user", "content": [ {"type": "image_url", "image_url": {"url": "https://..."}}, {"type": "text", "text": "Describe this image."}, ], }], max_tokens=512, ) ``` Tool calling follows the standard OpenAI schema — the chat template emits `[AVAILABLE_TOOLS]` / `[TOOL_CALLS]` tokens which the `mistral` tool-call parser surfaces as `message.tool_calls`. ## Troubleshooting - **OOM at full 256K context** on 2xH200: drop `--max-model-len` to 131072 or 65536, or move to NVFP4. - **`reasoning_effort` rejected**: the chat template only accepts `"none"` and `"high"`. - **NVFP4 weight-loader errors** on older wheels: try Mistral's `mistralllm/vllm-ms4:latest` Docker image, which carries the parser / weight-loader fixes Mistral ships ahead of the upstream merge. ## References - [Model card](https://huggingface.co/mistralai/Mistral-Small-4-119B-2603) - [NVFP4 variant](https://huggingface.co/mistralai/Mistral-Small-4-119B-2603-NVFP4) - [EAGLE draft head](https://huggingface.co/mistralai/Mistral-Small-4-119B-2603-eagle) --- # mistralai/Voxtral-Mini-4B-Realtime-2602 on vLLM Source: https://recipes.vllm.ai/mistralai/Voxtral-Mini-4B-Realtime-2602 Provider: Mistral AI Released: 2026-01-21T17:22:02.000Z Updated: 2026-05-13 Architecture: dense Parameters: 4.4B Context length: 131,072 Min vLLM: 0.20.0 Tasks: multimodal Verified hardware: DGX Spark (GB10) Summary: Multilingual realtime speech transcription (13 languages) with a natively streaming causal audio encoder; configurable 80ms–2.4s transcription delay served via vLLM's Realtime API ## Variants - default: precision=bf16, vram_minimum_gb=16 BF16 weights — single 16GB+ GPU, full 131072 (~3h audio) context ## Recommended base flags ```bash vllm serve 'mistralai/Voxtral-Mini-4B-Realtime-2602' \ --tokenizer-mode \ mistral \ --compilation_config \ {"cudagraph_mode": "PIECEWISE"} ``` Environment: ```bash VLLM_DISABLE_COMPILE_CACHE='1' ``` ## Guide ## Overview Voxtral Mini 4B Realtime is Mistral AI's multilingual realtime speech transcription model — among the first open-source ASR systems to hit accuracy comparable to offline models with a **<500ms** end-to-end delay. - **Architecture**: ≈3.4B Mistral text LM + ≈970M custom **causal** audio encoder, both using sliding-window attention to support "infinite" streaming. - **Languages**: 13 (English, French, Spanish, German, Russian, Chinese, Japanese, Italian, Portuguese, Dutch, Arabic, Hindi, Korean). - **Configurable delay**: any multiple of 80ms between 80ms and 1200ms, plus 2400ms as a standalone value. Default is **480ms**, the sweet spot Mistral identified between latency and accuracy. - **Context**: 131072 tokens (≈3h of audio at 80ms/token). ## Prerequisites - **Hardware**: a single GPU with **≥ 16 GB** VRAM (BF16 weights only). - **vLLM**: **>= 0.20.0** — the Voxtral Realtime architecture has been registered since v0.16.0, but v0.20.0 is the first stable release with the architecture documented in the supported-models list. ### Install vLLM and audio dependencies ```bash uv venv && source .venv/bin/activate uv pip install -U vllm --torch-backend=auto uv pip install -U "mistral-common[audio]>=1.9.0" transformers ``` Verify the audio extras pulled `mistral_common >= 1.9.0`: ```bash python -c "import mistral_common; print(mistral_common.__version__)" ``` ## Launch command ```bash VLLM_DISABLE_COMPILE_CACHE=1 vllm serve mistralai/Voxtral-Mini-4B-Realtime-2602 \ --tokenizer-mode mistral \ --compilation_config '{"cudagraph_mode": "PIECEWISE"}' ``` > `--tokenizer-mode mistral` is **required**: Voxtral Realtime's tokenizer > only loads through `mistral_common`. Omitting it raises a tokenizer > initialization error at startup. Once it starts you should see the Realtime API route registered: ``` Route: /v1/realtime, Endpoint: realtime_endpoint ``` ### Tuning flags - `--max-num-batched-tokens` — balance throughput vs latency (higher means more throughput at the cost of per-request latency). - `--max-model-len` — defaults to 131072 (≈3h). Reduce it if you know your sessions are shorter; this cuts the memory reserved for pre-computed RoPE frequencies. As a rule of thumb, one text token ≈ 80ms of audio, so a 1h meeting needs `--max-model-len >= 3600/0.08 = 45000`. ## Client usage ### Recommended settings - Always set `temperature=0.0`. - Use **WebSockets** against `/v1/realtime` for streaming audio sessions. - Adjust the transcription delay by editing the `transcription_delay_ms` field in the model's [`tekken.json`](https://huggingface.co/mistralai/Voxtral-Mini-4B-Realtime-2602/blob/main/tekken.json) to any multiple of 80ms in `[80, 1200]`, or to `2400`. ### Stream an audio file See vLLM's [Realtime audio file client example](https://docs.vllm.ai/en/latest/examples/online_serving/openai_realtime_client/). ### Live microphone (Gradio demo) See vLLM's [Realtime microphone client example](https://docs.vllm.ai/en/latest/examples/online_serving/openai_realtime_microphone_client/) for an end-to-end live-transcription UI. ## Benchmarks (Fleurs, average WER) | Delay | AVG | English | French | Chinese | Japanese | |---------|--------|---------|--------|---------|----------| | 160ms | 12.60% | 6.46% | 9.75% | 17.67% | 19.17% | | 240ms | 10.80% | 5.91% | 8.00% | 13.84% | 15.17% | | **480ms** | **8.72%** | **4.90%** | **6.42%** | **10.45%** | **9.59%** | | 960ms | 7.70% | 4.34% | 5.68% | 8.99% | 6.80% | | 2400ms | 6.73% | 4.05% | 5.23% | 8.48% | 5.50% | At **480ms** Voxtral Mini Realtime matches Mistral's offline Voxtral Transcribe 2.0 on Long-form English and Short-form English benchmarks (within 1 WER point on TEDLIUM, Meanwhile, AMI IHM, etc.). ## Troubleshooting - **`Route: /v1/realtime` not registered** — your vLLM is < 0.16.0. Upgrade to 0.20.0+. - **Tokenizer initialization error** — you forgot `--tokenizer-mode mistral`. Voxtral Realtime's tokenizer can only load through `mistral_common`. - **`mistral_common` import error / wrong version** — install with the audio extras: `pip install -U "mistral-common[audio]>=1.9.0"`. - **Transformers v4 warning spam** — upgrade to Transformers v5 (`uv pip install -U transformers`), tracked in [vllm-project/vllm#34642](https://github.com/vllm-project/vllm/issues/34642). - **Hangs / crashes on long sessions** — known upstream issue, see [#39996](https://github.com/vllm-project/vllm/issues/39996) (encoder KV cache eviction) and [#38233](https://github.com/vllm-project/vllm/issues/38233) (multi-session encode). Restart sessions periodically as a workaround. ## References - [Mistral blog: Voxtral Transcribe 2](https://mistral.ai/news/voxtral-transcribe-2) - [vLLM Realtime API docs](https://docs.vllm.ai/en/latest/serving/openai_compatible_server/#realtime-api) - [vLLM streaming-input blog post](https://vllm.ai/blog/streaming-realtime.html) - [HuggingFace demo space](https://huggingface.co/spaces/mistralai/Voxtral-Mini-Realtime) --- # MiniMaxAI/MiniMax-M2 on vLLM Source: https://recipes.vllm.ai/MiniMaxAI/MiniMax-M2 Provider: MiniMax Released: 2025-10-22T13:45:10.000Z Updated: 2026-05-11 Architecture: moe Parameters: 230B Context length: 196,608 Min vLLM: 0.11.1 Tasks: text Verified hardware: H100, H200, MI300X, MI325X, MI355X Summary: MiniMax M2 MoE language model (230B total / 10B active) for coding, agent toolchains, and long-context reasoning — native FP8 checkpoint, with an NVFP4 variant for Blackwell ## Variants - default: precision=fp8, vram_minimum_gb=276 Native FP8 checkpoint — 4x H200/H20/H100 or 4x A100/A800 for weights, plus KV cache headroom - nvfp4: precision=nvfp4, vram_minimum_gb=138, model_id=RedHatAI/MiniMax-M2-NVFP4 NVFP4 (4-bit) quantized weights — requires Blackwell (B200/B300); roughly half the VRAM of the FP8 checkpoint ## Recommended base flags ```bash vllm serve 'MiniMaxAI/MiniMax-M2' \ --trust-remote-code ``` ## Guide ## Overview [MiniMax-M2](https://huggingface.co/MiniMaxAI/MiniMax-M2) is an advanced MoE language model from [MiniMax](https://www.minimax.io/). Highlights: - Superior intelligence — #1 among open-source models globally on math, science, coding, tool use - Advanced coding — multi-file edits, run-fix loops, test-validated repairs (SWE-Bench, Terminal-Bench) - Agent performance — plans and executes complex toolchains across shell, browser, and code - Efficient design — 10B active / 230B total for low latency and high throughput - 196K context length per sequence ## Prerequisites - **OS:** Linux - **Python:** 3.10 - 3.13 - **NVIDIA:** compute capability >= 7.0; ~220 GB for weights + 240 GB per 1M context tokens - **AMD:** MI300X / MI325X / MI350X / MI355X with ROCm 7.0+ ### Install vLLM (NVIDIA, stable) ```bash uv venv source .venv/bin/activate uv pip install -U vllm --torch-backend auto ``` ### Install vLLM (NVIDIA, nightly) If you hit corrupted output, upgrade to a nightly after commit `cf3eacfe58fa9e745c2854782ada884a9f992cf7`: ```bash uv pip install -U vllm --extra-index-url https://wheels.vllm.ai/nightly ``` ### Install vLLM (AMD ROCm) ```bash uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/ ``` ### Docker (NVIDIA, dedicated M2 image) ```bash docker run --gpus all \ -p 8000:8000 \ --ipc=host \ -v ~/.cache/huggingface:/root/.cache/huggingface \ vllm/vllm-openai:minimax27 MiniMaxAI/MiniMax-M2 \ --tensor-parallel-size 4 \ --tool-call-parser minimax_m2 \ --reasoning-parser minimax_m2 \ --enable-auto-tool-choice \ --trust-remote-code ``` ## Launching the Server ### NVIDIA — TP4 (4x H200/H20/H100 or 4x A100/A800) ```bash vllm serve MiniMaxAI/MiniMax-M2 \ --tensor-parallel-size 4 \ --tool-call-parser minimax_m2 \ --reasoning-parser minimax_m2 \ --enable-auto-tool-choice \ --trust-remote-code ``` Pure TP8 is not supported. For >4 GPUs use DP+EP or TP+EP: ### TP4+EP (recommended for H100) ```bash vllm serve MiniMaxAI/MiniMax-M2 \ --tensor-parallel-size 4 \ --enable-expert-parallel \ --tool-call-parser minimax_m2 \ --reasoning-parser minimax_m2 \ --enable-auto-tool-choice ``` ### DP8+EP ```bash vllm serve MiniMaxAI/MiniMax-M2 \ --data-parallel-size 8 \ --enable-expert-parallel \ --tool-call-parser minimax_m2 \ --reasoning-parser minimax_m2 \ --enable-auto-tool-choice ``` ### AMD ROCm — TP2 ```bash VLLM_ROCM_USE_AITER=1 vllm serve MiniMaxAI/MiniMax-M2 \ --tensor-parallel-size 2 \ --tool-call-parser minimax_m2 \ --reasoning-parser minimax_m2 \ --enable-auto-tool-choice \ --trust-remote-code ``` ## Benchmarking ```bash vllm bench serve \ --backend vllm \ --model MiniMaxAI/MiniMax-M2 \ --endpoint /v1/completions \ --dataset-name random \ --random-input 2048 \ --random-output 1024 \ --max-concurrency 10 \ --num-prompt 100 ``` ## Troubleshooting - **MiniMax QK norm fusion:** Latest vLLM applies the fusion automatically at runtime when the fused CUDA op is available; older vLLM builds used an explicit `fuse_minimax_qk_norm` compile pass. - **Corrupted output on stable release:** Upgrade to a nightly after commit `cf3eacfe58fa9e745c2854782ada884a9f992cf7`. - **DeepGEMM:** vLLM uses DeepGEMM by default; install via [install_deepgemm.sh](https://github.com/vllm-project/vllm/blob/main/tools/install_deepgemm.sh) when missing. - **AITER first launch:** AMD initial launch JIT-compiles CK-based FP8 MoE, RMSNorm, and activation kernels. Subsequent launches use cached kernels. ## Quantized Variant (NVFP4) [`RedHatAI/MiniMax-M2-NVFP4`](https://huggingface.co/RedHatAI/MiniMax-M2-NVFP4) is an NVFP4 (4-bit) checkpoint — roughly half the VRAM of the native FP8 checkpoint. Select the **nvfp4** variant above (it sets `--kv-cache-dtype fp8`; on vLLM v0.28.0+ the FlashInfer NVFP4 MoE kernels are selected automatically), or pass the repo id directly to `vllm serve`. NVFP4 requires a Blackwell GPU (compute capability 10.0+). ## References - [Model card](https://huggingface.co/MiniMaxAI/MiniMax-M2) - [NVFP4 variant](https://huggingface.co/RedHatAI/MiniMax-M2-NVFP4) - [MiniMax](https://www.minimax.io/) - [vLLM PR #43410 (automatic MiniMax QK norm fusion)](https://github.com/vllm-project/vllm/pull/43410) --- # PaddlePaddle/PaddleOCR-VL-1.5 on vLLM Source: https://recipes.vllm.ai/PaddlePaddle/PaddleOCR-VL-1.5 Provider: PaddlePaddle Released: 2026-01-28T12:43:01.000Z Updated: 2026-05-11 Architecture: dense Parameters: 0.9B Context length: 131,072 Min vLLM: 0.11.1 Tasks: multimodal Summary: PaddleOCR-VL-1.5 (0.9B) — next-gen compact VLM for document parsing; adds text spotting, seal recognition, and Tibetan/Bengali ## Variants - default: precision=bf16, vram_minimum_gb=2 BF16 weights — small footprint, runs on most GPUs ## Recommended base flags ```bash vllm serve 'PaddlePaddle/PaddleOCR-VL-1.5' \ --trust-remote-code \ --max-num-batched-tokens \ 16384 \ --no-enable-prefix-caching \ --mm-processor-cache-gb \ 0 ``` ## Guide ## Overview [PaddleOCR-VL-1.5](https://huggingface.co/PaddlePaddle/PaddleOCR-VL-1.5) is the next generation of PaddleOCR-VL — same 0.9B architecture (NaViT-style dynamic-resolution vision encoder + ERNIE-4.5-0.3B LM), with substantial accuracy gains and new tasks: - **94.5% on OmniDocBench v1.5** (SOTA) - **Text spotting** — line-level localization + recognition in one pass - **Seal recognition** — new task with SOTA results - **Irregular-shape localization** — polygonal detection under skew/warping - **Multilingual** — adds Tibetan and Bengali - Cross-page table merging and paragraph-heading recognition Architecture is identical to PaddleOCR-VL, so the vLLM launch command and feature flags are the same. ## Prerequisites - Hardware: 1x GPU (small VRAM footprint) - vLLM >= 0.11.1 (nightly if not released yet) - AMD ROCm wheels require Python 3.12, ROCm 7.2.3, and glibc >= 2.35. ### Install vLLM (NVIDIA) ```bash uv venv source .venv/bin/activate uv pip install -U vllm --pre \ --extra-index-url https://wheels.vllm.ai/nightly \ --extra-index-url https://download.pytorch.org/whl/cu129 \ --index-strategy unsafe-best-match ``` ### Install vLLM (AMD ROCm) ```bash uv venv --python 3.12 source .venv/bin/activate uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm ``` ## Launch command ```bash vllm serve PaddlePaddle/PaddleOCR-VL-1.5 \ --trust-remote-code \ --max-num-batched-tokens 16384 \ --no-enable-prefix-caching \ --mm-processor-cache-gb 0 ``` AMD MI300X/MI325X/MI355X: ```bash export SAFETENSORS_FAST_GPU=1 vllm serve PaddlePaddle/PaddleOCR-VL-1.5 \ --trust-remote-code \ --max-num-batched-tokens 16384 \ --mm-processor-cache-gb 0 ``` Tip: OCR workloads don't benefit much from prefix caching or image reuse, so disable those to avoid hashing/caching overhead. ## Client Usage Task-specific prompts (note the two new tasks `spotting` and `seal`): ```python from openai import OpenAI client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1", timeout=3600) TASKS = { "ocr": "OCR:", "table": "Table Recognition:", "formula": "Formula Recognition:", "chart": "Chart Recognition:", "spotting": "Spotting:", "seal": "Seal Recognition:", } response = client.chat.completions.create( model="PaddlePaddle/PaddleOCR-VL-1.5", messages=[{ "role": "user", "content": [ {"type": "image_url", "image_url": {"url": "https://.../receipt.png"}}, {"type": "text", "text": TASKS["spotting"]}, ], }], temperature=0.0, ) print(response.choices[0].message.content) ``` ## Offline Inference with PP-DocLayoutV2 Use separate venvs for `vllm` and `paddlepaddle` to avoid conflicts. If you see "The model PaddleOCR-VL-1.5-0.9B does not exist.", add `--served-model-name PaddleOCR-VL-1.5-0.9B`. ```bash uv pip install paddlepaddle-gpu==3.2.1 --extra-index-url https://www.paddlepaddle.org.cn/packages/stable/cu126/ uv pip install -U "paddleocr[doc-parser]" uv pip install safetensors ``` ```python from paddleocr import PaddleOCRVL pipeline = PaddleOCRVL( vl_rec_model_name="PaddleOCR-VL-1.5-0.9B", vl_rec_backend="vllm-server", vl_rec_server_url="http://localhost:8000/v1", layout_detection_model_name="PP-DocLayoutV2", layout_detection_model_dir="/path/to/your/PP-DocLayoutV2/", ) output = pipeline.predict("https://paddle-model-ecology.bj.bcebos.com/paddlex/imgs/demo_image/paddleocr_vl_demo.png") for i, res in enumerate(output): res.save_to_json(save_path=f"output_{i}.json") res.save_to_markdown(save_path=f"output_{i}.md") ``` ## References - [PaddleOCR-VL-1.5 on Hugging Face](https://huggingface.co/PaddlePaddle/PaddleOCR-VL-1.5) - [PaddleOCR GitHub](https://github.com/PaddlePaddle/PaddleOCR) - [vLLM support tracking issue](https://github.com/vllm-project/vllm/issues/33554) --- # jinaai/jina-embeddings-v5-text-small on vLLM Source: https://recipes.vllm.ai/jinaai/jina-embeddings-v5-text-small Provider: Jina AI Released: 2026-01-22T09:42:41.000Z Updated: 2026-05-09 Architecture: dense Parameters: 0.7B Context length: 32,768 Min vLLM: 0.20.0 Tasks: embedding Summary: Jina AI's fifth-gen multilingual text embedding model (677M, Qwen3-0.6B-Base) with task-specific LoRA adapters for retrieval, text-matching, classification, and clustering. ## Variants - default: precision=bf16, vram_minimum_gb=2, model_id=jinaai/jina-embeddings-v5-text-small-retrieval Retrieval task: query/document encoding for RAG and search. Adapter pre-merged into the base weights. - text_matching: precision=bf16, vram_minimum_gb=2, model_id=jinaai/jina-embeddings-v5-text-small-text-matching Text-matching task: semantic similarity, dedup, paraphrase detection. Adapter pre-merged into the base weights. - classification: precision=bf16, vram_minimum_gb=2, model_id=jinaai/jina-embeddings-v5-text-small-classification Classification task: linear probing, intent detection. Adapter pre-merged into the base weights. - clustering: precision=bf16, vram_minimum_gb=2, model_id=jinaai/jina-embeddings-v5-text-small-clustering Clustering task: k-means, topic discovery. Adapter pre-merged into the base weights. ## Recommended base flags ```bash vllm serve 'jinaai/jina-embeddings-v5-text-small' \ --trust-remote-code \ --runner \ pooling ``` ## Guide ## Overview [`jina-embeddings-v5-text-small`](https://huggingface.co/jinaai/jina-embeddings-v5-text-small) is the fifth generation of Jina AI's multilingual text embedding family, released February 18, 2026. It scores **71.7 on MTEB English v2** and **67.7 on MMTEB** with only 677M parameters — the highest among multilingual embedding models under 1B — and supports **119+ languages with up to 32K-token context**. Built on `Qwen3-0.6B-Base` and trained by distilling `Qwen3-Embedding-4B` plus task-specific contrastive losses, it produces 1024-dim embeddings that stay robust under truncation (Matryoshka dims: 32–1024) and binary quantization. vLLM support landed in [PR #39575](https://github.com/vllm-project/vllm/pull/39575) via the `JinaEmbeddingsV5Model` architecture and the `--runner pooling` API. ### Task-specific adapters v5 ships four LoRA adapters — one per supported task. For each task, Jina AI publishes a sibling repo with that adapter **pre-merged into the base weights**; these are the simplest path for vLLM and what this recipe serves. Pick a task above; the recipe swaps the model id accordingly: | Task | Pre-merged repo | |------|----------------| | Retrieval | `jinaai/jina-embeddings-v5-text-small-retrieval` | | Text-matching | `jinaai/jina-embeddings-v5-text-small-text-matching` | | Classification | `jinaai/jina-embeddings-v5-text-small-classification` | | Clustering | `jinaai/jina-embeddings-v5-text-small-clustering` | ## Prerequisites - **Hardware**: any single GPU with ≥ 2 GB VRAM (T4 / L4 / A10 / A100 / H100 / H200 / B200 / MI300X all fine — bf16 weights are ~1.4 GB). - **vLLM**: requires a build that includes PR #39575. Use the nightly wheel until the next stable release ships: ```bash uv pip install -U vllm --pre \ --extra-index-url https://wheels.vllm.ai/nightly \ --index-strategy unsafe-best-match ``` ## Launch command Use the launch command above (it points at the chosen task's pre-merged repo). The full form looks like: ```bash vllm serve jinaai/jina-embeddings-v5-text-small-retrieval \ --trust-remote-code \ --runner pooling \ --host 0.0.0.0 --port 8000 ``` Swap the model id for `-text-matching`, `-classification`, or `-clustering` to serve the corresponding task — or just toggle the **Variant** pill above. ### Alternative: base repo + `--hf-overrides` If you'd rather download a single checkpoint and switch tasks at startup, serve the base `jinaai/jina-embeddings-v5-text-small` repo and pass the task via `--hf-overrides`: ```bash vllm serve jinaai/jina-embeddings-v5-text-small \ --trust-remote-code \ --runner pooling \ --hf-overrides '{"jina_task": "retrieval"}' ``` Allowed values: `retrieval`, `text-matching`, `classification`, `clustering`. This loads the base weights and merges the requested adapter at startup. ## Client usage ### Embeddings (`/v1/embeddings`) ```bash curl -s http://localhost:8000/v1/embeddings \ -H "Content-Type: application/json" \ -d '{ "model": "jinaai/jina-embeddings-v5-text-small-retrieval", "input": ["Query: What is climate change?"] }' | python3 -m json.tool ``` ### Python (OpenAI SDK) ```python from openai import OpenAI client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY") resp = client.embeddings.create( model="jinaai/jina-embeddings-v5-text-small-retrieval", input=[ "Climate change has led to rising sea levels.", "Overview of climate change impacts on coastal cities.", ], ) for d in resp.data: print(d.index, len(d.embedding)) ``` ### Multilingual text-matching example Switch the **Variant** pill to **text-matching** (or serve the `-text-matching` repo) and embed semantically equivalent strings across languages: ```python texts = [ "غروب جميل على الشاطئ", "海滩上美丽的日落", "A beautiful sunset over the beach", "Un beau coucher de soleil sur la plage", "浜辺に沈む美しい夕日", ] resp = client.embeddings.create( model="jinaai/jina-embeddings-v5-text-small-text-matching", input=texts, ) ``` ## Configuration tips - **Pick the task that matches your workload** — retrieval prompts (`query` vs. `document`) are baked into the retrieval adapter, so the wrong task measurably degrades recall. - **Matryoshka truncation**: embeddings stay useful when truncated to 32 / 64 / 128 / 256 / 512 / 768 dims — keep the prefix and renormalize. - **Throughput**: with TP=1 on a single small GPU, the bottleneck is usually tokenization — batch your inputs (`input: [...]` with up to a few hundred short docs per request). - **bf16 vs fp16**: README recommends bf16 on modern GPUs; PR #39575's test used fp16. Either dtype works; bf16 is more numerically stable on Hopper / Blackwell / MI300X+. ## References - [Model card](https://huggingface.co/jinaai/jina-embeddings-v5-text-small) - [Release blog](https://jina.ai/news/jina-embeddings-v5-text-distilling-4b-quality-into-sub-1b-multilingual-embeddings) - [Technical report (arXiv:2602.15547)](https://arxiv.org/abs/2602.15547) - [vLLM PR #39575 — Add Jina Embeddings v5 model support](https://github.com/vllm-project/vllm/pull/39575) - [Pre-merged retrieval repo](https://huggingface.co/jinaai/jina-embeddings-v5-text-small-retrieval) - [Pre-merged text-matching repo](https://huggingface.co/jinaai/jina-embeddings-v5-text-small-text-matching) - [Pre-merged classification repo](https://huggingface.co/jinaai/jina-embeddings-v5-text-small-classification) - [Pre-merged clustering repo](https://huggingface.co/jinaai/jina-embeddings-v5-text-small-clustering) --- # mistralai/Ministral-3-8B-Reasoning-2512 on vLLM Source: https://recipes.vllm.ai/mistralai/Ministral-3-8B-Reasoning-2512 Provider: Mistral AI Released: 2025-10-31T08:41:36.000Z Updated: 2026-05-06 Architecture: dense Parameters: 8B Context length: 262,144 Min vLLM: 0.11.0 Tasks: multimodal Verified hardware: H200, GB200 NVL4, MI300X Summary: Ministral 3 Reasoning family (3B/8B/14B) with BF16 weights, vision support, and 256K context ## Variants - default: precision=bf16, vram_minimum_gb=22 Native BF16 weights (8B) - 3b: precision=bf16, vram_minimum_gb=8, model_id=mistralai/Ministral-3-3B-Reasoning-2512 Smallest 3B variant with tied embeddings - 14b: precision=bf16, vram_minimum_gb=32, model_id=mistralai/Ministral-3-14B-Reasoning-2512 Largest 14B variant; 2xH200 recommended for full context ## Recommended base flags ```bash vllm serve 'mistralai/Ministral-3-8B-Reasoning-2512' \ --tokenizer_mode \ mistral \ --config_format \ mistral \ --load_format \ mistral ``` ## Guide ## Overview Ministral-3 Reasoning comes with BF16 weights in 3 sizes: - 3B (tied embeddings) - 8B, 14B (independent embeddings/outputs) Each variant has vision support and 256K max context. On GB200, we observe significant speed-ups with NVFP4 Marlin fallback for older GPUs. ## Prerequisites - Hardware: 1x H200 (3B/8B), 2x H200 recommended for 14B with full context - vLLM >= 0.11.0 ### Install vLLM ```bash uv venv source .venv/bin/activate uv pip install -U vllm --torch-backend auto ``` ## Launch command 3B or 8B on 1x H200: ```bash vllm serve mistralai/Ministral-3-8B-Reasoning-2512 \ --tokenizer_mode mistral --config_format mistral --load_format mistral \ --enable-auto-tool-choice --tool-call-parser mistral \ --reasoning-parser mistral ``` 14B on 2x H200: ```bash vllm serve mistralai/Ministral-3-14B-Reasoning-2512 \ --tensor-parallel-size 2 \ --tokenizer_mode mistral --config_format mistral --load_format mistral \ --enable-auto-tool-choice --tool-call-parser mistral \ --reasoning-parser mistral ``` Key flags: - `enable-auto-tool-choice`: required for tool usage - `tool-call-parser mistral`: required for tool usage - `reasoning-parser mistral`: required to extract reasoning content ## Client Usage Streaming reasoning + answer: ```python from openai import OpenAI client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1") stream = client.chat.completions.create( model="mistralai/Ministral-3-8B-Reasoning-2512", messages=[{"role": "user", "content": "Solve: use 2,5,6,3 to make 24."}], stream=True, temperature=0.7, top_p=0.95, max_tokens=262144, ) for chunk in stream: delta = chunk.choices[0].delta # vLLM 0.11.0 uses reasoning_content; 0.11.1+ uses reasoning. rc = getattr(delta, "reasoning", getattr(delta, "reasoning_content", None)) if rc: print(rc, end="", flush=True) if delta.content: print(delta.content, end="", flush=True) ``` ## Troubleshooting - OOM on 14B: use `--tensor-parallel-size 2` or lower `--max-model-len`. ## References - [Ministral-3-8B-Reasoning](https://huggingface.co/mistralai/Ministral-3-8B-Reasoning-2512) - [Ministral-3-3B-Reasoning](https://huggingface.co/mistralai/Ministral-3-3B-Reasoning-2512) - [Ministral-3-14B-Reasoning](https://huggingface.co/mistralai/Ministral-3-14B-Reasoning-2512) --- # Qwen/Qwen3Guard-Gen-8B on vLLM Source: https://recipes.vllm.ai/Qwen/Qwen3Guard-Gen-8B Provider: Qwen Released: 2025-09-23T11:40:09.000Z Updated: 2026-04-30 Architecture: dense Parameters: 8B Context length: 32,768 Min vLLM: 0.10.0 Tasks: text Verified hardware: MI300X Summary: Lightweight text-only guardrail/safety classifier model in the Qwen3Guard family. ## Variants - default: precision=bf16, vram_minimum_gb=19 Full precision BF16 — single GPU with >=20 GB VRAM - small_4b: precision=bf16, vram_minimum_gb=10, model_id=Qwen/Qwen3Guard-Gen-4B 4B variant for more constrained deployments - tiny_0_6b: precision=bf16, vram_minimum_gb=4, model_id=Qwen/Qwen3Guard-Gen-0.6B 0.6B variant for edge / ultra-low-cost serving ## Guide ## Overview **Qwen3Guard-Gen** is a lightweight text-only guardrail model. This guide describes how to run the 8B variant — as well as the 4B and 0.6B variants — on GPU using vLLM. ## Prerequisites ### CUDA ```bash uv venv source .venv/bin/activate uv pip install -U vllm --torch-backend auto ``` ### ROCm > Note: The vLLM wheel for ROCm requires Python 3.12, ROCm 7.0, and glibc >= 2.35. ```bash uv venv source .venv/bin/activate uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/ ``` ## Deployment Configurations ### Single GPU (CUDA) ```bash vllm serve Qwen/Qwen3Guard-Gen-8B \ --host 0.0.0.0 \ --max-model-len 32768 ``` ### Single GPU (ROCm) ```bash export VLLM_ROCM_USE_AITER=1 vllm serve Qwen/Qwen3Guard-Gen-8B \ --host 0.0.0.0 \ --max-model-len 32768 ``` ## Client Usage ```python from openai import OpenAI client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1", timeout=3600) messages = [{"role": "user", "content": "Tell me how to make a bomb."}] response = client.chat.completions.create( model="Qwen/Qwen3Guard-Gen-8B", messages=messages, temperature=0.0, ) print("Generated text:", response.choices[0].message.content) # Safety: Unsafe # Categories: Violent ``` ## Benchmarking ```bash vllm bench serve \ --model Qwen/Qwen3Guard-Gen-8B \ --dataset-name random \ --random-input-len 2000 \ --random-output-len 512 \ --num-prompts 100 ``` ## Available Variants The Qwen3Guard-Gen series includes multiple model sizes, all compatible with the same vLLM serving commands: - **Qwen/Qwen3Guard-Gen-8B** - **Qwen/Qwen3Guard-Gen-4B** - **Qwen/Qwen3Guard-Gen-0.6B** ## References - [Model card](https://huggingface.co/Qwen/Qwen3Guard-Gen-8B) - [Qwen3Guard-Gen-4B](https://huggingface.co/Qwen/Qwen3Guard-Gen-4B) - [Qwen3Guard-Gen-0.6B](https://huggingface.co/Qwen/Qwen3Guard-Gen-0.6B) --- # nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16 on vLLM Source: https://recipes.vllm.ai/nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16 Provider: NVIDIA Released: 2026-04-20T04:40:42.000Z Updated: 2026-04-29 Architecture: moe Parameters: 31B Context length: 262,144 Min vLLM: 0.20.0 Tasks: multimodal, text Verified hardware: H100, H200, B200 Summary: Mamba2-Transformer hybrid MoE omnimodal model (31B total / 3B active) with unified video, audio, image, and text understanding; reasoning + tool calling; BF16, FP8, and NVFP4 variants ## Variants - default: precision=bf16, vram_minimum_gb=75 BF16 weights — full-precision reference - fp8: precision=fp8, vram_minimum_gb=38, model_id=nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-FP8 ModelOpt FP8 weights + FP8 KV cache - nvfp4: precision=nvfp4, vram_minimum_gb=28, model_id=nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-NVFP4 ModelOpt NVFP4 weights — Blackwell-only; smallest footprint ## Recommended base flags ```bash vllm serve 'nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16' \ --trust-remote-code \ --max-model-len \ 131072 \ --media-io-kwargs \ {"video": {"num_frames": 512, "fps": 1}} \ --video-pruning-rate \ 0.5 ``` ## Guide ## Overview NVIDIA Nemotron-3-Nano-Omni-30B-A3B-Reasoning is a Mamba2-Transformer hybrid MoE omnimodal model (31B total / 3B active) that unifies video, audio, image, and text understanding. It is built on the Nemotron-3-Nano-30B-A3B LLM backbone with a CRADIO v4-H vision encoder and a Parakeet speech encoder, and ships in BF16, ModelOpt FP8, and ModelOpt NVFP4 variants. Capabilities: - Video (mp4, up to 2 minutes, sampled at 1–2 FPS / 128–256 frames) - Audio (wav, mp3, up to 1 hour, ≥8 kHz) - Image (jpeg, png) - Text (English, up to 256K context) - Reasoning with chain-of-thought (`` tags) - Tool calling - Word-level timestamps for transcription ## Prerequisites - vLLM **0.28.0** (pinned: `pip install vllm[audio]==0.28.0`, or pull `vllm/vllm-openai:v0.28.0`) - Hardware: 1× B200 / H200 / H100 (single-GPU TP1 is the documented profile) - The `audio` extra is required for any audio input, including `use_audio_in_video=true` ### Pull the Docker image ```bash # CUDA 13: docker pull vllm/vllm-openai:v0.28.0 # CUDA 12.9: docker pull vllm/vllm-openai:v0.28.0-cu129 ``` ## Launch command General single-GPU invocation (B200 / H200 / H100): ```bash vllm serve nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16 \ --served-model-name nemotron \ --host 0.0.0.0 \ --port 5000 \ --tensor-parallel-size 1 \ --max-model-len 131072 \ --trust-remote-code \ --video-pruning-rate 0.5 \ --media-io-kwargs '{"video": {"num_frames": 512, "fps": 1}}' \ --reasoning-parser nemotron_v3 \ --enable-auto-tool-choice \ --tool-call-parser qwen3_coder ``` Swap the model id for the FP8 or NVFP4 checkpoint and add `--kv-cache-dtype fp8`: ```bash vllm serve nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-NVFP4 \ --kv-cache-dtype fp8 \ ... ``` ### Platform-specific notes - **RTX Pro:** append `--moe-backend triton` (FlashInfer + RTX Pro bug). - **NVFP4 + TP>1:** append `--moe-backend flashinfer_cutlass` (TRTLLM_GEN MoE kernel bug at TP>1 on NVFP4). - **DGX Spark (aarch64):** unified LPDDR5X memory; lower `--gpu-memory-utilization` to 0.70 and reduce `--max-model-len` (e.g. 32768) if you hit OOM. Use `--max-num-seqs 8`. ## Recommended sampling | Mode | temperature | top_p | top_k | max_tokens | reasoning_budget | |------|------------:|------:|------:|-----------:|------------------:| | Thinking | 0.6 | 0.95 | — | 20480 | 16384 | | Instruct | 0.2 | — | 1 | 1024 | — | Toggle thinking mode via `chat_template_kwargs={"enable_thinking": true}` (default) or `false` to disable. ## Client usage ```python from openai import OpenAI client = OpenAI(base_url="http://localhost:8000/v1", api_key="") resp = client.chat.completions.create( model="nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-NVFP4", messages=[{"role": "user", "content": "Hello!"}], extra_body={"chat_template_kwargs": {"enable_thinking": True}}, ) ``` ## Benchmarking ```bash vllm bench serve \ --model nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-FP8 \ --trust-remote-code \ --dataset-name random \ --random-input-len 1024 --random-output-len 1024 \ --num-warmups 20 --ignore-eos \ --max-concurrency 1024 --num-prompts 2048 ``` ## References - [BF16 model card](https://huggingface.co/nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16) - [FP8 model card](https://huggingface.co/nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-FP8) - [NVFP4 model card](https://huggingface.co/nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-NVFP4) - [Nemotron 3 Nano LLM backbone](https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16) --- # poolside/Laguna-XS.2 on vLLM Source: https://recipes.vllm.ai/poolside/Laguna-XS.2 Provider: Poolside Released: 2026-04-23T20:50:45.000Z Updated: 2026-04-29 Architecture: moe Parameters: 33B Context length: 262,144 Min vLLM: 0.21.0 Tasks: text Verified hardware: H200, MI300X, MI325X, MI355X Summary: Poolside's 33B total / 3B activated MoE coding model with mixed sliding-window + global attention, native interleaved reasoning, and 256K context — designed for agentic coding. ## Variants - default: precision=bf16, vram_minimum_gb=80 BF16 weights — fits on a single 80GB+ GPU (H100/H200/B200) - fp8: precision=fp8, vram_minimum_gb=40, model_id=poolside/Laguna-XS.2-FP8 Block-FP8 weights + FP8 KV cache — halves VRAM, fits on a single 40GB+ GPU - nvfp4: precision=nvfp4, vram_minimum_gb=26, model_id=poolside/Laguna-XS.2-NVFP4 NVFP4 weights + FP8 KV cache (Blackwell only) — the most compact variant at ~26GB - int4: precision=int4, vram_minimum_gb=29, model_id=poolside/Laguna-XS.2-INT4 INT4 (W4A16) weights + FP8 KV cache — fits on a single 32GB+ GPU - redhat_fp8: precision=fp8, vram_minimum_gb=40, model_id=RedHatAI/Laguna-XS.2-FP8 RedHatAI FP8 quantization - redhat_nvfp4: precision=nvfp4, vram_minimum_gb=22, model_id=RedHatAI/Laguna-XS.2-NVFP4 RedHatAI NVFP4 quantization ## Recommended base flags ```bash vllm serve 'poolside/Laguna-XS.2' \ --trust-remote-code ``` ## Guide ## Overview [Laguna XS.2](https://huggingface.co/poolside/Laguna-XS.2) is Poolside's 33B-total / 3B-activated Mixture-of-Experts model purpose-built for agentic coding and long-horizon work. It combines mixed sliding-window + global attention (3:1 across 40 layers) with sigmoid per-head gating and FP8 KV cache, so it stays compact enough to run locally while supporting a 256K-token context. ### Key features - **Mixed SWA + global attention**: 30 sliding-window layers (window=512) interleaved with 10 global-attention layers, each with per-layer rotary scaling. - **Native FP8 KV cache**: KV cache is quantized to FP8 to reduce memory per token. - **Interleaved reasoning**: thinking blocks emitted between tool calls; toggled per-request via `enable_thinking`. - **Tool calling**: Poolside-specific XML-style tool-call protocol, parsed via `poolside_v1`. - **256 experts + 1 shared expert** with top-8 routing. ## Prerequisites Laguna XS.2 support is on the open vLLM PR ([vllm-project/vllm#41129](https://github.com/vllm-project/vllm/pull/41129)) — install from a nightly wheel or the pinned Docker image below until the PR lands in a stable release. ### Docker (recommended) ```bash docker pull vllm/vllm-openai:laguna ``` ### pip (nightly) ```bash uv venv source .venv/bin/activate uv pip install -U vllm --pre \ --extra-index-url https://wheels.vllm.ai/nightly/cu130 \ --extra-index-url https://download.pytorch.org/whl/cu130 \ --index-strategy unsafe-best-match ``` ## Quantized variants Pre-quantized checkpoints are published alongside the BF16 weights — select one with the **Variant** control above: | Variant | Checkpoint | Weights | ~VRAM | Notes | |---------|-----------|---------|-------|-------| | BF16 | [Laguna-XS.2](https://huggingface.co/poolside/Laguna-XS.2) | BF16 | 80 GB | Default | | FP8 | [Laguna-XS.2-FP8](https://huggingface.co/poolside/Laguna-XS.2-FP8) | Block-FP8 | 40 GB | Set `VLLM_BLOCKSCALE_FP8_GEMM_FLASHINFER=0` | | NVFP4 | [Laguna-XS.2-NVFP4](https://huggingface.co/poolside/Laguna-XS.2-NVFP4) | NVFP4 | 26 GB | Blackwell only | | INT4 | [Laguna-XS.2-INT4](https://huggingface.co/poolside/Laguna-XS.2-INT4) | INT4 (W4A16) | 29 GB | Hopper or Blackwell | Quantization is detected automatically from each checkpoint's `quantization_config`, so the launch command is identical — just swap the model ID (select the variant above). All three quantized checkpoints ship an **FP8-quantized KV cache**. > **vLLM ≥ 0.22.0 required for the quantized variants.** The FP8 KV cache needs the per-layer attention-head fix in [vllm#42650](https://github.com/vllm-project/vllm/pull/42650); earlier versions produce scrambled output on non-Hopper GPUs. On older vLLM you can disable the FP8 KV cache with `--kv-cache-dtype-skip-layers $(seq 0 39)`. ## Deployment Configurations ### Single GPU (H100/H200/B200, BF16) ```bash vllm serve poolside/Laguna-XS.2 \ --trust-remote-code \ --max-model-len 262144 \ --enable-auto-tool-choice \ --tool-call-parser poolside_v1 \ --reasoning-parser poolside_v1 ``` ### Docker ```bash docker run -itd --name laguna-xs2 \ --ipc=host --network host --shm-size 16G --gpus all \ -v ~/.cache/huggingface:/root/.cache/huggingface \ vllm/vllm-openai:laguna \ --model poolside/Laguna-XS.2 \ --trust-remote-code \ --max-model-len 262144 \ --enable-auto-tool-choice \ --tool-call-parser poolside_v1 \ --reasoning-parser poolside_v1 \ --host 0.0.0.0 --port 8000 ``` ### AMD MI300X / MI325X / MI355X ```bash vllm serve poolside/Laguna-XS.2 \ --trust-remote-code \ --max-model-len 262144 \ --enable-auto-tool-choice \ --tool-call-parser poolside_v1 \ --reasoning-parser poolside_v1 ``` ### Docker (AMD MI300X / MI325X / MI355X) ```bash docker run --rm -it \ --cap-add=SYS_PTRACE \ --ipc=host \ --privileged=true \ --shm-size=128GB \ --network=host \ --device=/dev/kfd \ --device=/dev/dri \ --group-add video \ vllm/vllm-openai-rocm:latest \ --model poolside/Laguna-XS.2 \ --trust-remote-code \ --max-model-len 262144 \ --enable-auto-tool-choice \ --tool-call-parser poolside_v1 \ --reasoning-parser poolside_v1 ``` ## Controlling reasoning Reasoning is **off by default** in the chat template. Enable it per-request: ```python from openai import OpenAI client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY") resp = client.chat.completions.create( model="poolside/Laguna-XS.2", messages=[{"role": "user", "content": "Write a Python retry wrapper with exponential backoff."}], extra_body={"chat_template_kwargs": {"enable_thinking": True}}, temperature=0.7, top_p=1.0, extra_query={"top_k": 20}, ) print(resp.choices[0].message.reasoning) print(resp.choices[0].message.content) ``` Or default-on with `--default-chat-template-kwargs '{"enable_thinking": true}'`. ## Speculative decoding (DFlash) Enable the **Spec Decoding** toggle above to attach Poolside's [DFlash draft model](https://huggingface.co/poolside/Laguna-XS.2-speculator.dflash) — a 5-layer Llama-style speculator that proposes up to 7 tokens per step. Reported per-position acceptance with reasoning enabled is ~70% at position 1 across coding, math, QA, and writing workloads. Requires: - vLLM built from [PR #41880](https://github.com/vllm-project/vllm/pull/41880) (extends the base Laguna PR with DFlash support). - `VLLM_USE_DEEP_GEMM=0` in the launch environment — DeepGEMM is currently incompatible with the DFlash draft path. Example: ```bash VLLM_USE_DEEP_GEMM=0 vllm serve poolside/Laguna-XS.2 \ --trust-remote-code \ --max-model-len 16384 \ --enable-auto-tool-choice \ --tool-call-parser poolside_v1 \ --reasoning-parser poolside_v1 \ --speculative-config '{"model":"poolside/Laguna-XS.2-speculator.dflash","num_speculative_tokens":7,"method":"dflash"}' ``` ## References - [Model card](https://huggingface.co/poolside/Laguna-XS.2) - [Release blog post](https://poolside.ai/blog/laguna-a-deeper-dive) - [vLLM support PR #41129](https://github.com/vllm-project/vllm/pull/41129) - [vLLM DFlash spec-decoding PR #41880](https://github.com/vllm-project/vllm/pull/41880) - [DFlash draft model](https://huggingface.co/poolside/Laguna-XS.2-speculator.dflash) --- # nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16 on vLLM Source: https://recipes.vllm.ai/nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16 Provider: NVIDIA Released: 2026-03-07T01:23:35.000Z Updated: 2026-04-28 Architecture: dense Parameters: 4B Context length: 262,144 Min vLLM: 0.11.2 Tasks: text Verified hardware: H100, H200, B200, Arc Pro B60, Arc Pro B70 Summary: NVIDIA Nemotron-3-Nano 4B (Mamba-hybrid dense) — compact reasoning + tool-use model with BF16 and FP8 variants ## Variants - default: precision=bf16, vram_minimum_gb=10 BF16 weights - fp8: precision=fp8, vram_minimum_gb=5, model_id=nvidia/NVIDIA-Nemotron-3-Nano-4B-FP8 FP8 weights + FP8 KV cache ## Recommended base flags ```bash vllm serve 'nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16' \ --trust-remote-code \ --async-scheduling ``` ## Guide ## Overview Nemotron-3-Nano-4B is the smallest member of the Nemotron-3 hybrid-Mamba family. It's tuned for low-latency reasoning, tool calling, and edge deployments — DGX Spark and Jetson Thor are both supported alongside standard Hopper/Blackwell servers. ## Prerequisites - Hardware: 1x H100/H200/B200, DGX Spark, Jetson Thor, or Intel Arc Pro B60/B70 - vLLM >= 0.11.2 (0.28.0 recommended) - Docker with NVIDIA Container Toolkit (recommended) ### Pull Docker Image ```bash docker pull --platform linux/amd64 vllm/vllm-openai:v0.28.0 ``` Jetson Thor: ```bash docker pull ghcr.io/nvidia-ai-iot/vllm:latest-jetson-thor ``` Intel XPU (B60 / B70): ```bash docker pull vllm/vllm-openai-xpu:latest ``` ## Launch commands BF16: ```bash wget https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16/resolve/main/nano_v3_reasoning_parser.py vllm serve nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16 \ --trust-remote-code \ --async-scheduling \ --max-model-len 262144 \ --tensor-parallel-size 1 \ --enable-auto-tool-choice \ --tool-call-parser qwen3_coder \ --reasoning-parser-plugin nano_v3_reasoning_parser.py \ --reasoning-parser nano_v3 ``` FP8: ```bash vllm serve nvidia/NVIDIA-Nemotron-3-Nano-4B-FP8 \ --trust-remote-code \ --async-scheduling \ --kv-cache-dtype fp8 \ --tensor-parallel-size 1 ``` ### Intel XPU (B60, B70) Validated on 1× Intel Arc Pro B60 / B70 (B60 24 GB, B70 32 GB per card) with the official vLLM XPU image `vllm/vllm-openai-xpu:latest`. BF16: ```bash docker run --device /dev/dri \ -v /dev/dri/by-path:/dev/dri/by-path --shm-size=16g \ --privileged --ipc=host -p 8000:8000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ vllm/vllm-openai-xpu:latest \ nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16 \ --trust-remote-code \ --async-scheduling \ --kv-cache-dtype auto \ --tensor-parallel-size 1 \ --enforce-eager \ --enable-auto-tool-choice \ --tool-call-parser qwen3_coder \ --reasoning-parser-plugin nano_v3_reasoning_parser.py \ --reasoning-parser nano_v3 \ --max-model-len 8192 ``` FP8: ```bash docker run --device /dev/dri \ -v /dev/dri/by-path:/dev/dri/by-path --shm-size=16g \ --privileged --ipc=host -p 8000:8000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ vllm/vllm-openai-xpu:latest \ nvidia/NVIDIA-Nemotron-3-Nano-4B-FP8 \ --trust-remote-code \ --async-scheduling \ --kv-cache-dtype fp8 \ --tensor-parallel-size 1 \ --enforce-eager \ --max-model-len 8192 ``` ## Benchmarking ```bash vllm bench serve \ --model nvidia/NVIDIA-Nemotron-3-Nano-4B-FP8 \ --trust-remote-code \ --dataset-name random \ --random-input-len 1024 --random-output-len 1024 \ --num-warmups 20 \ --ignore-eos \ --max-concurrency 256 \ --num-prompts 1024 ``` ## References - [Nemotron-3-Nano-4B-BF16](https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16) - [Nemotron-3-Nano-4B-FP8](https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Nano-4B-FP8) --- # nvidia/NVIDIA-Nemotron-Nano-9B-v2 on vLLM Source: https://recipes.vllm.ai/nvidia/NVIDIA-Nemotron-Nano-9B-v2 Provider: NVIDIA Released: 2025-08-12T22:43:32.000Z Updated: 2026-04-28 Architecture: dense Parameters: 9B Context length: 131,072 Min vLLM: 0.10.1 Tasks: text Verified hardware: H100, H200, B200 Summary: NVIDIA Nemotron-Nano 9B (Mamba-hybrid dense) reasoning + tool-use model with FP8 / NVFP4 / Japanese variants ## Variants - default: precision=bf16, vram_minimum_gb=22 BF16 weights on 1 GPU - fp8: precision=fp8, vram_minimum_gb=11, model_id=nvidia/NVIDIA-Nemotron-Nano-9B-v2-FP8 FP8 weights - nvfp4: precision=nvfp4, vram_minimum_gb=6, model_id=nvidia/NVIDIA-Nemotron-Nano-9B-v2-NVFP4 NVFP4 weights for Blackwell - base: precision=bf16, vram_minimum_gb=22, model_id=nvidia/NVIDIA-Nemotron-Nano-9B-v2-Base Pre-RL base checkpoint (BF16) - japanese: precision=bf16, vram_minimum_gb=22, model_id=nvidia/NVIDIA-Nemotron-Nano-9B-v2-Japanese Japanese-specialized fine-tune (BF16) - redhat_fp8: precision=fp8, vram_minimum_gb=11, model_id=RedHatAI/NVIDIA-Nemotron-Nano-9B-v2-FP8-dynamic RedHatAI FP8 (E4M3) with dynamic per-token activation quantization ## Recommended base flags ```bash vllm serve 'nvidia/NVIDIA-Nemotron-Nano-9B-v2' \ --trust-remote-code ``` ## Guide ## Overview Nemotron-Nano-9B-v2 is a 9B Mamba-hybrid dense reasoning model that runs on a single H100/H200/B200. Variants ship in BF16, FP8, and NVFP4; a Japanese-specialized fine-tune and a pre-RL Base checkpoint are also available. ## Prerequisites - Hardware: 1x H100/H200/B200 (or comparable) - vLLM >= 0.10.1 (`pip install -U vllm`) - Docker with NVIDIA Container Toolkit (recommended) ## Launch command ```bash wget https://huggingface.co/nvidia/NVIDIA-Nemotron-Nano-9B-v2/resolve/main/nemotron_toolcall_parser_no_streaming.py vllm serve nvidia/NVIDIA-Nemotron-Nano-9B-v2 \ --trust-remote-code \ --max-model-len 131072 \ --tensor-parallel-size 1 \ --enable-auto-tool-choice \ --tool-parser-plugin nemotron_toolcall_parser_no_streaming.py \ --tool-call-parser nemotron_json ``` FP8: ```bash vllm serve nvidia/NVIDIA-Nemotron-Nano-9B-v2-FP8 \ --trust-remote-code \ --tensor-parallel-size 1 ``` NVFP4 (Blackwell only): ```bash vllm serve nvidia/NVIDIA-Nemotron-Nano-9B-v2-NVFP4 \ --trust-remote-code \ --tensor-parallel-size 1 ``` ## Benchmarking ```bash vllm bench serve \ --model nvidia/NVIDIA-Nemotron-Nano-9B-v2 \ --trust-remote-code \ --dataset-name random \ --random-input-len 1024 --random-output-len 1024 \ --num-warmups 20 \ --ignore-eos \ --max-concurrency 256 \ --num-prompts 1024 ``` ## References - [Nemotron-Nano-9B-v2](https://huggingface.co/nvidia/NVIDIA-Nemotron-Nano-9B-v2) - [Nemotron-Nano-9B-v2-FP8](https://huggingface.co/nvidia/NVIDIA-Nemotron-Nano-9B-v2-FP8) - [Nemotron-Nano-9B-v2-NVFP4](https://huggingface.co/nvidia/NVIDIA-Nemotron-Nano-9B-v2-NVFP4) - [Nemotron-Nano-9B-v2-Base](https://huggingface.co/nvidia/NVIDIA-Nemotron-Nano-9B-v2-Base) - [Nemotron-Nano-9B-v2-Japanese](https://huggingface.co/nvidia/NVIDIA-Nemotron-Nano-9B-v2-Japanese) --- # Wan-AI/Wan2.2-T2V-A14B-Diffusers on vLLM Source: https://recipes.vllm.ai/Wan-AI/Wan2.2-T2V-A14B-Diffusers Provider: Wan (Alibaba) Released: 2025-07-28T09:04:28.000Z Updated: 2026-04-27 Architecture: moe Parameters: 28B Min vLLM: 0.12.0 Tasks: omni Summary: Wan2.2 video generation models — T2V/I2V MoE (14B active) and unified TI2V (5B dense), served via vLLM-Omni ## Variants - default: precision=bf16, vram_minimum_gb=152 BF16 — variants moved to omni.tasks (T2V / I2V / TI2V each pick a different checkpoint) ## Guide ## Overview Wan2.2 is a video generation family served via **vLLM-Omni** with optional **Cache-DiT** acceleration: - `Wan-AI/Wan2.2-T2V-A14B-Diffusers` — Text-to-Video (MoE, 14B active) - `Wan-AI/Wan2.2-I2V-A14B-Diffusers` — Image-to-Video (MoE, 14B active) - `Wan-AI/Wan2.2-TI2V-5B-Diffusers` — Unified Text+Image-to-Video (dense 5B) ## Prerequisites - vLLM-Omni on top of vLLM 0.12.0 - diffusers (bundled in vLLM-Omni CLI scripts) ## Installation ```bash uv venv source .venv/bin/activate uv pip install vllm==0.12.0 uv pip install git+https://github.com/vllm-project/vllm-omni.git@ef01223c42be10ee260b9f6e5ec31894cd09d86e ``` ### AMD ROCm (MI300X/MI325X/MI355X) ```bash uv venv source .venv/bin/activate uv pip install vllm==0.12.0 --extra-index-url https://wheels.vllm.ai/rocm/ uv pip install git+https://github.com/vllm-project/vllm-omni.git@ef01223c42be10ee260b9f6e5ec31894cd09d86e ``` > ⚠️ The vLLM ROCm wheel requires Python 3.12, ROCm 7.0, and glibc >= 2.35. ## Text-to-Video (T2V) ```python from vllm_omni.entrypoints.omni import Omni omni = Omni(model="Wan-AI/Wan2.2-T2V-A14B-Diffusers") frames = omni.generate( "Two anthropomorphic cats in comfy boxing gear fight on a spotlighted stage.", height=720, width=1280, num_frames=81, num_inference_steps=40, guidance_scale=4.0, ) ``` CLI: ```bash python examples/offline_inference/text_to_video/text_to_video.py \ --model Wan-AI/Wan2.2-T2V-A14B-Diffusers \ --prompt "A serene lakeside sunrise with mist over the water." \ --height 720 --width 1280 \ --num_frames 81 --num_inference_steps 40 \ --guidance_scale 4.0 --fps 24 \ --output t2v_output.mp4 ``` ### Running T2V on MI300X/MI325X/MI355X GPUs ```bash VLLM_ROCM_USE_AITER=1 \ SAFETENSORS_FAST_GPU=1 \ python examples/offline_inference/text_to_video/text_to_video.py \ --model Wan-AI/Wan2.2-T2V-A14B-Diffusers \ --prompt "A serene lakeside sunrise with mist over the water." \ --height 720 --width 1280 \ --num_frames 81 --num_inference_steps 40 \ --guidance_scale 4.0 --fps 24 \ --output t2v_output.mp4 ``` ## Image-to-Video (I2V) ```python import PIL.Image from vllm_omni.entrypoints.omni import Omni omni = Omni(model="Wan-AI/Wan2.2-I2V-A14B-Diffusers") image = PIL.Image.open("input.jpg").convert("RGB") frames = omni.generate( "A cat playing with yarn", pil_image=image, height=480, width=832, num_frames=81, num_inference_steps=50, guidance_scale=5.0, ) ``` ### Running I2V on MI300X/MI325X/MI355X GPUs ```bash VLLM_ROCM_USE_AITER=1 \ SAFETENSORS_FAST_GPU=1 \ python examples/offline_inference/image_to_video/image_to_video.py \ --model Wan-AI/Wan2.2-I2V-A14B-Diffusers \ --image input.jpg --prompt "A cat playing with yarn" \ --num_frames 81 --num_inference_steps 50 \ --guidance_scale 5.0 --fps 16 --output i2v_output.mp4 ``` TI2V CLI: ```bash python examples/offline_inference/image_to_video/image_to_video.py \ --model Wan-AI/Wan2.2-TI2V-5B-Diffusers \ --image input.jpg --prompt "A cat playing with yarn" \ --num_frames 81 --num_inference_steps 50 \ --guidance_scale 5.0 --fps 16 --output ti2v_output.mp4 ``` ### Running TI2V on MI300X/MI325X/MI355X GPUs ```bash VLLM_ROCM_USE_AITER=1 \ SAFETENSORS_FAST_GPU=1 \ python examples/offline_inference/image_to_video/image_to_video.py \ --model Wan-AI/Wan2.2-TI2V-5B-Diffusers \ --image input.jpg --prompt "A cat playing with yarn" \ --num_frames 81 --num_inference_steps 50 \ --guidance_scale 5.0 --fps 16 --output ti2v_output.mp4 ``` ## Cache-DiT Acceleration ```python omni = Omni( model="Wan-AI/Wan2.2-T2V-A14B-Diffusers", cache_backend="cache_dit", cache_config={ "Fn_compute_blocks": 8, "Bn_compute_blocks": 0, "max_warmup_steps": 4, "residual_diff_threshold": 0.12, }, ) ``` ## Key Parameters | Parameter | Default | Description | |-----------|---------|-------------| | `height` | 720 (T2V) / auto (I2V) | Video height (multiples of 16) | | `width` | 1280 (T2V) / auto (I2V) | Video width (multiples of 16) | | `num_frames` | 81 | Frames to generate | | `num_inference_steps` | 40–50 | Denoising steps | | `guidance_scale` | 4.0–5.0 | Classifier-free guidance scale | | `boundary_ratio` | 0.875 | MoE boundary split ratio | | `flow_shift` | 5.0 (720p) / 12.0 (480p) | Scheduler flow shift | ## References - [Wan2.2-T2V-A14B](https://huggingface.co/Wan-AI/Wan2.2-T2V-A14B-Diffusers) - [Wan2.2-I2V-A14B](https://huggingface.co/Wan-AI/Wan2.2-I2V-A14B-Diffusers) - [Wan2.2-TI2V-5B](https://huggingface.co/Wan-AI/Wan2.2-TI2V-5B-Diffusers) - [Cache-DiT Acceleration](https://github.com/vipshop/cache-dit) --- # XiaomiMiMo/MiMo-V2.5-Pro on vLLM Source: https://recipes.vllm.ai/XiaomiMiMo/MiMo-V2.5-Pro Provider: MiMo (Xiaomi) Released: 2026-04-27T12:52:53.000Z Updated: 2026-04-27 Architecture: moe Parameters: 1T Context length: 1,048,576 Min vLLM: 0.21.0 Tasks: text Verified hardware: H200 Summary: Xiaomi's flagship MoE reasoning model (1.02T total / 42B active) with hybrid attention, native FP8 weights, and Multi-Token Prediction ## Variants - default: precision=fp8, vram_minimum_gb=1224 Native FP8 weights (block-wise e4m3 128x128); 8x H200 with TP8 ## Recommended base flags ```bash vllm serve 'XiaomiMiMo/MiMo-V2.5-Pro' \ --trust-remote-code \ --max-model-len auto \ --generation-config \ vllm ``` ## Guide ## Overview MiMo-V2.5-Pro is Xiaomi's flagship MoE reasoning model with 1.02T total parameters and 42B active per token. It uses 384 routed experts (top-8) with hybrid attention (full-attention + sliding-window 128 at 6:1 ratio) over 70 layers (1 dense + 69 MoE) and ships with native FP8 (block-wise e4m3) weights. A 3-layer Multi-Token Prediction (MTP) head enables speculative decoding for ~3x output speed. ## Prerequisites - Hardware: 8x H200 (TP8) ### Pull the vLLM docker image Stable vLLM does not yet support MiMo V2.5. Use the pre-built image: ```bash docker pull vllm/vllm-openai:mimov25-cu129 ``` ## Launch commands Single-node TP8 (H200): ```bash vllm serve XiaomiMiMo/MiMo-V2.5-Pro \ --tensor-parallel-size 8 \ --trust-remote-code \ --gpu-memory-utilization 0.95 \ --max-model-len auto \ --generation-config vllm ``` With tool calling + reasoning: ```bash vllm serve XiaomiMiMo/MiMo-V2.5-Pro \ --tensor-parallel-size 8 \ --trust-remote-code \ --gpu-memory-utilization 0.95 \ --max-model-len auto \ --reasoning-parser mimo \ --tool-call-parser mimo \ --enable-auto-tool-choice \ --generation-config vllm ``` ## Client Usage ```bash curl -X POST http://localhost:8000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "XiaomiMiMo/MiMo-V2.5-Pro", "messages": [{"role": "user", "content": "Hello MiMo!"}], "chat_template_kwargs": {"enable_thinking": true} }' ``` Set `"enable_thinking": false` (or omit the kwargs) to disable thinking mode. ## References - [MiMo-V2.5-Pro on Hugging Face](https://huggingface.co/XiaomiMiMo/MiMo-V2.5-Pro) --- # XiaomiMiMo/MiMo-V2.5 on vLLM Source: https://recipes.vllm.ai/XiaomiMiMo/MiMo-V2.5 Provider: MiMo (Xiaomi) Released: 2026-04-27T13:37:38.000Z Updated: 2026-04-27 Architecture: moe Parameters: 311B Context length: 1,048,576 Min vLLM: 0.21.0 Tasks: multimodal, text Verified hardware: H200 Summary: MiMo-V2.5 is a native omnimodal model with strong agentic capabilities, supporting text, image, video, and audio understanding within a unified architecture ## Variants - default: precision=fp8, vram_minimum_gb=374 Native FP8 weights (block-wise e4m3 128x128); 4x H200 with TP4 ## Recommended base flags ```bash vllm serve 'XiaomiMiMo/MiMo-V2.5' \ --trust-remote-code \ --generation-config \ vllm ``` ## Guide ## Overview MiMo-V2.5 is Xiaomi's native omnimodal MoE model with 310B total parameters and 15B active per token, supporting text, image, video, and audio understanding within a unified architecture. Built on the MiMo-V2-Flash backbone with dedicated vision (729M) and audio (261M) encoders, it uses 256 routed experts (top-8) with hybrid attention (SWA-128 + full-attention at 5:1 ratio) over 48 layers (1 dense + 47 MoE) and ships with native FP8 (block-wise e4m3) weights. A 3-layer Multi-Token Prediction (MTP) head is included for speculative decoding. ## Prerequisites - Hardware: 4x H200 (TP4) ### Pull the vLLM docker image Stable vLLM does not yet support MiMo V2.5. Use the pre-built image: ```bash docker pull vllm/vllm-openai:mimov25-cu129 ``` ## Launch commands Single-node TP4 (H200): ```bash vllm serve XiaomiMiMo/MiMo-V2.5 \ --tensor-parallel-size 4 \ --trust-remote-code \ --gpu-memory-utilization 0.95 \ --max-model-len auto \ --generation-config vllm ``` With tool calling + reasoning: ```bash vllm serve XiaomiMiMo/MiMo-V2.5 \ --tensor-parallel-size 4 \ --trust-remote-code \ --gpu-memory-utilization 0.95 \ --max-model-len auto \ --reasoning-parser mimo \ --tool-call-parser mimo \ --enable-auto-tool-choice \ --generation-config vllm ``` Tunable flags: - `--max-model-len` — full context is 1,048,576; use `auto` to size to KV budget. - `--max-num-batched-tokens=32768` for prompt-heavy workloads; lower for latency. - `--gpu-memory-utilization=0.95` to maximize KV cache. ## Client Usage ```bash curl -X POST http://localhost:8000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "XiaomiMiMo/MiMo-V2.5", "messages": [{"role": "user", "content": "Hello MiMo!"}], "chat_template_kwargs": {"enable_thinking": true} }' ``` Set `"enable_thinking": false` (or omit the kwargs) to disable thinking mode. ## Benchmarking Launch the server with `--no-enable-prefix-caching` to get consistent measurements. ### VisionArena-Chat ``` vllm bench serve \ --model XiaomiMiMo/MiMo-V2.5 \ --backend openai-chat \ --endpoint /v1/chat/completions \ --dataset-name hf \ --dataset-path lmarena-ai/VisionArena-Chat \ --num-prompts 128 ``` ### Random Synthetic ```bash vllm bench serve \ --model XiaomiMiMo/MiMo-V2.5 \ --dataset-name random --random-input-len 8000 --random-output-len 1000 \ --request-rate 3 --num-prompts 1800 --ignore-eos ``` ## References - [MiMo-V2.5 on Hugging Face](https://huggingface.co/XiaomiMiMo/MiMo-V2.5) --- # ByteDance-Seed/Seed-OSS-36B-Instruct on vLLM Source: https://recipes.vllm.ai/ByteDance-Seed/Seed-OSS-36B-Instruct Provider: Seed (ByteDance) Released: 2025-08-20T15:03:26.000Z Updated: 2026-04-17 Architecture: dense Parameters: 36B Context length: 524,288 Min vLLM: 0.11.0 Tasks: text Summary: ByteDance Seed-OSS 36B dense model with unique 'thinking budget' control and 512K context support ## Variants - default: precision=bf16, vram_minimum_gb=86 Native BF16 weights on 8x GPU (TP=8) ## Guide ## Overview Seed-OSS-36B is a dense language model from ByteDance Seed with a unique **thinking budget** feature for controlled reasoning, and up to 512K context. Users can choose tensor-parallel (low latency) or data-parallel (high throughput). ## Prerequisites - Hardware: 8x GPU recommended (TP=8); also runs on consumer hardware like RTX 3090 - vLLM >= 0.11.0 (support may require main branch) - Latest transformers for compatibility ### Install vLLM (NVIDIA) ```bash uv venv source .venv/bin/activate uv pip install -U vllm --torch-backend auto uv pip install git+https://github.com/huggingface/transformers.git@56d68c6706ee052b445e1e476056ed92ac5eb383 ``` ### Install vLLM (AMD ROCm) ```bash uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/ ``` ## Launch commands NVIDIA: ```bash vllm serve ByteDance-Seed/Seed-OSS-36B-Instruct \ --host localhost --port 8000 \ --tensor-parallel-size 8 \ --enable-auto-tool-choice \ --tool-call-parser seed_oss ``` AMD: ```bash export VLLM_ROCM_USE_AITER=1 vllm serve ByteDance-Seed/Seed-OSS-36B-Instruct \ --tensor-parallel-size 8 \ --enable-auto-tool-choice --tool-call-parser seed_oss \ --trust-remote-code ``` Tuning: - `--max-model-len=65536` works well; max is 512K. - `--max-num-batched-tokens=32768` for prompt-heavy; reduce to 8K–16K for latency. - `--gpu-memory-utilization=0.95` to maximize KV cache. ## Thinking Budget Control the model's chain-of-thought length via `chat_template_kwargs`. Recommended values are multiples of 512 (512, 1K, 2K, 4K, 8K, 16K). Use `0` for direct answers. ```python from openai import OpenAI client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1") model = client.models.list().data[0].id response = client.chat.completions.create( model=model, messages=[ {"role": "system", "content": "You are a helpful assistant"}, {"role": "user", "content": "Janet's ducks lay 16 eggs per day..."}, ], extra_body={"chat_template_kwargs": {"thinking_budget": 512}}, ) print(response.choices[0].message.content) ``` curl: ```bash curl http://localhost:8000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "ByteDance-Seed/Seed-OSS-36B-Instruct", "messages": [{"role": "user", "content": "Explain quantum computing"}], "chat_template_kwargs": {"thinking_budget": 512} }' ``` The model emits `` blocks with `` markers that report token usage against the budget. ## Benchmarking ```bash vllm bench serve \ --backend vllm --model ByteDance-Seed/Seed-OSS-36B-Instruct \ --endpoint /v1/completions --host localhost --port 8000 \ --dataset-name random --random-input 800 --random-output 100 \ --request-rate 2 --num-prompt 100 ``` ## References - [Seed-OSS-36B-Instruct on Hugging Face](https://huggingface.co/ByteDance-Seed/Seed-OSS-36B-Instruct) --- # Google/translategemma-27b-it on vLLM Source: https://recipes.vllm.ai/Google/translategemma-27b-it Provider: Google Released: 2026-01-12T16:12:41.000Z Updated: 2026-04-17 Architecture: dense Parameters: 27B Context length: 131,072 Min vLLM: 0.14.1 Tasks: text Summary: Lightweight open translation model from Google (based on Gemma 3) supporting 55 languages. Served via the vLLM-optimized Infomaniak-AI checkpoint. ## Variants - default: precision=bf16, vram_minimum_gb=65 Original Google checkpoint in BF16 (has vLLM compatibility issues — prefer vllm-optimized variant) - vllm_optimized: precision=bf16, vram_minimum_gb=65, model_id=Infomaniak-AI/vllm-translategemma-27b-it Infomaniak-AI vLLM-optimized checkpoint — recommended. Fixes RoPE config, EOS token, and replaces custom JSON inputs with string delimiters. - small_4b: precision=bf16, vram_minimum_gb=10, model_id=google/translategemma-4b-it 4B variant for lower-resource deployments (prefer Infomaniak-AI/vllm-translategemma-4b-it) ## Guide ## Overview [TranslateGemma](https://huggingface.co/collections/google/translategemma) is a family of lightweight, state-of-the-art open translation models from Google, based on the Gemma 3 family. TranslateGemma models handle translation across 55 languages and are small enough to deploy on laptops, desktops, and modest cloud GPU environments. ### Original Models - [google/translategemma-27b-it](https://huggingface.co/google/translategemma-27b-it) - [google/translategemma-4b-it](https://huggingface.co/google/translategemma-4b-it) ### Optimized vLLM Models - [Infomaniak-AI/vllm-translategemma-27b-it](https://huggingface.co/Infomaniak-AI/vllm-translategemma-27b-it) - [Infomaniak-AI/vllm-translategemma-4b-it](https://huggingface.co/Infomaniak-AI/vllm-translategemma-4b-it) ### Why use the vLLM-optimized models? The original Google models have compatibility issues with standard inference engines like vLLM. The optimized versions from Infomaniak-AI ([detailed changes](https://huggingface.co/Infomaniak-AI/vllm-translategemma-27b-it#changes-from-original-model)): - **vLLM Compatibility**: Originals require custom JSON parameters (`source_lang_code`/`target_lang_code`). The optimized version uses string delimiters. - **RoPE Simplification**: Originals use a complex RoPE configuration for sliding attention. Optimized uses a standard linear RoPE format (`factor: 8.0`). - **EOS Token Fix**: Corrects the EOS token from `` to ``. ## Prerequisites ### Docker ```bash docker pull vllm/vllm-openai:latest ``` ## Deployment Configurations Verified for both 4B and 27B: ```bash docker run -itd --name google-translategemma-27b-it \ --ipc=host \ --network host \ --shm-size 16G \ --gpus all \ -v ~/.cache/huggingface:/root/.cache/huggingface \ vllm/vllm-openai:latest \ Infomaniak-AI/vllm-translategemma-27b-it \ --served-model-name translategemma-27b-it \ --gpu-memory-utilization 0.8 \ --host 0.0.0.0 \ --port 8000 ``` ## Client Usage Tips: - **Prompt Delimiters**: Encode language metadata directly in the content string: `<<>>{src_lang}<<>>{tgt_lang}<<>>{text}` - **Language Codes**: ISO 639-1 Alpha-2 (e.g. `en`, `zh`) and regional variants (e.g. `en_US`, `zh_CN`). - **Context Limit**: ~2K tokens. ### cURL ```bash curl -X POST http://localhost:8000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "translategemma-27b-it", "messages": [{ "role": "user", "content": "<<>>en<<>>zh<<>>We distribute two models for language identification, which can recognize 176 languages." }] }' ``` ## References - [TranslateGemma collection](https://huggingface.co/collections/google/translategemma) - [Infomaniak-AI optimized 27B](https://huggingface.co/Infomaniak-AI/vllm-translategemma-27b-it) - [Infomaniak-AI optimized 4B](https://huggingface.co/Infomaniak-AI/vllm-translategemma-4b-it) --- # MiniMaxAI/MiniMax-M2.1 on vLLM Source: https://recipes.vllm.ai/MiniMaxAI/MiniMax-M2.1 Provider: MiniMax Released: 2025-12-20T05:45:05.000Z Updated: 2026-04-17 Architecture: moe Parameters: 230B Context length: 196,608 Min vLLM: 0.11.1 Tasks: text Verified hardware: H100, MI300X, MI325X, MI355X Summary: MiniMax M2.1 MoE language model (230B total / 10B active) for coding, agent toolchains, and long-context reasoning — native FP8 checkpoint ## Variants - default: precision=fp8, vram_minimum_gb=276 Native FP8 checkpoint — 4x H200/H20/H100 or 4x A100/A800 for weights, plus KV cache headroom ## Recommended base flags ```bash vllm serve 'MiniMaxAI/MiniMax-M2.1' \ --trust-remote-code ``` ## Guide ## Overview [MiniMax-M2.1](https://huggingface.co/MiniMaxAI/MiniMax-M2.1) is part of the MiniMax M2 series of advanced MoE language models. It retains the M2 architecture (10B active, 230B total) with improvements over the original M2 release. Supports 196K context per sequence. ## Prerequisites - **OS:** Linux - **Python:** 3.10 - 3.13 - **NVIDIA:** compute capability >= 7.0; ~220 GB for weights + 240 GB per 1M context tokens - **AMD:** MI300X / MI325X / MI350X / MI355X with ROCm 7.0+ ### Install vLLM (NVIDIA) ```bash uv venv source .venv/bin/activate uv pip install -U vllm --torch-backend auto ``` ### Docker (dedicated M2-series image) ```bash docker run --gpus all \ -p 8000:8000 \ --ipc=host \ -v ~/.cache/huggingface:/root/.cache/huggingface \ vllm/vllm-openai:minimax27 MiniMaxAI/MiniMax-M2.1 \ --tensor-parallel-size 4 \ --tool-call-parser minimax_m2 \ --reasoning-parser minimax_m2 \ --enable-auto-tool-choice \ --trust-remote-code ``` ## Launching the Server ### NVIDIA — TP4 ```bash vllm serve MiniMaxAI/MiniMax-M2.1 \ --tensor-parallel-size 4 \ --tool-call-parser minimax_m2 \ --reasoning-parser minimax_m2 \ --enable-auto-tool-choice \ --trust-remote-code ``` Pure TP8 is not supported. For >4 GPUs use DP+EP or TP+EP. ### TP4+EP (recommended for H100) ```bash vllm serve MiniMaxAI/MiniMax-M2.1 \ --tensor-parallel-size 4 \ --enable-expert-parallel \ --tool-call-parser minimax_m2 \ --reasoning-parser minimax_m2 \ --enable-auto-tool-choice ``` ### AMD ROCm — TP2 ```bash VLLM_ROCM_USE_AITER=1 vllm serve MiniMaxAI/MiniMax-M2.1 \ --tensor-parallel-size 2 \ --tool-call-parser minimax_m2 \ --reasoning-parser minimax_m2 \ --enable-auto-tool-choice \ --trust-remote-code ``` ## Benchmarking ```bash vllm bench serve \ --backend vllm \ --model MiniMaxAI/MiniMax-M2.1 \ --endpoint /v1/completions \ --dataset-name random \ --random-input 2048 \ --random-output 1024 \ --max-concurrency 10 \ --num-prompt 100 ``` ## Troubleshooting - See [MiniMax-M2](./MiniMax-M2.yaml) for shared troubleshooting notes (latest vLLM fusion behavior, nightly vs stable, DeepGEMM, AITER). ## References - [Model card](https://huggingface.co/MiniMaxAI/MiniMax-M2.1) - [MiniMax](https://www.minimax.io/) --- # OpenGVLab/InternVL3_5-8B on vLLM Source: https://recipes.vllm.ai/OpenGVLab/InternVL3_5-8B Provider: InternVL (OpenGVLab) Released: 2025-08-25T16:38:47.000Z Updated: 2026-04-17 Architecture: dense Parameters: 8B Context length: 40,960 Min vLLM: 0.10.0 Tasks: multimodal Verified hardware: H100, MI300X, MI325X, MI355X Summary: InternVL 3.5 vision-language models from Shanghai AI Lab with thinking-mode prompting ## Variants - default: precision=bf16, vram_minimum_gb=19 BF16 weights for the 8B variant ## Recommended base flags ```bash vllm serve 'OpenGVLab/InternVL3_5-8B' \ --trust-remote-code ``` ## Guide ## Overview [InternVL3.5](https://github.com/OpenGVLab/InternVL) is a vision-language model developed by Shanghai AI Laboratory. It supports single-image and multi-image prompts, plus an optional "thinking mode" via a custom system prompt. ## Prerequisites - Hardware: 1x GPU with >=20 GB VRAM (A100, L40S, H100, etc.) - vLLM >= 0.10.0 ### Install vLLM (CUDA) ```bash uv venv source .venv/bin/activate uv pip install -U vllm --torch-backend auto ``` ### Install vLLM (AMD ROCm MI300X/MI325X/MI355X) ```bash uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/0.14.1/rocm700 ``` ## Launch command ```bash vllm serve OpenGVLab/InternVL3_5-8B --trust-remote-code ``` On AMD: ```bash export VLLM_ROCM_USE_AITER=1 vllm serve OpenGVLab/InternVL3_5-8B --trust-remote-code ``` ## Client Usage Single image: ```python from openai import OpenAI client = OpenAI(api_key="", base_url="http://0.0.0.0:8000/v1") model_name = client.models.list().data[0].id response = client.chat.completions.create( model=model_name, messages=[{ "role": "user", "content": [ {"type": "text", "text": "Describe the image."}, {"type": "image_url", "image_url": {"url": "https://raw.githubusercontent.com/open-mmlab/mmdeploy/main/tests/data/tiger.jpeg"}}, ], }], temperature=0.0, ) print(response.choices[0].message.content) ``` ## Thinking Mode Set a thinking system prompt and use `temperature=0.6` to mitigate repetition: ```python THINKING_SYSTEM_PROMPT = """ You are an AI assistant that rigorously follows this response protocol: 1. First, conduct a detailed analysis of the question. Consider different angles, potential solutions, and reason through the problem step-by-step. Enclose this entire thinking process within and tags. 2. After the thinking section, provide a clear, concise, and direct answer to the user's question. Separate the answer from the think section with a newline. """.strip() ``` ## References - [InternVL3.5-8B](https://huggingface.co/OpenGVLab/InternVL3_5-8B) - [InternVL GitHub](https://github.com/OpenGVLab/InternVL) --- # PaddlePaddle/PaddleOCR-VL on vLLM Source: https://recipes.vllm.ai/PaddlePaddle/PaddleOCR-VL Provider: PaddlePaddle Released: 2025-10-16T10:14:45.000Z Updated: 2026-04-17 Architecture: dense Parameters: 0.9B Context length: 131,072 Min vLLM: 0.11.1 Tasks: multimodal Summary: PaddleOCR-VL (0.9B) — compact vision-language model for document parsing, OCR, tables, formulas, charts ## Variants - default: precision=bf16, vram_minimum_gb=2 BF16 weights — small footprint, runs on most GPUs ## Recommended base flags ```bash vllm serve 'PaddlePaddle/PaddleOCR-VL' \ --trust-remote-code \ --max-num-batched-tokens \ 16384 \ --no-enable-prefix-caching \ --mm-processor-cache-gb \ 0 ``` ## Guide ## Overview [PaddleOCR-VL](https://huggingface.co/PaddlePaddle/PaddleOCR-VL) is a SOTA resource-efficient model for document parsing. Its core (PaddleOCR-VL-0.9B) combines a NaViT-style dynamic resolution visual encoder with an ERNIE-4.5-0.3B language model, optimized for OCR, tables, formulas, and chart recognition. ## Prerequisites - Hardware: 1x GPU (small VRAM footprint) - vLLM >= 0.11.1 (nightly if not released yet) - AMD ROCm wheels require Python 3.12, ROCm 7.2.3, and glibc >= 2.35. ### Install vLLM (NVIDIA) ```bash uv venv source .venv/bin/activate uv pip install -U vllm --pre \ --extra-index-url https://wheels.vllm.ai/nightly \ --extra-index-url https://download.pytorch.org/whl/cu129 \ --index-strategy unsafe-best-match ``` ### Install vLLM (AMD ROCm) ```bash uv venv --python 3.12 source .venv/bin/activate uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm ``` ## Launch command ```bash vllm serve PaddlePaddle/PaddleOCR-VL \ --trust-remote-code \ --max-num-batched-tokens 16384 \ --no-enable-prefix-caching \ --mm-processor-cache-gb 0 ``` AMD MI300X/MI325X/MI355X: ```bash export SAFETENSORS_FAST_GPU=1 vllm serve PaddlePaddle/PaddleOCR-VL \ --trust-remote-code \ --max-num-batched-tokens 16384 \ --no-enable-prefix-caching \ --mm-processor-cache-gb 0 ``` Tip: OCR workloads don't benefit much from prefix caching or image reuse, so disable those to avoid hashing/caching overhead. ## Client Usage Task-specific prompts: ```python from openai import OpenAI client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1", timeout=3600) TASKS = { "ocr": "OCR:", "table": "Table Recognition:", "formula": "Formula Recognition:", "chart": "Chart Recognition:", } response = client.chat.completions.create( model="PaddlePaddle/PaddleOCR-VL", messages=[{ "role": "user", "content": [ {"type": "image_url", "image_url": {"url": "https://.../receipt.png"}}, {"type": "text", "text": TASKS["ocr"]}, ], }], temperature=0.0, ) print(response.choices[0].message.content) ``` ## Offline Inference with PP-DocLayoutV2 Use separate venvs for `vllm` and `paddlepaddle` to avoid conflicts. If you see "The model PaddleOCR-VL-0.9B does not exist.", add `--served-model-name PaddleOCR-VL-0.9B`. ```bash uv pip install paddlepaddle-gpu==3.2.1 --extra-index-url https://www.paddlepaddle.org.cn/packages/stable/cu126/ uv pip install -U "paddleocr[doc-parser]" uv pip install safetensors ``` ```python from paddleocr import PaddleOCRVL pipeline = PaddleOCRVL( vl_rec_backend="vllm-server", vl_rec_server_url="http://localhost:8000/v1", layout_detection_model_name="PP-DocLayoutV2", layout_detection_model_dir="/path/to/your/PP-DocLayoutV2/", ) output = pipeline.predict("https://paddle-model-ecology.bj.bcebos.com/paddlex/imgs/demo_image/paddleocr_vl_demo.png") for i, res in enumerate(output): res.save_to_json(save_path=f"output_{i}.json") res.save_to_markdown(save_path=f"output_{i}.md") ``` ## References - [PaddleOCR-VL on Hugging Face](https://huggingface.co/PaddlePaddle/PaddleOCR-VL) - [PaddleOCR GitHub](https://github.com/PaddlePaddle/PaddleOCR) --- # Qwen/Qwen-Image on vLLM Source: https://recipes.vllm.ai/Qwen/Qwen-Image Provider: Qwen Released: 2025-08-02T04:58:07.000Z Updated: 2026-04-17 Architecture: dense Parameters: 20B Min vLLM: 0.18.0 Tasks: omni Summary: Text-to-image diffusion model (20B parameters) from the Qwen-Image family, served via vLLM-Omni. ## Variants - default: precision=bf16, vram_minimum_gb=48 Full precision BF16 — use CPU offload / layerwise offload for lower VRAM - fp8: precision=fp8, vram_minimum_gb=24 FP8 quantization with `img_mlp` kept at full precision for quality. - int8: precision=int8, vram_minimum_gb=24 INT8 quantization (`--quantization int8`). ## Guide ## Overview Qwen-Image is a diffusion-based text-to-image model. This recipe documents the Qwen-Image family served via **vLLM-Omni**: | Model | HuggingFace | Description | |-------|-------------|-------------| | **Qwen-Image** | [Qwen/Qwen-Image](https://huggingface.co/Qwen/Qwen-Image) | Text-to-image (20B, Aug 2025) | | **Qwen-Image-2512** | [Qwen/Qwen-Image-2512](https://huggingface.co/Qwen/Qwen-Image-2512) | Updated T2I with enhanced realism (Dec 2025) | | **Qwen-Image-Edit** | [Qwen/Qwen-Image-Edit](https://huggingface.co/Qwen/Qwen-Image-Edit) | Single-image editing (Aug 2025) | | **Qwen-Image-Edit-2509** | [Qwen/Qwen-Image-Edit-2509](https://huggingface.co/Qwen/Qwen-Image-Edit-2509) | Multi-image editing (Sep 2025) | | **Qwen-Image-Edit-2511** | [Qwen/Qwen-Image-Edit-2511](https://huggingface.co/Qwen/Qwen-Image-Edit-2511) | Enhanced consistency + built-in LoRA (Nov 2025) | | **Qwen-Image-Layered** | [Qwen/Qwen-Image-Layered](https://huggingface.co/Qwen/Qwen-Image-Layered) | Decomposes input into RGBA layers (Dec 2025) | All models share the same DiT transformer core — acceleration methods are applicable across the entire series. ## Prerequisites ```bash git clone https://github.com/vllm-project/vllm-omni.git cd vllm-omni uv venv source .venv/bin/activate uv pip install -e . vllm==0.18.0 ``` ## Usage ### Text-to-Image ```bash python3 ./examples/offline_inference/text_to_image/text_to_image.py \ --model Qwen/Qwen-Image \ --prompt "a cup of coffee on the table" \ --output output_qwen_image.png \ --num-inference-steps 50 \ --cfg-scale 4.0 ``` ### Image Editing (Qwen-Image-Edit) ```bash python3 ./examples/offline_inference/image_to_image/image_edit.py \ --model Qwen/Qwen-Image-Edit \ --image qwen_bear.png \ --prompt "Let this mascot dance under the moon, surrounded by floating stars" \ --output output_image_edit.png \ --num-inference-steps 50 \ --cfg-scale 4.0 ``` ### Layered RGBA Decomposition ```bash python3 ./examples/offline_inference/image_to_image/image_edit.py \ --model Qwen/Qwen-Image-Layered \ --image input.png \ --prompt "" \ --output layered \ --num-inference-steps 50 \ --cfg-scale 4.0 \ --layers 4 \ --color-format "RGBA" ``` ## Acceleration Pick one cache backend AND any supported parallel strategy. ### Cache-DiT ```bash python3 ./examples/offline_inference/text_to_image/text_to_image.py \ --model Qwen/Qwen-Image --prompt "..." --cache-backend cache_dit ``` ### TeaCache ```bash python3 ./examples/offline_inference/text_to_image/text_to_image.py \ --model Qwen/Qwen-Image --prompt "..." --cache-backend tea_cache ``` ### Ulysses / Ring Sequence Parallelism ```bash python3 ./examples/offline_inference/text_to_image/text_to_image.py \ --model Qwen/Qwen-Image --prompt "..." --ulysses-degree 4 ``` ```bash python3 ./examples/offline_inference/text_to_image/text_to_image.py \ --model Qwen/Qwen-Image --prompt "..." --ring-degree 4 ``` ### CFG Parallelism (2 GPUs, non-distilled models with `cfg-scale > 1`) ```bash python3 ./examples/offline_inference/image_to_image/image_edit.py \ --model Qwen/Qwen-Image-Edit --image qwen_bear.png --prompt "..." \ --cfg-parallel-size 2 --num-inference-steps 50 --cfg-scale 4.0 ``` ### Tensor Parallelism ```bash python3 ./examples/offline_inference/text_to_image/text_to_image.py \ --model Qwen/Qwen-Image --prompt "..." --tensor-parallel-size 2 ``` ### CPU / Layerwise Offload (low VRAM) ```bash python3 ./examples/offline_inference/text_to_image/text_to_image.py \ --model Qwen/Qwen-Image --prompt "..." --enable-cpu-offload ``` ```bash python3 ./examples/offline_inference/image_to_image/image_edit.py \ --model Qwen/Qwen-Image-Edit --image qwen_bear.png --prompt "..." \ --enable-layerwise-offload ``` ### VAE Patch Parallelism ```bash python3 ./examples/offline_inference/text_to_image/text_to_image.py \ --model Qwen/Qwen-Image --prompt "..." \ --height 1536 --width 1536 \ --ulysses-degree 2 --vae-patch-parallel-size 2 ``` Must be combined with another parallel method. ### Quantization (Qwen-Image / Qwen-Image-2512 only) ```bash python3 ./examples/offline_inference/text_to_image/text_to_image.py \ --model Qwen/Qwen-Image --prompt "..." --quantization fp8 \ --ignored-layers "img_mlp" ``` ```bash python3 ./examples/offline_inference/text_to_image/text_to_image.py \ --model Qwen/Qwen-Image --prompt "..." --quantization int8 ``` Qwen-Image-Edit variants do **not** support quantization. ## Configuration Tips - **Cache + SP** is the recommended combo for long-sequence generation. - **Sequence parallelism** (Ulysses / Ring) beats TP for high-res / long-sequence. - **Tensor parallelism** is most useful when model weights alone don't fit on one GPU. - **CFG parallelism** targets non-distilled diffusion with full CFG (not for guidance-distilled models). - To reduce peak VRAM, use CPU/layerwise offload and/or VAE patch parallelism. - TeaCache and Cache-DiT cannot be used together. - `--enforce-eager` disables torch.compile if needed. See the [Feature Support Table](https://github.com/vllm-project/vllm-omni/blob/main/docs/user_guide/diffusion_features.md#supported-models) and [Feature Compatibility Guide](https://github.com/vllm-project/vllm-omni/blob/main/docs/user_guide/feature_compatibility.md) for combinations. ## References - [Model card](https://huggingface.co/Qwen/Qwen-Image) - [vLLM-Omni](https://github.com/vllm-project/vllm-omni) --- # Qwen/Qwen2.5-VL-72B-Instruct on vLLM Source: https://recipes.vllm.ai/Qwen/Qwen2.5-VL-72B-Instruct Provider: Qwen Released: 2025-01-27T04:12:04.000Z Updated: 2026-04-17 Architecture: dense Parameters: 72B Context length: 128,000 Min vLLM: 0.10.2 Tasks: multimodal, text Verified hardware: MI300X, MI325X, MI355X Summary: Qwen2.5-VL dense vision-language model (72B) for high-quality image and video understanding. ## Variants - default: precision=bf16, vram_minimum_gb=173 Full precision BF16 — 4x A100 80GB or 4x MI300X/MI325X/MI355X - awq: precision=int4, vram_minimum_gb=43, model_id=Qwen/Qwen2.5-VL-72B-Instruct-AWQ AWQ 4-bit quantized weights - fp8: precision=fp8, vram_minimum_gb=87, model_id=RedHatAI/Qwen2.5-VL-72B-Instruct-FP8-dynamic RedHatAI FP8 (E4M3) with dynamic per-token activation quantization ## Guide ## Overview This guide describes how to run Qwen2.5-VL series on the targeted accelerated stack. Since BF16 is the commonly used precision for Qwen2.5-VL training, using BF16 in inference ensures the best accuracy. ## Prerequisites ### NVIDIA ```bash uv venv source .venv/bin/activate uv pip install -U vllm --torch-backend auto ``` ### AMD ROCm (MI300X, MI325X, MI355X) > Note: The vLLM wheel for ROCm requires Python 3.12, ROCm 7.0, and glibc >= 2.35. Use the Docker flow if your environment is incompatible. ```bash uv venv source .venv/bin/activate uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm ``` ### TPU Deployment TPU support has been verified for the smaller [Qwen2.5-VL-7B-Instruct](/Qwen/Qwen2.5-VL-7B-Instruct) on a single v6e chip — the 72B is not yet validated on Trillium. See the [upstream TPU recipe](https://github.com/AI-Hypercomputer/tpu-recipes/tree/main/inference/trillium/vLLM/Qwen2.5-VL) (7B). ## Deployment Configurations ### 4xA100 (BF16, TP=4) ```bash export CUDA_VISIBLE_DEVICES=0,1,2,3 vllm serve Qwen/Qwen2.5-VL-72B-Instruct \ --host 0.0.0.0 \ --port 8000 \ --tensor-parallel-size 4 \ --mm-encoder-tp-mode data \ --limit-mm-per-prompt '{"image":2,"video":0}' ``` ### 4xMI300X/MI325X/MI355X (BF16, TP=4) ```bash export CUDA_VISIBLE_DEVICES=0,1,2,3 export VLLM_ROCM_USE_AITER=1 vllm serve Qwen/Qwen2.5-VL-72B-Instruct \ --host 0.0.0.0 \ --port 8000 \ --tensor-parallel-size 4 \ --mm-encoder-tp-mode data \ --limit-mm-per-prompt '{"image":2,"video":0}' ``` ### Qwen2.5-VL-7B-Instruct (DP=4) For medium-size 7B model, data parallelism works better. ```bash export CUDA_VISIBLE_DEVICES=0,1,2,3 vllm serve Qwen/Qwen2.5-VL-7B-Instruct \ --host 0.0.0.0 \ --port 8000 \ --data-parallel-size 4 \ --limit-mm-per-prompt '{"image":2,"video":0}' ``` ## Configuration Tips - `--max-model-len 65536` is usually good for most scenarios (native context is 128K). - For A100-80GB devices, TP must be >= 2 to avoid OOM. - `--limit-mm-per-prompt` caps incoming multimodal requests. - `--mm-encoder-tp-mode data` deploys the small ViT encoder in DP fashion (ViT ≈ 675M vs 72B LM). - vLLM uses 90% of GPU memory by default; set `--gpu-memory-utilization=0.95` to maximize KV cache. ## Benchmarking Launch the server with `--no-enable-prefix-caching` to get consistent measurements. ### VisionArena-Chat ```bash vllm bench serve \ --host 0.0.0.0 \ --port 8000 \ --backend openai-chat \ --endpoint /v1/chat/completions \ --model Qwen/Qwen2.5-VL-72B-Instruct \ --dataset-name hf \ --dataset-path lmarena-ai/VisionArena-Chat \ --num-prompts 128 ``` ### Random Synthetic ```bash vllm bench serve \ --host 0.0.0.0 \ --port 8000 \ --model Qwen/Qwen2.5-VL-72B-Instruct \ --dataset-name random \ --random-input-len 8000 \ --random-output-len 1000 \ --num-prompts 128 ``` Workload mixes: - Prompt-heavy: 8000 in / 1000 out - Decode-heavy: 1000 in / 8000 out - Balanced: 1000 in / 1000 out ## References - [Model card](https://huggingface.co/Qwen/Qwen2.5-VL-72B-Instruct) - [vLLM multimodal inputs guide](https://docs.vllm.ai/en/latest/features/multimodal_inputs.html) --- # Qwen/Qwen3-235B-A22B-Instruct-2507 on vLLM Source: https://recipes.vllm.ai/Qwen/Qwen3-235B-A22B-Instruct-2507 Provider: Qwen Released: 2025-07-21T06:46:56.000Z Updated: 2026-04-17 Architecture: moe Parameters: 235B Context length: 262,144 Min vLLM: 0.10.0 Tasks: text Verified hardware: H200, MI300X, MI325X, MI355X Summary: Flagship Qwen3 MoE instruct model with 235B total and 22B active parameters, tuned for high-quality text generation. ## Variants - default: precision=bf16, vram_minimum_gb=564 Full precision BF16 — requires 4x H200 or 8x MI300X/MI325X/MI355X - fp8: precision=fp8, vram_minimum_gb=240, model_id=Qwen/Qwen3-235B-A22B-FP8 Qwen official FP8 checkpoint for improved efficiency on SM90+ - nvfp4: precision=nvfp4, vram_minimum_gb=141, model_id=nvidia/Qwen3-235B-A22B-Instruct-2507-NVFP4 NVIDIA NVFP4 quantized weights for Blackwell GPUs - redhat_nvfp4: precision=nvfp4, vram_minimum_gb=152, model_id=RedHatAI/Qwen3-235B-A22B-Instruct-2507-NVFP4 RedHatAI NVFP4 quantization ## Recommended base flags ```bash vllm serve 'Qwen/Qwen3-235B-A22B-Instruct-2507' \ --trust-remote-code ``` ## Guide ## Overview [Qwen3](https://huggingface.co/Qwen/Qwen3-235B-A22B-Instruct-2507) is the flagship instruct MoE model in the Qwen3 series, with 235B total parameters and 22B active parameters. This guide covers deploying the model efficiently with vLLM on NVIDIA and AMD GPUs. ## Prerequisites ### NVIDIA CUDA (pip) ```bash uv venv source .venv/bin/activate uv pip install -U vllm --torch-backend=auto ``` ### AMD ROCm (pip) > Note: The vLLM wheel for ROCm requires Python 3.12, ROCm 7.0, and glibc >= 2.35. Use the Docker flow if your environment is incompatible. ```bash uv venv source .venv/bin/activate uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/0.14.1/rocm700 ``` ## Deployment Configurations ### BF16 on MI300X/MI325X/MI355X (4 GPUs) ```bash HIP_VISIBLE_DEVICES="4,5,6,7" \ VLLM_ROCM_USE_AITER=1 \ VLLM_ROCM_USE_AITER_MHA=0 \ SAFETENSORS_FAST_GPU=1 \ vllm serve Qwen/Qwen3-235B-A22B \ --trust-remote-code \ -tp 4 \ --swap-space 32 \ --distributed-executor-backend mp \ --max-num-batched-tokens 32768 \ --max-model-len 32768 \ --no-enable-prefix-caching \ --gpu-memory-utilization 0.8 ``` ### FP8 on MI300X/MI325X/MI355X (4 GPUs) ```bash HIP_VISIBLE_DEVICES="4,5,6,7" \ VLLM_ROCM_USE_AITER=1 \ VLLM_ROCM_USE_AITER_MHA=0 \ SAFETENSORS_FAST_GPU=1 \ vllm serve Qwen/Qwen3-235B-A22B-FP8 \ --trust-remote-code \ -tp 4 \ --swap-space 16 \ --distributed-executor-backend mp \ --max-num-batched-tokens 32768 \ --max-model-len 32768 \ --no-enable-prefix-caching \ --gpu-memory-utilization 0.8 ``` ### TPU Deployment - [Qwen3-32B on Trillium (v6e)](https://github.com/AI-Hypercomputer/tpu-recipes/tree/main/inference/trillium/vLLM/Qwen3) - [Qwen3-4B on Trillium (v6e)](https://github.com/AI-Hypercomputer/tpu-recipes/tree/main/inference/trillium/vLLM/Qwen3) ## Client Usage ```bash vllm bench serve \ --model "Qwen/Qwen3-235B-A22B-FP8" \ --dataset-name random \ --random-input-len 8192 \ --random-output-len 1024 \ --request-rate 10000 \ --num-prompts 16 \ --ignore-eos \ --trust-remote-code ``` ## Troubleshooting - Use `--max-num-batched-tokens` and `--max-model-len` to fit memory constraints on smaller nodes. - If OOM at startup, lower `--gpu-memory-utilization` (e.g. 0.8) and disable prefix caching. - AMD builds require `VLLM_ROCM_USE_AITER=1` for best performance. ## References - [Model card](https://huggingface.co/Qwen/Qwen3-235B-A22B-Instruct-2507) - [FP8 checkpoint](https://huggingface.co/Qwen/Qwen3-235B-A22B-FP8) - [vLLM documentation](https://docs.vllm.ai/) --- # Qwen/Qwen3-ASR-1.7B on vLLM Source: https://recipes.vllm.ai/Qwen/Qwen3-ASR-1.7B Provider: Qwen Released: 2026-01-28T03:22:40.000Z Updated: 2026-04-17 Architecture: dense Parameters: 2.3B Context length: 65,536 Min vLLM: 0.16.0 Tasks: multimodal Summary: Speech-to-text model supporting 11 languages, multiple accents, and singing voice with customizable text-context prompting. ## Variants - default: precision=bf16, vram_minimum_gb=4 Full precision BF16 — fits on a single mid-range GPU ## Guide ## Overview Qwen3-ASR is a speech-to-text model that achieves accurate and robust recognition across 11 languages and multiple accents. It supports prompting the model with text context in any format to produce customized ASR results and performs well on singing-voice recognition. This guide demonstrates how to deploy Qwen3-ASR efficiently with vLLM. ## Prerequisites Install vLLM with audio dependencies: ```bash uv venv source .venv/bin/activate uv pip install -U vllm --pre \ --extra-index-url https://wheels.vllm.ai/nightly/cu129 \ --extra-index-url https://download.pytorch.org/whl/cu129 \ --index-strategy unsafe-best-match uv pip install "vllm[audio]" ``` ## Launching with vLLM ```bash vllm serve Qwen/Qwen3-ASR-1.7B ``` ## Client Usage ### Chat Completions (OpenAI SDK) ```python from openai import OpenAI client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY") response = client.chat.completions.create( model="Qwen/Qwen3-ASR-1.7B", messages=[{ "role": "user", "content": [{ "type": "audio_url", "audio_url": {"url": "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen3-ASR-Repo/asr_en.wav"}, }], }], ) print(response.choices[0].message.content) ``` ### Transcription API ```python import httpx from openai import OpenAI client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY") audio_url = "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen3-ASR-Repo/asr_en.wav" audio_file = httpx.get(audio_url).content transcription = client.audio.transcriptions.create( model="Qwen/Qwen3-ASR-1.7B", file=audio_file, ) print(transcription.text) ``` ### cURL ```bash curl http://localhost:8000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "messages": [ {"role": "user", "content": [ {"type": "audio_url", "audio_url": {"url": "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen3-ASR-Repo/asr_en.wav"}} ]} ] }' ``` ### Offline Inference ```python from vllm import LLM, SamplingParams from vllm.assets.audio import AudioAsset llm = LLM(model="Qwen/Qwen3-ASR-1.7B") audio_asset = AudioAsset("winning_call") conversation = [{ "role": "user", "content": [{"type": "audio_url", "audio_url": {"url": audio_asset.url}}], }] sampling_params = SamplingParams(temperature=0.01, max_tokens=256) outputs = llm.chat(conversation, sampling_params=sampling_params) print(outputs[0].outputs[0].text) ``` ## Troubleshooting - Make sure `vllm[audio]` extras are installed or audio requests will fail. - Use the nightly wheel until Qwen3-ASR support lands in the stable release. ## References - [Model card](https://huggingface.co/Qwen/Qwen3-ASR-1.7B) - [vLLM audio support](https://docs.vllm.ai/en/latest/features/multimodal_inputs.html) --- # Qwen/Qwen3-Coder-480B-A35B-Instruct on vLLM Source: https://recipes.vllm.ai/Qwen/Qwen3-Coder-480B-A35B-Instruct Provider: Qwen Released: 2025-07-22T14:52:38.000Z Updated: 2026-04-17 Architecture: moe Parameters: 480B Context length: 262,144 Min vLLM: 0.10.0 Tasks: text Verified hardware: H200, MI300X, MI325X, MI355X, TPU v7 (Ironwood) Summary: Large coder MoE with 480B total / 35B active parameters, strong tool-use and code generation capabilities. ## Variants - default: precision=bf16, vram_minimum_gb=1152 Full BF16 — 8x H200/H20 (141GB × 8) recommended - fp8: precision=fp8, vram_minimum_gb=576, model_id=Qwen/Qwen3-Coder-480B-A35B-Instruct-FP8 Qwen official FP8 checkpoint — required for DP=8 serving - nvfp4: precision=nvfp4, vram_minimum_gb=288, model_id=nvidia/Qwen3-Coder-480B-A35B-Instruct-NVFP4 NVIDIA NVFP4 quantized weights for Blackwell GPUs - redhat_fp8: precision=fp8, vram_minimum_gb=576, model_id=RedHatAI/Qwen3-Coder-480B-A35B-Instruct-FP8 RedHatAI FP8 quantization ## Guide ## Overview [Qwen3-Coder](https://github.com/QwenLM/Qwen3-Coder) is an advanced large language model created by the Qwen team. `Qwen3-Coder-480B-A35B-Instruct` is the flagship coder MoE with 480B total / 35B active parameters. vLLM supports it including tool calling; the guide below covers BF16 and FP8 serving on NVIDIA and AMD GPUs. TPU support is provided through [vLLM TPU](https://github.com/vllm-project/tpu-inference) with a recipe for [Ironwood](https://github.com/AI-Hypercomputer/tpu-recipes/tree/main/inference/ironwood/vLLM/Qwen3-Coder-480B-A35B). The Ironwood docker command rendered by the hardware picker uses the `vllm/vllm-tpu` image; pin to the tag specified by the upstream recipe. ## Prerequisites ### CUDA ```bash uv venv source .venv/bin/activate uv pip install -U vllm --torch-backend auto ``` ### ROCm (MI300X, MI325X, MI355X) ```bash uv venv source .venv/bin/activate uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/ ``` The ROCm wheel requires Python 3.12, ROCm 7.0, and glibc >= 2.35. ## Deployment Configurations ### 8xH200 / 8xH20 BF16 ```bash vllm serve Qwen/Qwen3-Coder-480B-A35B-Instruct \ --max-model-len 32000 \ --enable-expert-parallel \ --tensor-parallel-size 8 \ --enable-auto-tool-choice \ --tool-call-parser qwen3_coder ``` ### 8xH200 / 8xH20 FP8 (DP=8, recommended) ```bash VLLM_USE_DEEP_GEMM=1 vllm serve Qwen/Qwen3-Coder-480B-A35B-Instruct-FP8 \ --max-model-len 131072 \ --enable-expert-parallel \ --data-parallel-size 8 \ --enable-auto-tool-choice \ --tool-call-parser qwen3_coder ``` ### 8xMI300X/MI325X/MI355X BF16 ```bash VLLM_ROCM_USE_AITER=1 vllm serve Qwen/Qwen3-Coder-480B-A35B-Instruct \ --max-model-len 32000 \ --attention-backend rocm_aiter_fa \ --moe-backend aiter \ --enable-expert-parallel \ --tensor-parallel-size 8 \ --enable-auto-tool-choice \ --tool-call-parser qwen3_coder ``` ### 8xMI300X/MI325X/MI355X FP8 ```bash VLLM_ROCM_USE_AITER=1 vllm serve Qwen/Qwen3-Coder-480B-A35B-Instruct-FP8 \ --trust-remote-code \ --max-model-len 131072 \ --attention-backend rocm_aiter_fa \ --moe-backend aiter \ --enable-expert-parallel \ --data-parallel-size 8 \ --enable-auto-tool-choice \ --tool-call-parser qwen3_coder ``` ## Client Usage ```bash vllm bench serve \ --backend vllm \ --model Qwen/Qwen3-Coder-480B-A35B-Instruct-FP8 \ --endpoint /v1/completions \ --dataset-name random \ --random-input 2048 \ --random-output 1024 \ --max-concurrency 10 \ --num-prompt 100 ``` ## Troubleshooting - **Context-length OOM**: A single H20 node cannot serve the native 262144 context. Reduce `--max-model-len` or raise `--gpu-memory-utilization`. - **TP=8 failure on FP8**: Expect `ValueError: The output_size of gate's and up's weight = 320 is not divisible by weight quantization block_n = 128.` on FP8 with TP=8. Switch to `--data-parallel-size 8` instead. - **DeepGEMM**: set `VLLM_USE_DEEP_GEMM=1` for faster FP8 matmul. Follow the [setup instructions](https://github.com/vllm-project/vllm/blob/main/benchmarks/kernels/deepgemm/README.md#setup) to install it. - **Tool calling**: add `--tool-call-parser qwen3_coder` as shown above. ## Evaluation | Dataset | Test Type | Pass@1 Score | |------------|--------------------|--------------| | HumanEval | Base tests | 0.939 | | HumanEval+ | Base + extra tests | 0.902 | | MBPP | Base tests | 0.918 | | MBPP+ | Base + extra tests | 0.794 | ## References - [Model card](https://huggingface.co/Qwen/Qwen3-Coder-480B-A35B-Instruct) - [FP8 checkpoint](https://huggingface.co/Qwen/Qwen3-Coder-480B-A35B-Instruct-FP8) - [Qwen3-Coder GitHub](https://github.com/QwenLM/Qwen3-Coder) - [EvalPlus](https://github.com/evalplus/evalplus) - [TPU recipes: Ironwood](https://github.com/AI-Hypercomputer/tpu-recipes/tree/main/inference/ironwood/vLLM/Qwen3-Coder-480B-A35B) --- # Qwen/Qwen3-VL-235B-A22B-Instruct on vLLM Source: https://recipes.vllm.ai/Qwen/Qwen3-VL-235B-A22B-Instruct Provider: Qwen Released: 2025-09-22T03:54:32.000Z Updated: 2026-04-17 Architecture: moe Parameters: 235B Context length: 262,144 Min vLLM: 0.11.0 Tasks: multimodal, text Verified hardware: H100, H200, B200, MI300X, MI325X, MI355X Summary: Qwen3-VL flagship MoE vision-language model with 235B total / 22B active parameters, supporting images, video, and long context. ## Variants - default: precision=bf16, vram_minimum_gb=564 Full BF16 — ideal on H200/B200 with 8 GPUs - fp8: precision=fp8, vram_minimum_gb=282, model_id=Qwen/Qwen3-VL-235B-A22B-Instruct-FP8 Qwen official FP8 checkpoint for optimal H100 memory efficiency - nvfp4: precision=nvfp4, vram_minimum_gb=141, model_id=nvidia/Qwen3-VL-235B-A22B-Instruct-NVFP4 NVIDIA NVFP4 quantized weights for Blackwell GPUs - redhat_fp8: precision=fp8, vram_minimum_gb=282, model_id=RedHatAI/Qwen3-VL-235B-A22B-Instruct-FP8-dynamic RedHatAI FP8 (E4M3) with dynamic per-token activation quantization - redhat_fp8_block: precision=fp8, vram_minimum_gb=282, model_id=RedHatAI/Qwen3-VL-235B-A22B-Instruct-FP8-block RedHatAI FP8 block quantization - redhat_nvfp4: precision=nvfp4, vram_minimum_gb=150, model_id=RedHatAI/Qwen3-VL-235B-A22B-Instruct-NVFP4 RedHatAI NVFP4 quantization ## Guide ## Overview [Qwen3-VL](https://github.com/QwenLM/Qwen3-VL) is the most powerful vision-language model in the Qwen series, delivering upgrades to text understanding & generation, visual perception & reasoning, extended context, spatial/video dynamics, and agent interaction. The flagship `Qwen3-VL-235B-A22B-Instruct` is a MoE model that requires at least 8 GPUs with ≥80 GB memory each (A100/H100/H200 class). ## Prerequisites ```bash uv venv source .venv/bin/activate # Install vLLM >= 0.11.0 uv pip install -U vllm # Install Qwen-VL utility library (recommended for offline inference) uv pip install qwen-vl-utils==0.0.14 ``` ## Deployment Configurations ### H100 (Image + Video, FP8) ```bash vllm serve Qwen/Qwen3-VL-235B-A22B-Instruct-FP8 \ --tensor-parallel-size 8 \ --mm-encoder-tp-mode data \ --enable-expert-parallel \ --async-scheduling ``` ### H100 (Image-Only, FP8, TP4) ```bash vllm serve Qwen/Qwen3-VL-235B-A22B-Instruct-FP8 \ --tensor-parallel-size 4 \ --limit-mm-per-prompt.video 0 \ --async-scheduling \ --gpu-memory-utilization 0.95 \ --max-num-seqs 128 ``` ### A100 & H100 (Image-Only, BF16) ```bash vllm serve Qwen/Qwen3-VL-235B-A22B-Instruct \ --tensor-parallel-size 8 \ --limit-mm-per-prompt.video 0 \ --async-scheduling ``` ### A100 & H100 (Image + Video, BF16) ```bash vllm serve Qwen/Qwen3-VL-235B-A22B-Instruct \ --tensor-parallel-size 8 \ --max-model-len 128000 \ --async-scheduling ``` ### H200 & B200 ```bash vllm serve Qwen/Qwen3-VL-235B-A22B-Instruct \ --tensor-parallel-size 8 \ --mm-encoder-tp-mode data \ --async-scheduling ``` ### MI300X/MI325X/MI355X (BF16, TP8) ```bash export SAFETENSORS_FAST_GPU="1" export HIP_FORCE_DEV_KERNARG="1" export HIP_VISIBLE_DEVICES="0,1,2,3,4,5,6,7" export VLLM_WORKER_MULTIPROC_METHOD="spawn" export VLLM_ROCM_USE_AITER="1" export VLLM_ROCM_USE_AITER_MHA="1" export VLLM_ROCM_SHUFFLE_KV_CACHE_LAYOUT="1" vllm serve Qwen/Qwen3-VL-235B-A22B-Instruct \ --tensor-parallel-size 8 \ --mm-encoder-tp-mode data \ --enable-expert-parallel \ --async-scheduling \ --gpu-memory-utilization 0.94 \ --max-model-len 32768 \ --max-num-seqs 10240 \ --max-num-batched-tokens 32768 \ --attention-backend ROCM_AITER_FA ``` ### MI300X/MI325X/MI355X (FP8, TP8) ```bash export SAFETENSORS_FAST_GPU="1" export HIP_FORCE_DEV_KERNARG="1" export HIP_VISIBLE_DEVICES="0,1,2,3,4,5,6,7" export VLLM_WORKER_MULTIPROC_METHOD="spawn" export VLLM_ROCM_USE_AITER="1" export VLLM_ROCM_USE_AITER_MHA="1" export VLLM_ROCM_SHUFFLE_KV_CACHE_LAYOUT="1" vllm serve Qwen/Qwen3-VL-235B-A22B-Instruct-FP8 \ --tensor-parallel-size 8 \ --mm-encoder-tp-mode data \ --enable-expert-parallel \ --async-scheduling \ --enable-chunked-prefill \ --limit-mm-per-prompt '{"image": 1}' \ --gpu-memory-utilization 0.94 \ --max-model-len 32768 \ --max-num-seqs 10240 \ --max-num-batched-tokens 32768 \ --kv-cache-dtype fp8 \ --compilation-config '{"mode": 3, "cudagraph_mode": "FULL_AND_PIECEWISE", "custom_ops": ["+rms_norm", "+quant_fp8"]}' \ --attention-backend ROCM_AITER_FA ``` > The above configuration runs efficiently at high concurrencies (up to 512) for context lengths up to 8k. If TPOT is not satisfactory, lower `--max-num-batched-tokens`. For longer contexts, `--max-num-seqs` and `--max-num-batched-tokens` may need retuning. ### MI300X / MI355X (FP8, EAGLE3 Speculative Decoding) Adds EAGLE3 speculative decoding (`RedHatAI/Qwen3-VL-235B-A22B-Instruct-speculator.eagle3`) on top of the FP8 target. **`--attention-backend ROCM_AITER_FA` is mandatory** — the default `ROCM_ATTN` path makes SD slower than baseline on ROCm. Keep `VLLM_ROCM_SHUFFLE_KV_CACHE_LAYOUT=0` and omit `--async-scheduling` (incompatible with SD). ```bash export SAFETENSORS_FAST_GPU="1" export HIP_FORCE_DEV_KERNARG="1" export HIP_VISIBLE_DEVICES="0,1,2,3,4,5,6,7" export VLLM_WORKER_MULTIPROC_METHOD="spawn" export VLLM_ROCM_USE_AITER="1" export VLLM_ROCM_USE_AITER_MHA="1" export VLLM_ROCM_USE_AITER_RMSNORM="0" export VLLM_ROCM_SHUFFLE_KV_CACHE_LAYOUT="0" vllm serve Qwen/Qwen3-VL-235B-A22B-Instruct-FP8 \ --tensor-parallel-size 8 \ --gpu-memory-utilization 0.94 \ --distributed-executor-backend mp \ --enable-chunked-prefill \ --max-model-len 16384 \ --max-num-seqs 32 \ --max-num-batched-tokens 8192 \ --mm-encoder-tp-mode data \ --enable-expert-parallel \ --attention-backend ROCM_AITER_FA \ --compilation-config '{"mode": 3, "cudagraph_mode": "FULL_AND_PIECEWISE", "custom_ops": ["-rms_norm"], "pass_config": {"fuse_norm_quant":false}}' \ --speculative-config '{"model": "RedHatAI/Qwen3-VL-235B-A22B-Instruct-speculator.eagle3", "method": "eagle3", "num_speculative_tokens": 4}' ``` ## Configuration Tips - Use `--limit-mm-per-prompt.video 0` if your server only serves image inputs to save memory. - `OMP_NUM_THREADS=1` reduces CPU contention during preprocessing. - The model's context length is 262K. Reduce `--max-model-len` (e.g. 128000) if you don't need the full range. - `--async-scheduling` overlaps scheduling with decoding for better throughput. - `--mm-encoder-tp-mode data` deploys the vision encoder in data-parallel fashion for better performance. - If your inputs are mostly unique, pass `--mm-processor-cache-gb 0` to skip caching overhead. - Extend context to 1M tokens with the **Long Context** toggle, which applies YaRN via `--hf-overrides` and raises `--max-model-len`. Static YaRN costs quality on short prompts. Text-only mode: pass `--limit-mm-per-prompt.video 0 --limit-mm-per-prompt.image 0` to free memory for KV cache when serving text-only traffic. ## Client Usage ```python import time from openai import OpenAI client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1", timeout=3600) messages = [{ "role": "user", "content": [ {"type": "image_url", "image_url": {"url": "https://ofasys-multimodal-wlcb-3-toshanghai.oss-accelerate.aliyuncs.com/wpf272043/keepme/image/receipt.png"}}, {"type": "text", "text": "Read all the text in the image."}, ], }] start = time.time() response = client.chat.completions.create( model="Qwen/Qwen3-VL-235B-A22B-Instruct", messages=messages, max_tokens=2048, ) print(f"Response costs: {time.time() - start:.2f}s") print(response.choices[0].message.content) ``` ## Troubleshooting - OOM on A100 / H100 BF16: reduce `--max-model-len`, drop to image-only, or switch to the FP8 checkpoint. - If enabling `--mm-encoder-tp-mode data` raises memory pressure, lower `--gpu-memory-utilization`. ## References - [Model card](https://huggingface.co/Qwen/Qwen3-VL-235B-A22B-Instruct) - [FP8 checkpoint](https://huggingface.co/Qwen/Qwen3-VL-235B-A22B-Instruct-FP8) - [Qwen3-VL GitHub](https://github.com/QwenLM/Qwen3-VL) - [vLLM multimodal inputs guide](https://docs.vllm.ai/en/latest/features/multimodal_inputs.html) --- # arcee-ai/Trinity-Large-Thinking on vLLM Source: https://recipes.vllm.ai/arcee-ai/Trinity-Large-Thinking Provider: Arcee AI Released: 2026-04-01T03:35:06.000Z Updated: 2026-04-17 Architecture: moe Parameters: 398B Context length: 262,144 Min vLLM: 0.11.1 Tasks: text Summary: Arcee AI's reasoning-focused sparse MoE (AfmoeForCausalLM) with structured traces and agentic tool use ## Variants - default: precision=bf16, vram_minimum_gb=955 Full precision BF16 on multi-GPU (sparse MoE — multi-GPU recommended) - nvfp4: precision=nvfp4, vram_minimum_gb=239, model_id=arcee-ai/Trinity-Large-Thinking-NVFP4 NVIDIA NVFP4 quantized weights for Blackwell GPUs - redhat_nvfp4: precision=nvfp4, vram_minimum_gb=254, model_id=RedHatAI/Trinity-Large-Thinking-NVFP4 RedHatAI NVFP4 quantization ## Recommended base flags ```bash vllm serve 'arcee-ai/Trinity-Large-Thinking' \ --dtype \ bfloat16 ``` ## Guide ## Overview [Trinity-Large-Thinking](https://huggingface.co/arcee-ai/Trinity-Large-Thinking) is Arcee AI's reasoning-focused Trinity Large checkpoint — a sparse MoE model designed for long-horizon planning, tool use, and multi-step agent workflows. It uses the `AfmoeForCausalLM` architecture and emits explicit reasoning traces inside `...` blocks. For multi-turn chat and agentic loops, reasoning tokens should be preserved across turns as part of the working state. ## Prerequisites - vLLM >= 0.11.1 - Hardware: multi-GPU recommended for production deployments ### Install vLLM ```bash uv venv source .venv/bin/activate uv pip install -U vllm openai --torch-backend auto ``` ## Launch command ```bash vllm serve arcee-ai/Trinity-Large-Thinking \ --dtype bfloat16 \ --reasoning-parser deepseek_r1 \ --enable-auto-tool-choice \ --tool-call-parser qwen3_coder ``` Why these flags: - `--reasoning-parser deepseek_r1` extracts `...` into `message.reasoning`. - `--enable-auto-tool-choice` lets the model decide when to call tools. - `--tool-call-parser qwen3_coder` converts tool calls into OpenAI-style `tool_calls`. - `--dtype bfloat16` matches the recommended serving dtype. Add parallelism flags (`--tensor-parallel-size`, `--data-parallel-size`, or `--enable-expert-parallel`) for your hardware. Lower `--max-model-len` if you don't need the full long-context config. ## Validation Request ```python from openai import OpenAI client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1") model = client.models.list().data[0].id tools = [{ "type": "function", "function": { "name": "get_weather", "description": "Get the current weather for a location.", "parameters": { "type": "object", "properties": {"location": {"type": "string"}}, "required": ["location"], }, }, }] response = client.chat.completions.create( model=model, messages=[{"role": "user", "content": "What is the weather in Paris right now?"}], tools=tools, tool_choice="auto", ) msg = response.choices[0].message reasoning = msg.reasoning print("reasoning:", reasoning) print("content:", msg.content) print("tool_calls:", msg.tool_calls) ``` ## Preserving Reasoning Across Turns Pass reasoning back as `reasoning` on assistant messages: ```python assistant_msg = {"role": "assistant", "content": msg.content or ""} if reasoning: assistant_msg["reasoning"] = reasoning if msg.tool_calls: assistant_msg["tool_calls"] = [ {"id": tc.id, "type": "function", "function": {"name": tc.function.name, "arguments": tc.function.arguments}} for tc in msg.tool_calls ] messages.append(assistant_msg) ``` Rules: - Pass reasoning back as `reasoning`. - Keep `content` as an empty string (not `null`) on tool-only turns. - Append the assistant message before tool-result messages. - Use `/v1/chat/completions` for structured reasoning output. ## Troubleshooting - **No reasoning:** start server with `--reasoning-parser deepseek_r1`; use `/v1/chat/completions`. - **Tool calls as plain text:** enable `--enable-auto-tool-choice` and `--tool-call-parser qwen3_coder`. - **Loses coherence after tool turns:** preserve `reasoning` on each assistant turn; don't set content to `null`. - **OOM:** lower `--max-model-len`; scale parallelism; use a local checkpoint path. ## References - [Trinity-Large-Thinking on Hugging Face](https://huggingface.co/arcee-ai/Trinity-Large-Thinking) --- # baidu/ERNIE-4.5-21B-A3B-PT on vLLM Source: https://recipes.vllm.ai/baidu/ERNIE-4.5-21B-A3B-PT Provider: Ernie (Baidu) Released: 2025-06-28T06:13:30.000Z Updated: 2026-04-17 Architecture: moe Parameters: 21B Context length: 131,072 Min vLLM: 0.10.1 Tasks: text Summary: Baidu ERNIE 4.5 MoE text models (21B-A3B, 300B-A47B) with BF16 and FP8 support plus ERNIE-MTP speculative decoding ## Variants - default: precision=bf16, vram_minimum_gb=106 BF16 weights; fits on 1x80GB GPU (21B variant) - 300b: precision=bf16, vram_minimum_gb=640, model_id=baidu/ERNIE-4.5-300B-A47B-PT 300B total / 47B active; 8x80GB with FP8 online, 16x80GB for BF16 ## Guide ## Overview ERNIE 4.5 is Baidu's MoE language model family. This recipe covers the text-only variants: - `baidu/ERNIE-4.5-21B-A3B-PT` — 21B total / 3B active (fits on 1x80GB) - `baidu/ERNIE-4.5-300B-A47B-PT` — 300B total / 47B active (8x80GB FP8 or 16x80GB BF16) Both support ERNIE-MTP speculative decoding via `--speculative-config`. ## Prerequisites - transformers >= 4.54.0 - vLLM >= 0.10.1 - Hardware depends on variant (see above) ### Install vLLM ```bash uv venv --python 3.12 --seed source .venv/bin/activate uv pip install vllm --torch-backend=auto ``` ## Launch commands 21B on 1x80GB GPU: ```bash vllm serve baidu/ERNIE-4.5-21B-A3B-PT ``` 300B on 8x80GB with vLLM FP8 online quantization: ```bash vllm serve baidu/ERNIE-4.5-300B-A47B-PT \ --tensor-parallel-size 8 \ --gpu-memory-utilization 0.95 \ --quantization fp8 ``` 300B on 16x80GB native BF16 (multi-node via Ray): ```bash vllm serve baidu/ERNIE-4.5-300B-A47B-PT --tensor-parallel-size 16 ``` ERNIE-MTP speculative decoding (21B example): ```bash vllm serve baidu/ERNIE-4.5-21B-A3B-PT \ --speculative-config '{"method":"ernie_mtp","model":"baidu/ERNIE-4.5-21B-A3B-PT","num_speculative_tokens":1}' ``` ## Client Usage Standard OpenAI-compatible API; model ID is the HF repo. ## Benchmarking ```bash vllm bench serve \ --model baidu/ERNIE-4.5-21B-A3B-PT \ --dataset-name random \ --random-input-len 8000 --random-output-len 1000 \ --request-rate 10 --num-prompts 16 --ignore-eos ``` Test configurations: prompt-heavy (8k/1k), decode-heavy (1k/8k), balanced (1k/1k). Vary `--num-prompts` across 1, 16, 32, 64, 128, 256, 512. ## References - [ERNIE-4.5-21B-A3B-PT](https://huggingface.co/baidu/ERNIE-4.5-21B-A3B-PT) - [ERNIE-4.5-300B-A47B-PT](https://huggingface.co/baidu/ERNIE-4.5-300B-A47B-PT) - [vLLM multi-node deployment](https://docs.vllm.ai/en/latest/serving/parallelism_scaling.html) --- # baidu/ERNIE-4.5-VL-28B-A3B-PT on vLLM Source: https://recipes.vllm.ai/baidu/ERNIE-4.5-VL-28B-A3B-PT Provider: Ernie (Baidu) Released: 2025-06-28T05:50:33.000Z Updated: 2026-04-17 Architecture: moe Parameters: 28B Context length: 131,072 Min vLLM: 0.11.0 Tasks: multimodal Verified hardware: MI300X, MI325X, MI355X Summary: Baidu ERNIE 4.5 VL MoE vision-language models (28B-A3B, 424B-A47B) with heterogeneous text/vision experts ## Variants - default: precision=bf16, vram_minimum_gb=67 BF16 weights; fits on 1x80GB GPU (28B VL variant) - 424b: precision=bf16, vram_minimum_gb=1120, model_id=baidu/ERNIE-4.5-VL-424B-A47B-PT 424B total / 47B active; 8x140GB BF16 or 16x80GB BF16 - 424b_fp8: precision=fp8, vram_minimum_gb=640, model_id=baidu/ERNIE-4.5-VL-424B-A47B-PT 424B with FP8 online quantization + CPU offload for 8x80GB testing ## Recommended base flags ```bash vllm serve 'baidu/ERNIE-4.5-VL-28B-A3B-PT' \ --trust-remote-code ``` ## Guide ## Overview ERNIE 4.5 VL is Baidu's multimodal MoE model with heterogeneous experts (separate text and vision experts). Because of the heterogeneous architecture, **torch.compile and CUDA graphs are not supported**. - `baidu/ERNIE-4.5-VL-28B-A3B-PT` — 28B total / 3B active (1x80GB) - `baidu/ERNIE-4.5-VL-424B-A47B-PT` — 424B total / 47B active (8x140GB BF16, 8x80GB FP8+offload, or 16x80GB BF16) ## Prerequisites - vLLM: support added to main branch recently; install latest - Hardware depends on variant ### Install vLLM (CUDA) ```bash uv venv --python 3.12 --seed source .venv/bin/activate uv pip install -U vllm --torch-backend auto ``` ### Install vLLM (AMD ROCm MI300X/MI325X/MI355X) ```bash uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/ ``` ## Launch commands 28B on 1x80GB: ```bash vllm serve baidu/ERNIE-4.5-VL-28B-A3B-PT --trust-remote-code ``` 424B BF16 on 8x140GB: ```bash vllm serve baidu/ERNIE-4.5-VL-424B-A47B-PT \ --trust-remote-code \ --tensor-parallel-size 8 ``` 424B with FP8 + CPU offload on 8x80GB (testing only): ```bash vllm serve baidu/ERNIE-4.5-VL-424B-A47B-PT \ --trust-remote-code \ --tensor-parallel-size 8 \ --quantization fp8 \ --cpu-offload-gb 50 ``` 28B on AMD MI300X+: ```bash VLLM_ROCM_USE_AITER=1 SAFETENSORS_FAST_GPU=1 \ vllm serve baidu/ERNIE-4.5-VL-28B-A3B-PT \ --tensor-parallel-size 4 \ --gpu-memory-utilization 0.9 \ --trust-remote-code ``` ## Benchmarking ```bash vllm bench serve \ --model baidu/ERNIE-4.5-VL-28B-A3B-PT \ --dataset-name random \ --random-input-len 8000 --random-output-len 1000 \ --request-rate 10 --num-prompts 16 --ignore-eos --trust-remote-code ``` ## References - [ERNIE-4.5-VL-28B-A3B-PT](https://huggingface.co/baidu/ERNIE-4.5-VL-28B-A3B-PT) - [ERNIE-4.5-VL-424B-A47B-PT](https://huggingface.co/baidu/ERNIE-4.5-VL-424B-A47B-PT) --- # deepseek-ai/DeepSeek-OCR-2 on vLLM Source: https://recipes.vllm.ai/deepseek-ai/DeepSeek-OCR-2 Provider: DeepSeek Released: 2026-01-27T02:56:54.000Z Updated: 2026-04-17 Architecture: dense Parameters: 3B Context length: 8,192 Min vLLM: 0.16.0 Tasks: multimodal Summary: Next-generation DeepSeek OCR model with improved document-to-markdown grounding and optical context compression. ## Variants - default: precision=bf16, vram_minimum_gb=7 Full precision BF16 (~3.4B params) ## Recommended base flags ```bash vllm serve 'deepseek-ai/DeepSeek-OCR-2' \ --trust-remote-code \ --logits_processors \ vllm.model_executor.models.deepseek_ocr:NGramPerReqLogitsProcessor \ --no-enable-prefix-caching \ --mm-processor-cache-gb \ 0 ``` ## Guide ## Overview DeepSeek-OCR-2 is a frontier OCR model exploring optical context compression for LLMs. It iterates on DeepSeek-OCR with better grounding and markdown conversion, and supports prompts like `\n<|grounding|>Convert the document to markdown.` for richer document parsing. ## Prerequisites - **Hardware**: Single GPU with >=8 GB VRAM is typically sufficient for BF16 inference. - **vLLM**: Current stable release (tested with `uv pip install -U vllm --torch-backend auto`). - **Python**: 3.10+ Install vLLM: ```bash uv venv source .venv/bin/activate uv pip install -U vllm --torch-backend auto ``` ## Client Usage ### Offline OCR (Python) ```python from vllm import LLM, SamplingParams from vllm.model_executor.models.deepseek_ocr import NGramPerReqLogitsProcessor from PIL import Image llm = LLM( model="deepseek-ai/DeepSeek-OCR-2", enable_prefix_caching=False, mm_processor_cache_gb=0, logits_processors=[NGramPerReqLogitsProcessor], ) image_1 = Image.open("path/to/your/image_1.png").convert("RGB") image_2 = Image.open("path/to/your/image_2.png").convert("RGB") # prompt = "\nFree OCR. " prompt = "\n<|grounding|>Convert the document to markdown. " model_input = [ {"prompt": prompt, "multi_modal_data": {"image": image_1}}, {"prompt": prompt, "multi_modal_data": {"image": image_2}}, ] sampling_param = SamplingParams( temperature=0.0, max_tokens=8192, extra_args=dict( ngram_size=30, window_size=90, whitelist_token_ids={128821, 128822}, # , ), skip_special_tokens=False, ) for output in llm.generate(model_input, sampling_param): print(output.outputs[0].text) ``` ### Online OCR serving ```bash vllm serve deepseek-ai/DeepSeek-OCR-2 \ --logits_processors vllm.model_executor.models.deepseek_ocr:NGramPerReqLogitsProcessor \ --no-enable-prefix-caching \ --mm-processor-cache-gb 0 ``` ```python import time from openai import OpenAI client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1", timeout=3600) messages = [ { "role": "user", "content": [ {"type": "image_url", "image_url": {"url": "https://ofasys-multimodal-wlcb-3-toshanghai.oss-accelerate.aliyuncs.com/wpf272043/keepme/image/receipt.png"}}, {"type": "text", "text": "Free OCR."}, ], } ] start = time.time() response = client.chat.completions.create( model="deepseek-ai/DeepSeek-OCR-2", messages=messages, max_tokens=2048, temperature=0.0, extra_body={ "skip_special_tokens": False, "vllm_xargs": { "ngram_size": 30, "window_size": 90, "whitelist_token_ids": [128821, 128822], }, }, ) print(f"Response costs: {time.time() - start:.2f}s") print(f"Generated text: {response.choices[0].message.content}") ``` ## Troubleshooting / Configuration Tips - **Use the custom logits processor** along with the model for optimal OCR and markdown generation performance. - Unlike multi-turn chat, OCR tasks do not typically benefit from prefix caching or image reuse, so disable these features to avoid unnecessary hashing and caching overhead. - DeepSeek-OCR-2 works better with plain prompts than instruction formats. See the [official main prompts](https://huggingface.co/deepseek-ai/DeepSeek-OCR-2#main-prompts). - Depending on your hardware, adjust `max_num_batched_tokens` for better throughput. ## References - [DeepSeek-OCR-2 on Hugging Face](https://huggingface.co/deepseek-ai/DeepSeek-OCR-2) - [vLLM multimodal inputs guide](https://docs.vllm.ai/en/latest/features/multimodal_inputs.html#offline-inference) --- # deepseek-ai/DeepSeek-OCR on vLLM Source: https://recipes.vllm.ai/deepseek-ai/DeepSeek-OCR Provider: DeepSeek Released: 2025-10-17T06:22:05.000Z Updated: 2026-04-17 Architecture: dense Parameters: 3B Context length: 8,192 Min vLLM: 0.12.0 Tasks: multimodal Verified hardware: MI300X, MI325X, MI355X, Arc Pro B60, Arc Pro B70 Summary: Frontier OCR model exploring optical context compression for LLMs, optimized for document parsing and markdown generation. ## Variants - default: precision=bf16, vram_minimum_gb=7 Full precision BF16 (~3.3B params) ## Recommended base flags ```bash vllm serve 'deepseek-ai/DeepSeek-OCR' \ --trust-remote-code \ --logits-processors \ vllm.model_executor.models.deepseek_ocr:NGramPerReqLogitsProcessor \ --mm-processor-cache-gb \ 0 ``` ## Guide ## Overview DeepSeek-OCR is a frontier OCR model exploring optical context compression for LLMs. It is optimized for document parsing, free-form OCR, and markdown generation from images, and ships with a custom n-gram logits processor for optimal quality. ## Prerequisites - **Hardware**: Single GPU with >=8 GB VRAM (CUDA, MI300X / MI325X / MI355X, or Intel XPU B60 / B70) - **vLLM**: Current stable release - **Python**: 3.10+ (CUDA) or 3.12 (ROCm) ### CUDA ```bash uv venv source .venv/bin/activate uv pip install -U vllm --torch-backend auto ``` ### ROCm (MI300X, MI325X, MI355X) Requires Python 3.12, ROCm 7.2.1, and glibc >= 2.35. Use the Docker flow from the [vLLM installation guide](https://docs.vllm.ai/en/latest/getting_started/installation/gpu/#pre-built-images) if your host environment does not meet those requirements. ```bash uv venv --python 3.12 source .venv/bin/activate uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/ ``` ### Intel XPU (B60 / B70) ```bash docker pull vllm/vllm-openai-xpu:latest ``` ## Client Usage ### Offline OCR (Python) ```python from vllm import LLM, SamplingParams from vllm.model_executor.models.deepseek_ocr import NGramPerReqLogitsProcessor from PIL import Image llm = LLM( model="deepseek-ai/DeepSeek-OCR", enable_prefix_caching=False, mm_processor_cache_gb=0, logits_processors=[NGramPerReqLogitsProcessor], ) image_1 = Image.open("path/to/your/image_1.png").convert("RGB") image_2 = Image.open("path/to/your/image_2.png").convert("RGB") prompt = "\nFree OCR." model_input = [ {"prompt": prompt, "multi_modal_data": {"image": image_1}}, {"prompt": prompt, "multi_modal_data": {"image": image_2}}, ] sampling_param = SamplingParams( temperature=0.0, max_tokens=8192, extra_args=dict( ngram_size=30, window_size=90, whitelist_token_ids={128821, 128822}, # , ), skip_special_tokens=False, ) for output in llm.generate(model_input, sampling_param): print(output.outputs[0].text) ``` ### Online OCR serving **CUDA** ```bash vllm serve deepseek-ai/DeepSeek-OCR \ --logits-processors vllm.model_executor.models.deepseek_ocr:NGramPerReqLogitsProcessor \ --mm-processor-cache-gb 0 ``` **ROCm (MI300X / MI325X / MI355X)** ```bash export SAFETENSORS_FAST_GPU=1 export VLLM_ROCM_USE_AITER=1 vllm serve deepseek-ai/DeepSeek-OCR \ --logits-processors vllm.model_executor.models.deepseek_ocr:NGramPerReqLogitsProcessor \ --mm-processor-cache-gb 0 ``` **Intel XPU (B60, B70)** ```bash docker run --device /dev/dri \ -v /dev/dri/by-path:/dev/dri/by-path --shm-size=16g \ --privileged --ipc=host -p 8000:8000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ vllm/vllm-openai-xpu:latest \ deepseek-ai/DeepSeek-OCR \ --logits-processors vllm.model_executor.models.deepseek_ocr:NGramPerReqLogitsProcessor \ --mm-processor-cache-gb 0 ``` Query the running server with the OpenAI client (same for CUDA / ROCm / XPU): ```python import time from openai import OpenAI client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1", timeout=3600) messages = [ { "role": "user", "content": [ {"type": "image_url", "image_url": {"url": "https://ofasys-multimodal-wlcb-3-toshanghai.oss-accelerate.aliyuncs.com/wpf272043/keepme/image/receipt.png"}}, {"type": "text", "text": "Free OCR."}, ], } ] start = time.time() response = client.chat.completions.create( model="deepseek-ai/DeepSeek-OCR", messages=messages, max_tokens=2048, temperature=0.0, extra_body={ "skip_special_tokens": False, "vllm_xargs": { "ngram_size": 30, "window_size": 90, "whitelist_token_ids": [128821, 128822], }, }, ) print(f"Response costs: {time.time() - start:.2f}s") print(f"Generated text: {response.choices[0].message.content}") ``` ## Troubleshooting / Configuration Tips - **Use the custom logits processor** along with the model for optimal OCR and markdown generation performance. - Unlike multi-turn chat, OCR tasks do not typically benefit from prefix caching or image reuse, so disable these features to avoid unnecessary hashing and caching overhead. - DeepSeek-OCR works better with plain prompts than instruction formats. See the [official example prompts](https://github.com/deepseek-ai/DeepSeek-OCR/blob/2ac6d64a00656693b79c4f759a5e62c1b78bbeb1/DeepSeek-OCR-master/DeepSeek-OCR-vllm/config.py#L27-L37). - Depending on your hardware, adjust `max_num_batched_tokens` for better throughput. ## References - [DeepSeek-OCR on Hugging Face](https://huggingface.co/deepseek-ai/DeepSeek-OCR) - [vLLM multimodal inputs guide](https://docs.vllm.ai/en/latest/features/multimodal_inputs.html#offline-inference) --- # inclusionAI/Ring-1T-FP8 on vLLM Source: https://recipes.vllm.ai/inclusionAI/Ring-1T-FP8 Provider: inclusionAI Updated: 2026-04-17 Architecture: moe Parameters: 1T Context length: 65,536 Min vLLM: 0.11.0 Tasks: text Verified hardware: H200, MI300X, MI325X, MI355X Summary: Ring-1T (BailingMoeV2) FP8 model (~1T total params) for 8xH200 or 8xMI300X deployment ## Variants - default: precision=fp8, vram_minimum_gb=1200 FP8 weights on 8x H200 (80 GB) with FP8 KV cache ## Recommended base flags ```bash vllm serve 'inclusionAI/Ring-1T-FP8' \ --trust-remote-code \ --tensor-parallel-size \ 8 \ --max-num-seqs \ 32 \ --kv-cache-dtype \ fp8 \ --served-model-name \ Ring-1T-FP8 ``` ## Guide ## Overview Ring-1T-FP8 is inclusionAI's BailingMoeV2 FP8 model (~1T total parameters). This recipe covers pure tensor-parallel deployment across 8 GPUs on NVIDIA H200 or AMD MI300X+. ## Prerequisites - Hardware: 8x H200 or 8x MI300X/MI325X/MI355X - vLLM >= 0.11.0 ### Install vLLM (CUDA) ```bash uv venv source .venv/bin/activate uv pip install -U vllm --torch-backend auto ``` ### Install vLLM (AMD ROCm) ```bash uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/0.14.1/rocm700 ``` ## Launch commands 8x H200 (FP8 KV cache): ```bash vllm serve inclusionAI/Ring-1T-FP8 \ --trust-remote-code \ --tensor-parallel-size 8 \ --gpu-memory-utilization 0.97 \ --max-num-seqs 32 \ --kv-cache-dtype fp8 \ --compilation-config '{"use_inductor": false}' \ --served-model-name Ring-1T-FP8 ``` 8x MI300X/MI325X/MI355X: ```bash export VLLM_ROCM_USE_AITER=1 vllm serve inclusionAI/Ring-1T-FP8 \ --trust-remote-code \ --tensor-parallel-size 8 \ --gpu-memory-utilization 0.9 \ --max-num-seqs 32 \ --kv-cache-dtype fp8 \ --served-model-name Ring-1T-FP8 ``` Tuning flags: - `--max-model-len=65536` works well for most scenarios. - `--max-num-batched-tokens=32768` for prompt-heavy; 16384/8192 for lower latency. - Reduce `--gpu-memory-utilization` below 0.97 if you hit OOM. ## Client Usage ```bash curl http://localhost:8000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "Ring-1T-FP8", "messages": [{"role": "user", "content": "9.11 and 9.8, which is greater?"}] }' ``` ## References - [Ring-1T-FP8 on Hugging Face](https://huggingface.co/inclusionAI/Ring-1T-FP8) --- # internlm/Intern-S1 on vLLM Source: https://recipes.vllm.ai/internlm/Intern-S1 Provider: InternLM Released: 2025-07-24T06:05:13.000Z Updated: 2026-04-17 Architecture: moe Parameters: 241B Context length: 65,536 Min vLLM: 0.10.2 Tasks: multimodal Verified hardware: MI300X, MI325X, MI355X Summary: Intern-S1 vision-language model from Shanghai AI Lab with BF16/FP8 variants and thinking/non-thinking modes ## Variants - default: precision=bf16, vram_minimum_gb=578 BF16 on 8x H800 (80GB each) - fp8: precision=fp8, vram_minimum_gb=289, model_id=internlm/Intern-S1-FP8 FP8 on 4x H800 (80GB each) ## Recommended base flags ```bash vllm serve 'internlm/Intern-S1' \ --trust-remote-code ``` ## Guide ## Overview [Intern-S1](https://github.com/InternLM/Intern-S1) is a vision-language model developed by Shanghai AI Laboratory. It supports thinking and non-thinking modes via chat-template kwargs and ships in BF16 and FP8 variants. ## Prerequisites - Hardware: 8xH800 (80GB) for BF16, 4xH800 for FP8, or 4-8x MI300X/MI325X/MI355X - vLLM >= 0.10.2 ### Install vLLM (CUDA) ```bash uv venv source .venv/bin/activate uv pip install -U vllm --torch-backend auto ``` ### Install vLLM (AMD ROCm) ```bash uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/0.14.1/rocm700 ``` ## Launch commands BF16 on 8xH800: ```bash vllm serve internlm/Intern-S1 \ --trust-remote-code \ --tensor-parallel-size 8 \ --enable-auto-tool-choice \ --reasoning-parser deepseek_r1 \ --tool-call-parser internlm ``` FP8 on 4xH800: ```bash vllm serve internlm/Intern-S1-FP8 \ --trust-remote-code \ --tensor-parallel-size 4 \ --enable-auto-tool-choice \ --reasoning-parser deepseek_r1 \ --tool-call-parser internlm ``` FP8 on 8xMI300X/MI325X: ```bash export VLLM_ROCM_USE_AITER=1 export VLLM_ROCM_USE_AITER_MOE=0 vllm serve internlm/Intern-S1-FP8 \ --trust-remote-code --tensor-parallel-size 8 \ --enable-auto-tool-choice --reasoning-parser deepseek_r1 --tool-call-parser internlm ``` FP8 on 8xMI355X: set only `VLLM_ROCM_USE_AITER=1` (no need to disable AITER MoE). ## Switching Between Thinking and Non-Thinking Modes ```python from openai import OpenAI client = OpenAI(api_key="YOUR_API_KEY", base_url="http://0.0.0.0:8000/v1") model_name = client.models.list().data[0].id response = client.chat.completions.create( model=model_name, messages=[{"role": "user", "content": "9.11 and 9.8, which is greater?"}], temperature=0.8, top_p=0.8, extra_body={"chat_template_kwargs": {"enable_thinking": False}}, ) print(response) ``` ## Troubleshooting - `ValueError: No available memory for the cache blocks.` — add `--gpu-memory-utilization 0.95`. ## References - [Intern-S1 on Hugging Face](https://huggingface.co/internlm/Intern-S1) - [Intern-S1-FP8](https://huggingface.co/internlm/Intern-S1-FP8) - [Intern-S1 GitHub](https://github.com/InternLM/Intern-S1) --- # jinaai/jina-reranker-m0 on vLLM Source: https://recipes.vllm.ai/jinaai/jina-reranker-m0 Provider: Jina AI Released: 2025-03-27T11:08:51.000Z Updated: 2026-04-17 Architecture: dense Parameters: 2.4B Context length: 32,768 Min vLLM: 0.10.0 Tasks: embedding Summary: Multilingual, multimodal reranker for text and visual documents across 29+ languages via Qwen2-VL backbone ## Variants - default: precision=bf16, vram_minimum_gb=6 BF16 weights; 2x T4 or 2x L4 GPUs ## Recommended base flags ```bash vllm serve 'jinaai/jina-reranker-m0' \ --gpu-memory-utilization \ 0.75 \ --max-num-seqs \ 32 ``` ## Guide ## Overview [jinaai/jina-reranker-m0](https://huggingface.co/jinaai/jina-reranker-m0) is a multilingual, multimodal reranker that ranks visual documents across 29+ languages. It accepts text and visual content, including pages with mixed text, figures, tables, and various layouts. Deployment target: 2x NVIDIA T4 or 2x NVIDIA L4. ## Prerequisites - Hardware: 2x T4 or 2x L4 (or any 2x GPU with ~16 GB each) - vLLM >= 0.10.0 ### Install vLLM (CUDA) ```bash uv pip install vllm ``` ### Install vLLM (AMD ROCm) ```bash uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/0.14.1/rocm700 ``` ## Launch command ```bash vllm serve jinaai/jina-reranker-m0 \ --host 0.0.0.0 \ --port 8000 \ --tensor_parallel_size 2 \ --gpu-memory-utilization 0.75 \ --max-num-seqs 32 ``` On AMD: ```bash export VLLM_ROCM_USE_AITER=1 vllm serve jinaai/jina-reranker-m0 \ --tensor_parallel_size 2 --gpu-memory-utilization 0.75 --max-num-seqs 32 ``` ## Rerank API ```bash curl -X POST http://localhost:8000/v1/rerank \ -H "Content-Type: application/json" \ -d '{ "model": "jinaai/jina-reranker-m0", "query": "What are the health benefits of green tea?", "documents": [ "Green tea contains antioxidants called catechins...", "El precio del café ha aumentado un 20% este año...", "Studies show that drinking green tea regularly..." ], "top_n": 3, "return_documents": true }' ``` ## Score API Text-to-text: ```bash curl -X POST http://localhost:8000/v1/score \ -H "Content-Type: application/json" \ -d '{ "model": "jinaai/jina-reranker-m0", "text_1": ["What is the capital of Brazil?"], "text_2": ["The capital of Brazil is Brasilia."] }' ``` Multimodal (text vs. images): ```json { "model": "jinaai/jina-reranker-m0", "text_1": "A cat", "text_2": { "content": [ {"type": "image_url", "image_url": {"url": "cat_img.jpg"}}, {"type": "image_url", "image_url": {"url": "dog_img.jpg"}} ] } } ``` ## Offline Deployment ```python from vllm import LLM llm = LLM( model="jinaai/jina-reranker-m0", tensor_parallel_size=2, gpu_memory_utilization=0.75, max_model_len=1024, max_num_seqs=32, kv_cache_dtype="fp8", dtype="bfloat16", ) res = llm.score("fast recipes for weeknight dinners", [ "A 65-minute pasta with garlic and olive oil.", "Slow braised short ribs that cook for 5 hours.", "Stir-fry veggies with pre-cooked rice.", ]) for item in res: print(item.outputs.score) ``` ## References - [jina-reranker-m0 on Hugging Face](https://huggingface.co/jinaai/jina-reranker-m0) - [vLLM Score API documentation](https://docs.vllm.ai/en/latest/serving/openai_compatible_server.html#score-api) --- # meituan-longcat/LongCat-Image-Edit on vLLM Source: https://recipes.vllm.ai/meituan-longcat/LongCat-Image-Edit Provider: LongCat (Meituan) Released: 2025-12-05T07:34:53.000Z Updated: 2026-04-17 Architecture: dense Parameters: 6B Min vLLM: 0.12.0 Tasks: omni Summary: Bilingual (Chinese-English) image editing model from Meituan LongCat, served via vLLM-Omni ## Variants - default: precision=bf16, vram_minimum_gb=36 BF16 weights; served via vLLM-Omni ## Guide ## Overview LongCat-Image-Edit is the image-editing variant of LongCat-Image by Meituan. It supports bilingual (Chinese-English) editing instructions and is served via **vLLM-Omni** (not standard vLLM). It achieves SOTA performance among open-source image editing models. ## Prerequisites - Hardware: 1x GPU with >=40 GB VRAM - vLLM-Omni (runs on top of vLLM 0.12.0) - diffusers (latest from source) - xformers (latest) ## Installation ```bash # Clone and install vllm-omni git clone https://github.com/vllm-project/vllm-omni.git cd vllm-omni uv venv source .venv/bin/activate uv pip install -e . vllm==0.12.0 # Update xformers to the latest version uv pip install -U xformers --index-url https://download.pytorch.org/whl/cu128 # Update diffusers to the latest version git clone https://github.com/huggingface/diffusers.git cd diffusers uv pip install -e . ``` ## Usage ```bash cd vllm-omni python3 ./examples/offline_inference/image_to_image/image_edit.py \ --image qwen_bear.png \ --prompt "Add a white art board written with colorful text 'vLLM-Omni' on grassland. Add a paintbrush in the bear's hands. Position the bear standing in front of the art board as if painting." \ --output output_image_edit.png \ --num_inference_steps 50 \ --guidance_scale 4.5 \ --seed 42 \ --model meituan-longcat/LongCat-Image-Edit \ --cache_backend cache_dit \ --cache_dit_max_continuous_cached_steps 2 ``` ## References - [LongCat-Image-Edit on Hugging Face](https://huggingface.co/meituan-longcat/LongCat-Image-Edit) - [vLLM-Omni](https://github.com/vllm-project/vllm-omni) --- # microsoft/Phi-4-mini-instruct on vLLM Source: https://recipes.vllm.ai/microsoft/Phi-4-mini-instruct Provider: Microsoft Released: 2025-02-19T01:00:58.000Z Updated: 2026-04-17 Architecture: dense Parameters: 4B Context length: 131,072 Min vLLM: 0.7.0 Tasks: text, multimodal Verified hardware: H100 Summary: Microsoft's Phi-4 family of lightweight dense models (mini-instruct, reasoning, multimodal) with 128K context ## Variants - default: precision=bf16, vram_minimum_gb=10 Phi-4-mini-instruct, conversational instruction-tuned - mini_reasoning: precision=bf16, vram_minimum_gb=10, model_id=microsoft/Phi-4-mini-reasoning Optimized for reasoning tasks - reasoning: precision=bf16, vram_minimum_gb=30, model_id=microsoft/Phi-4-reasoning Advanced reasoning capabilities (14B) - multimodal: precision=bf16, vram_minimum_gb=16, model_id=microsoft/Phi-4-multimodal-instruct Multimodal instruction-following (text + image) - fp8: precision=fp8, vram_minimum_gb=5, model_id=RedHatAI/Phi-4-mini-instruct-FP8-dynamic RedHatAI FP8 (E4M3) with dynamic per-token activation quantization ## Guide ## Overview The Phi-4 family includes several lightweight, open models from Microsoft. These models can process text and, in some variants, multimodal inputs like images, to generate text outputs. They come with a 128K token context length. ## Prerequisites - Hardware: 1x GPU with >=16 GB VRAM (A100, L40S, H100, etc.) - vLLM >= 0.7.0 (the `multimodal` variant serves `Phi-4-multimodal-instruct`, which needs vLLM >= 0.8.0) ### Install vLLM ```bash uv venv source .venv/bin/activate uv pip install -U vllm --torch-backend auto ``` ## Launch commands Phi-4-mini-instruct on a single GPU: ```bash vllm serve microsoft/Phi-4-mini-instruct \ --host 0.0.0.0 \ --max-model-len 4000 ``` Phi-4-multimodal-instruct (requires --trust-remote-code for LoRA modules): ```bash vllm serve microsoft/Phi-4-multimodal-instruct \ --host 0.0.0.0 \ --max-model-len 4000 \ --trust-remote-code ``` ## Client Usage ```python from openai import OpenAI client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1", timeout=3600) response = client.chat.completions.create( model="microsoft/Phi-4-mini-instruct", messages=[{"role": "user", "content": "Write a short story"}], temperature=0.0, ) print(response.choices[0].message.content) ``` Multimodal (requires Phi-4-multimodal-instruct): ```python response = client.chat.completions.create( model="microsoft/Phi-4-multimodal-instruct", messages=[{ "role": "user", "content": [ {"type": "text", "text": "Describe this image."}, {"type": "image_url", "image_url": {"url": "https://example.com/image.jpg"}}, ], }], ) ``` ## Available Phi-4 Variants - `microsoft/Phi-4-mini-instruct` — conversational instruction-tuned - `microsoft/Phi-4-mini-reasoning` — optimized for reasoning - `microsoft/Phi-4-reasoning` — advanced reasoning - `microsoft/Phi-4-multimodal-instruct` — multimodal (text + image) ## Benchmarking ```bash vllm bench serve \ --model microsoft/Phi-4-mini-instruct \ --dataset-name random \ --random-input-len 2000 \ --random-output-len 512 \ --num-prompts 100 ``` ## Troubleshooting - Multimodal variant fails to load: add `--trust-remote-code`. ## References - [Phi-4-mini-instruct](https://huggingface.co/microsoft/Phi-4-mini-instruct) - [Phi-4-multimodal-instruct](https://huggingface.co/microsoft/Phi-4-multimodal-instruct) - [Phi-4-reasoning](https://huggingface.co/microsoft/Phi-4-reasoning) --- # mistralai/Mistral-Large-3-675B-Instruct-2512 on vLLM Source: https://recipes.vllm.ai/mistralai/Mistral-Large-3-675B-Instruct-2512 Provider: Mistral AI Released: 2025-11-28T18:05:12.000Z Updated: 2026-04-17 Architecture: moe Parameters: 675B Context length: 294,912 Min vLLM: 0.11.0 Tasks: multimodal Verified hardware: H100, B200 Summary: Mistral Large 3 (675B) with FP8 and NVFP4 weights for 8xH200 / 4xB200 deployments ## Variants - default: precision=fp8, vram_minimum_gb=810 FP8 weights on 8xH200 (recommended for fine-tuning; up to 256K context) - nvfp4: precision=nvfp4, vram_minimum_gb=405, model_id=mistralai/Mistral-Large-3-675B-Instruct-2512-NVFP4 NVFP4 weights on 4xB200 (use for <64K context; B200-native, Marlin fallback on A100/H100) - fp8: precision=fp8, vram_minimum_gb=810, model_id=mistralai/Mistral-Large-3-675B-Instruct-2512-FP8 FP8 quantized weights for Hopper/Blackwell GPUs - redhat_nvfp4: precision=nvfp4, vram_minimum_gb=405, model_id=RedHatAI/Mistral-Large-3-675B-Instruct-2512-NVFP4 RedHatAI NVFP4 quantization ## Recommended base flags ```bash vllm serve 'mistralai/Mistral-Large-3-675B-Instruct-2512' \ --tokenizer_mode \ mistral \ --config_format \ mistral \ --load_format \ mistral ``` ## Guide ## Overview Mistral-Large-3-675B-Instruct-2512 is available in FP8 and NVFP4 formats: - **FP8** (`mistralai/Mistral-Large-3-675B-Instruct-2512`): up to 256K context, 8xH200 - **NVFP4** (`mistralai/Mistral-Large-3-675B-Instruct-2512-NVFP4`): best for <64K context, 4xB200 NVFP4 gives significant speed-up on B200 (native FP4 support). On older GPUs (A100/H100), vLLM falls back to Marlin FP4, which matches FP8 speed while saving memory. For large contexts (>64K) we observed a performance regression on NVFP4 — use FP8 in those cases. A minor regression on vision datasets is expected with NVFP4 (calibration was mainly on text). ## Prerequisites - Hardware: 8xH200 for FP8, 4xB200 for NVFP4 - vLLM >= 0.11.0 ### Install vLLM ```bash uv venv source .venv/bin/activate uv pip install -U vllm --torch-backend auto ``` ## Launch commands FP8 on 8xH200: ```bash vllm serve mistralai/Mistral-Large-3-675B-Instruct-2512 \ --tensor-parallel-size 8 \ --tokenizer_mode mistral --config_format mistral --load_format mistral \ --enable-auto-tool-choice --tool-call-parser mistral ``` NVFP4 on 4xB200: ```bash vllm serve mistralai/Mistral-Large-3-675B-Instruct-2512-NVFP4 \ --tensor-parallel-size 4 \ --tokenizer_mode mistral --config_format mistral --load_format mistral \ --enable-auto-tool-choice --tool-call-parser mistral ``` Additional flags: - `--max-model-len`: default 262144; reduce to save memory - `--max-num-batched-tokens`: balance throughput vs. latency - `--limit-mm-per-prompt.image 0`: skip vision encoder for text-only tasks ## Client Usage ```python from openai import OpenAI client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1") response = client.chat.completions.create( model="mistralai/Mistral-Large-3-675B-Instruct-2512", messages=[{"role": "user", "content": "Write a sentence..."}], temperature=0.15, max_tokens=262144, ) print(response.choices[0].message.content) ``` ## Troubleshooting - Accuracy regression with NVFP4 at long context: switch to FP8 variant. - OOM: reduce `--max-model-len` or adjust TP size. ## References - [Mistral-Large-3 FP8](https://huggingface.co/mistralai/Mistral-Large-3-675B-Instruct-2512) - [Mistral-Large-3 NVFP4](https://huggingface.co/mistralai/Mistral-Large-3-675B-Instruct-2512-NVFP4) --- # moonshotai/Kimi-K2-Instruct on vLLM Source: https://recipes.vllm.ai/moonshotai/Kimi-K2-Instruct Provider: Moonshot AI Released: 2025-07-11T00:55:12.000Z Updated: 2026-04-17 Architecture: moe Parameters: 1T Context length: 131,072 Min vLLM: 0.12.0 Tasks: text Verified hardware: H200, MI300X, MI325X, MI355X Summary: Moonshot AI's Kimi-K2 is a trillion-parameter MoE instruction model (~32B active) with native FP8 weights and strong tool-calling capabilities. ## Variants - default: precision=fp8, vram_minimum_gb=1200 Native FP8 weights on 16xH800 / 16xH200 (smallest deployment for 128k seqlen) ## Recommended base flags ```bash vllm serve 'moonshotai/Kimi-K2-Instruct' \ --trust-remote-code \ --tokenizer-mode \ auto ``` ## Guide ## Overview Kimi-K2-Instruct is Moonshot AI's trillion-parameter Mixture-of-Experts instruction model (approximately 32B activated per token) shipped with native FP8 weights. The smallest deployment unit for Kimi-K2 FP8 weights with 128k seqlen on mainstream H800 platforms is a 16-GPU cluster using either Tensor Parallel (TP) or Data Parallel + Expert Parallel (DP+EP). This guide is partially adapted from the official [Kimi-K2-Instruct Deployment Guidance](https://huggingface.co/moonshotai/Kimi-K2-Instruct/blob/main/docs/deploy_guidance.md). ## Prerequisites - **Hardware (FP8)**: 16x H800 or 16x H200 GPUs (verified) - **vLLM**: Current stable release ```bash uv venv source .venv/bin/activate uv pip install -U vllm --torch-backend auto ``` ### AMD ROCm The ROCm wheel requires Python 3.12, ROCm 7.0+, and glibc >= 2.35. Use the ROCm Docker image when your host environment does not meet those requirements. ```bash uv venv --python 3.12 source .venv/bin/activate uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/ ``` ## Running Kimi-K2 with FP8 on 16xH800 ### Tensor Parallelism + Pipeline Parallelism (TP8+PP2) ```bash # node 0 (start Ray on both nodes first) vllm serve moonshotai/Kimi-K2-Instruct \ --trust-remote-code \ --tokenizer-mode auto \ --tensor-parallel-size 8 \ --pipeline-parallel-size 2 \ --dtype bfloat16 \ --quantization fp8 \ --max-model-len 2048 \ --max-num-seqs 1 \ --max-num-batched-tokens 1024 \ --enable-chunked-prefill \ --kv-cache-dtype fp8 \ -dcp 8 ``` Key parameter notes: - `--enable-auto-tool-choice`: required when enabling tool usage. - `--tool-call-parser kimi_k2`: required when enabling tool usage. ### Data Parallelism + Expert Parallelism (DP16+EP) You can install libraries like DeepEP and DeepGEMM as needed. Then run the command (example on H800): ```bash # node 0 vllm serve moonshotai/Kimi-K2-Instruct \ --port 8000 --served-model-name kimi-k2 \ --trust-remote-code \ --data-parallel-size 16 \ --data-parallel-size-local 8 \ --data-parallel-address $MASTER_IP \ --data-parallel-rpc-port $PORT \ --enable-expert-parallel \ --max-num-batched-tokens 8192 \ --max-num-seqs 256 \ --gpu-memory-utilization 0.85 \ --enable-auto-tool-choice \ --tool-call-parser kimi_k2 # node 1 vllm serve moonshotai/Kimi-K2-Instruct \ --headless \ --data-parallel-start-rank 8 \ --port 8000 --served-model-name kimi-k2 \ --trust-remote-code \ --data-parallel-size 16 \ --data-parallel-size-local 8 \ --data-parallel-address $MASTER_IP \ --data-parallel-rpc-port $PORT \ --enable-expert-parallel \ --max-num-batched-tokens 8192 \ --max-num-seqs 256 \ --gpu-memory-utilization 0.85 \ --enable-auto-tool-choice \ --tool-call-parser kimi_k2 ``` ### ROCm on 8x MI300X / MI325X / MI355X ```bash export SAFETENSORS_FAST_GPU=1 vllm serve moonshotai/Kimi-K2-Instruct \ --tensor-parallel-size 8 \ --trust-remote-code \ --enable-auto-tool-choice \ --tool-call-parser kimi_k2 ``` Additional flags: - `--max-model-len` preserves memory; `--max-model-len=65536` is usually good for most scenarios. - `--max-num-batched-tokens` balances throughput vs latency. `32768` is good for prompt-heavy workloads; reduce to 16k or 8k to cut activation memory and decrease latency. - vLLM conservatively uses 90% of GPU memory. Set `--gpu-memory-utilization=0.95` to maximize KV cache. ## Benchmarking ### FP8 Benchmark on 16xH800 ```bash vllm bench serve \ --model moonshotai/Kimi-K2-Instruct \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 512 \ --request-rate 1.0 \ --num-prompts 8 \ --ignore-eos \ --trust-remote-code ``` ### FP8 Benchmark on 16xH200 ```bash vllm bench serve \ --model moonshotai/Kimi-K2-Instruct \ --dataset-name random \ --random-input-len 8000 \ --random-output-len 1000 \ --request-rate 10000 \ --num-prompts 16 \ --ignore-eos \ --trust-remote-code ``` Adding `-dcp 8` at launch can further improve throughput on H200 (observed ~33% lower mean TTFT and higher tok/s in internal benchmarks). Test different batch sizes by changing `--num-prompts`, e.g. 1, 16, 32, 64, 128, 256, 512. ## References - [Kimi-K2-Instruct on Hugging Face](https://huggingface.co/moonshotai/Kimi-K2-Instruct) - [Official Kimi-K2 Deployment Guidance](https://huggingface.co/moonshotai/Kimi-K2-Instruct/blob/main/docs/deploy_guidance.md) - [vLLM Expert Parallelism docs](https://docs.vllm.ai/en/latest/serving/expert_parallel_deployment.html) --- # moonshotai/Kimi-K2-Thinking on vLLM Source: https://recipes.vllm.ai/moonshotai/Kimi-K2-Thinking Provider: Moonshot AI Released: 2025-11-04T08:25:31.000Z Updated: 2026-04-17 Architecture: moe Parameters: 1T Context length: 262,144 Min vLLM: 0.12.0 Tasks: text Verified hardware: H200, MI300X, MI325X, MI355X Summary: Kimi-K2-Thinking is an advanced reasoning MoE model with native INT4 QAT weights, designed for long-horizon agent workflows interleaving chain-of-thought reasoning with tool calls. ## Variants - default: precision=int4, vram_minimum_gb=600 Native INT4 (QAT) weights on 8xH200 / 8xH20; 2x low-latency speed-up vs FP8 - nvfp4: precision=nvfp4, vram_minimum_gb=600, model_id=nvidia/Kimi-K2-Thinking-NVFP4 NVIDIA NVFP4 quantized weights for Blackwell GPUs - fp8: precision=fp8, vram_minimum_gb=1200, model_id=RedHatAI/Kimi-K2-Thinking-FP8-Block RedHatAI FP8 block quantization ## Recommended base flags ```bash vllm serve 'moonshotai/Kimi-K2-Thinking' \ --trust-remote-code ``` ## Guide ## Overview [Kimi-K2-Thinking](https://huggingface.co/moonshotai/Kimi-K2-Thinking) is an advanced trillion-parameter MoE created by Moonshot AI with these highlights: - **Deep Thinking & Tool Orchestration**: End-to-end trained to interleave chain-of-thought reasoning with function calls, enabling autonomous research, coding, and writing workflows that last hundreds of steps without drift. - **Native INT4 Quantization**: Quantization-Aware Training (QAT) delivers lossless 2x speed-up in low-latency mode. - **Stable Long-Horizon Agency**: Maintains coherent goal-directed behavior across up to 200-300 consecutive tool invocations, surpassing prior models that degrade after 30-50 steps. ## Prerequisites - **Hardware**: 8x H200 or 8x H20 GPUs - **vLLM**: Current stable release ```bash uv venv source .venv/bin/activate uv pip install -U vllm --torch-backend auto ``` ### AMD ROCm The ROCm wheel requires Python 3.12, ROCm 7.0+, and glibc >= 2.35. Use the ROCm Docker image when your host environment does not meet those requirements. ```bash uv venv --python 3.12 source .venv/bin/activate uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/ ``` ## Launching Kimi-K2-Thinking with vLLM ### Low-Latency Scenarios (TP8) ```bash vllm serve moonshotai/Kimi-K2-Thinking \ --tensor-parallel-size 8 \ --enable-auto-tool-choice \ --tool-call-parser kimi_k2 \ --reasoning-parser kimi_k2 \ --trust-remote-code ``` The `--reasoning-parser` flag specifies the parser used to extract reasoning content from the model output. ### ROCm on 8x MI300X / MI325X / MI355X ```bash export SAFETENSORS_FAST_GPU=1 vllm serve moonshotai/Kimi-K2-Thinking \ --tensor-parallel-size 8 \ --enable-auto-tool-choice \ --tool-call-parser kimi_k2 \ --reasoning-parser kimi_k2 \ --trust-remote-code ``` ### High-Throughput Scenarios (TP8+DCP8) vLLM supports [Decode Context Parallel](https://docs.vllm.ai/en/latest/serving/context_parallel_deployment.html#decode-context-parallel), which provides significant benefits in high-throughput scenarios. Enable DCP by adding `--decode-context-parallel-size 8`: ```bash vllm serve moonshotai/Kimi-K2-Thinking \ --tensor-parallel-size 8 \ --decode-context-parallel-size 8 \ --enable-auto-tool-choice \ --tool-call-parser kimi_k2 \ --reasoning-parser kimi_k2 \ --trust-remote-code ``` ## Metrics (GSM8K) | Config | exact_match (flexible) | exact_match (strict) | |--------|-----------------------|----------------------| | TP8 | 0.9416 | 0.9409 | | TP8+DCP8 | 0.9386 | 0.9371 | ## Benchmarking We used the following script to benchmark `moonshotai/Kimi-K2-Thinking` on 8xH200: ```bash vllm bench serve \ --model moonshotai/Kimi-K2-Thinking \ --dataset-name random \ --random-input 8000 \ --random-output 4000 \ --request-rate 100 \ --num-prompt 1000 \ --trust-remote-code ``` ### DCP Gain Analysis | Metric | TP8 | TP8+DCP8 | Change | Improvement | |--------|-----|----------|--------|-------------| | Request throughput (req/s) | 1.25 | 1.57 | +0.32 | +25.6% | | Output token throughput (tok/s) | 485.78 | 695.13 | +209.35 | +43.1% | | Mean TTFT (s) | 271.2 | 227.8 | -43.4 | +16.0% | DCP multiplies the GPU KV cache size by `dcp_world_size`: - TP8 KV cache: `715,072` tokens - TP8+DCP8 KV cache: `5,721,088` tokens (8x) Enabling DCP delivers strong advantages (43% faster token generation, 26% higher throughput) with minimal drawbacks. Read the [DCP doc](https://docs.vllm.ai/en/latest/serving/context_parallel_deployment.html#decode-context-parallel) and try it in your LLM workloads. ## References - [Kimi-K2-Thinking on Hugging Face](https://huggingface.co/moonshotai/Kimi-K2-Thinking) - [vLLM Decode Context Parallel docs](https://docs.vllm.ai/en/latest/serving/context_parallel_deployment.html#decode-context-parallel) --- # moonshotai/Kimi-Linear-48B-A3B-Instruct on vLLM Source: https://recipes.vllm.ai/moonshotai/Kimi-Linear-48B-A3B-Instruct Provider: Moonshot AI Released: 2025-10-30T12:37:31.000Z Updated: 2026-04-17 Architecture: moe Parameters: 48B Context length: 1,048,576 Min vLLM: 0.11.2 Tasks: text Verified hardware: H200, MI300X, MI325X, MI355X Summary: Kimi-Linear is a 48B-parameter instruction-tuned MoE model (~3B activated) with a linear-attention variant supporting very long context (1M tokens). ## Variants - default: precision=bf16, vram_minimum_gb=115 Full precision BF16 on 4 or 8 GPUs (single node) ## Recommended base flags ```bash vllm serve 'moonshotai/Kimi-Linear-48B-A3B-Instruct' \ --trust-remote-code ``` ## Guide ## Overview Kimi-Linear is Moonshot AI's 48B-parameter instruction-tuned MoE model (`A3B` indicates ~3B active parameters per token) featuring a linear-attention variant that enables very long context windows (up to 1,048,576 tokens). ## Prerequisites - **Hardware**: 4 or 8 GPUs on a single node - **vLLM**: `0.11.2` recommended. Avoid vLLM `0.12.0`, which has a known bug `MLAModules.__init__() missing 1 required positional argument: 'indexer_rotary_emb'` that affects Kimi-Linear. ```bash uv venv source .venv/bin/activate # Install a stable version (avoid 0.12.0) uv pip install vllm==0.11.2 --torch-backend auto ``` ### AMD ROCm The ROCm wheel requires Python 3.12, ROCm 7.0+, and glibc >= 2.35. Use the ROCm Docker image when your host environment does not meet those requirements. ```bash uv venv --python 3.12 source .venv/bin/activate uv pip install vllm==0.11.2 --extra-index-url https://wheels.vllm.ai/rocm/ ``` ## Running Kimi-Linear The following snippets assume 4 or 8 GPUs on a single node. ### 4-GPU Tensor Parallel ```bash vllm serve moonshotai/Kimi-Linear-48B-A3B-Instruct \ --port 8000 \ --tensor-parallel-size 4 \ --max-model-len 1048576 \ --trust-remote-code ``` ### 8-GPU Tensor Parallel ```bash vllm serve moonshotai/Kimi-Linear-48B-A3B-Instruct \ --port 8000 \ --tensor-parallel-size 8 \ --max-model-len 1048576 \ --trust-remote-code ``` ### ROCm on 8x MI300X / MI325X / MI355X ```bash export SAFETENSORS_FAST_GPU=1 vllm serve moonshotai/Kimi-Linear-48B-A3B-Instruct \ --tensor-parallel-size 8 \ --max-model-len auto \ --trust-remote-code ``` If you see OOM, reduce `--max-model-len` (e.g. 65536) or increase `--gpu-memory-utilization` (<= 0.95). ## Client Usage Once the server is up, test it with: ```bash curl http://localhost:8000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{"model":"moonshotai/Kimi-Linear-48B-A3B-Instruct","messages":[{"role":"user","content":"Hello!"}]}' ``` ## Troubleshooting - **`MLAModules.__init__() missing 1 required positional argument: 'indexer_rotary_emb'`**: Known bug in vLLM 0.12.0 affecting Kimi-Linear. Pin to `vllm==0.11.2` instead. - **OOM**: Reduce `--max-model-len` or increase `--gpu-memory-utilization` up to 0.95. ## References - [Kimi-Linear-48B-A3B-Instruct on Hugging Face](https://huggingface.co/moonshotai/Kimi-Linear-48B-A3B-Instruct) --- # nvidia/NVIDIA-Nemotron-Nano-12B-v2-VL-BF16 on vLLM Source: https://recipes.vllm.ai/nvidia/NVIDIA-Nemotron-Nano-12B-v2-VL-BF16 Provider: NVIDIA Released: 2025-10-21T18:11:05.000Z Updated: 2026-04-17 Architecture: dense Parameters: 12B Context length: 131,072 Min vLLM: 0.11.1 Tasks: multimodal Verified hardware: H100, B200 Summary: NVIDIA Nemotron-Nano 12B vision-language model with video support and Efficient Video Sampling (EVS) ## Variants - default: precision=bf16, vram_minimum_gb=29 BF16 weights on 1 GPU - fp8: precision=fp8, vram_minimum_gb=14, model_id=nvidia/NVIDIA-Nemotron-Nano-12B-v2-VL-FP8 FP8 weights on 1 GPU - nvfp4: precision=nvfp4, vram_minimum_gb=8, model_id=nvidia/NVIDIA-Nemotron-Nano-12B-v2-VL-NVFP4-QAD NVFP4 (QAD) weights for Blackwell ## Recommended base flags ```bash vllm serve 'nvidia/NVIDIA-Nemotron-Nano-12B-v2-VL-BF16' \ --trust-remote-code ``` Environment: ```bash VLLM_VIDEO_LOADER_BACKEND='opencv' ``` ## Guide ## Overview Nemotron-Nano-12B-v2-VL is a vision-language model with image and video support. It includes Efficient Video Sampling (EVS) to prune video tokens and reduce compute. The model is available in BF16, FP8, and NVFP4 (QAD) precisions. ## Prerequisites - Hardware: 1x GPU (A100/H100/B200, etc.) - vLLM: 0.11.0 does NOT include this model; use latest nightly or install from source - DGX Spark: use `nvcr.io/nvidia/vllm:25.12.post1-py3` ### Install vLLM ```bash docker pull vllm/vllm-openai:nightly-8bff831f0aa239006f34b721e63e1340e3472067 # or for DGX Spark: docker pull nvcr.io/nvidia/vllm:25.12.post1-py3 ``` ## Launch command ```bash export VLLM_VIDEO_LOADER_BACKEND=opencv export CHECKPOINT_PATH="nvidia/NVIDIA-Nemotron-Nano-12B-v2-VL-BF16" export CUDA_VISIBLE_DEVICES=0 python3 -m vllm.entrypoints.openai.api_server \ --model ${CHECKPOINT_PATH} \ --trust-remote-code \ --media-io-kwargs '{"video": {"fps": 2, "num_frames": 128}}' \ --max-model-len 131072 \ --data-parallel-size 1 \ --port 5566 \ --allowed-local-media-path / \ --video-pruning-rate 0.75 \ --served-model-name "nvidia/NVIDIA-Nemotron-Nano-12B-v2-VL-BF16" ``` Flags: - `--max-model-len`: reduce for shorter contexts to save memory - `--allowed-local-media-path `: limit local-file access - `--video-pruning-rate <0..1>`: EVS compression; higher prunes more video tokens ## Client Usage Describe a video: ```python from openai import OpenAI client = OpenAI(base_url="http://localhost:5566/v1", api_key="") completion = client.chat.completions.create( model="nvidia/NVIDIA-Nemotron-Nano-12B-v2-VL-BF16", messages=[{ "role": "user", "content": [ {"type": "text", "text": "Describe the video."}, {"type": "video_url", "video_url": {"url": "file:///path/to/video.mp4"}}, ], }], ) print(completion.choices[0].message.content) ``` ## Offline / LLM API ```python from vllm import LLM, SamplingParams llm = LLM( "nvidia/NVIDIA-Nemotron-Nano-12B-v2-VL-BF16", trust_remote_code=True, max_model_len=2**17, allowed_local_media_path="/", video_pruning_rate=0.75, media_io_kwargs=dict(video=dict(fps=2, num_frames=128)), ) ``` ## Troubleshooting - Set `VLLM_VIDEO_LOADER_BACKEND=opencv` (required for video inputs). - OOM: lower `--max-model-len` or increase `--video-pruning-rate`. ## References - [Nemotron-Nano-12B-v2-VL-BF16](https://huggingface.co/nvidia/NVIDIA-Nemotron-Nano-12B-v2-VL-BF16) - [Nemotron-Nano-12B-v2-VL-FP8](https://huggingface.co/nvidia/NVIDIA-Nemotron-Nano-12B-v2-VL-FP8) - [Nemotron-Nano-12B-v2-VL-NVFP4-QAD](https://huggingface.co/nvidia/NVIDIA-Nemotron-Nano-12B-v2-VL-NVFP4-QAD) - [EVS paper](https://arxiv.org/abs/2510.14624) --- # stabilityai/stable-diffusion-3.5-medium on vLLM Source: https://recipes.vllm.ai/stabilityai/stable-diffusion-3.5-medium Provider: Stability AI Released: 2024-10-29T10:27:32.000Z Updated: 2026-04-17 Architecture: dense Parameters: 2.5B Min vLLM: 0.12.0 Tasks: omni Verified hardware: MI300X, MI325X, MI355X Summary: Stability AI's Stable Diffusion 3.5 text-to-image family (medium 2.5B, large 8.1B, large-turbo) via vLLM-Omni with Cache-DiT acceleration ## Variants - default: precision=bf16, vram_minimum_gb=44 Stable Diffusion 3.5 medium (2.5B) - large: precision=bf16, vram_minimum_gb=24, model_id=stabilityai/stable-diffusion-3.5-large Stable Diffusion 3.5 large (8.1B) - large_turbo: precision=bf16, vram_minimum_gb=24, model_id=stabilityai/stable-diffusion-3.5-large-turbo Stable Diffusion 3.5 large-turbo (8.1B, timestep-distilled for few-step inference) ## Guide ## Overview Stable Diffusion 3.5 text-to-image generation models, served via **vLLM-Omni** with optional **Cache-DiT** acceleration. Supported variants: - `stabilityai/stable-diffusion-3.5-medium` — 2.5B params - `stabilityai/stable-diffusion-3.5-large` — 8.1B params - `stabilityai/stable-diffusion-3.5-large-turbo` — 8.1B params (timestep-distilled for few-step inference) ## Prerequisites - vLLM-Omni on top of vLLM 0.12.0 - `diffusers` library ## Installation ```bash uv venv source .venv/bin/activate uv pip install vllm==0.12.0 uv pip install git+https://github.com/vllm-project/vllm-omni.git ``` ### AMD ROCm (MI300X, MI325X, MI355X) > **Note:** The vLLM wheel for ROCm requires Python 3.12, ROCm 7.2.1, and glibc >= 2.35. > If your environment does not meet these requirements, please use the Docker-based setup. ```bash uv venv --python 3.12 source .venv/bin/activate VLLM_OMNI_TARGET_DEVICE=rocm uv pip install git+https://github.com/vllm-project/vllm-omni.git vllm==0.20.0+rocm721 --extra-index-url https://wheels.vllm.ai/rocm/0.20.0/rocm721 ``` ## Python Usage ```python from vllm_omni.entrypoints.omni import Omni omni = Omni(model="stabilityai/stable-diffusion-3.5-medium") images = omni.generate( prompt="a cat wearing sunglasses, cyberpunk style", negative_prompt="blurry, low quality", height=1024, width=1024, num_inference_steps=28, guidance_scale=7.5, num_outputs_per_prompt=2, ) ``` ## CLI Usage ```bash python examples/offline_inference/text_to_image/text_to_image.py \ --model stabilityai/stable-diffusion-3.5-medium \ --prompt "a cat wearing sunglasses, cyberpunk style" \ --negative-prompt "blurry, low quality" \ --height 1024 --width 1024 \ --num-inference-steps 28 \ --guidance-scale 7.5 ``` ## Cache-DiT Acceleration Enable caching for significant speed-ups: ```python omni = Omni( model="stabilityai/stable-diffusion-3.5-medium", cache_backend="cache_dit", cache_config={ "Fn_compute_blocks": 8, "Bn_compute_blocks": 0, "max_warmup_steps": 4, "residual_diff_threshold": 0.12, }, ) ``` ## Key Parameters | Parameter | Default | Description | |-----------|---------|-------------| | `height` | 1024 | Image height (multiples of 16) | | `width` | 1024 | Image width (multiples of 16) | | `num_inference_steps` | 28 | Denoising steps | | `guidance_scale` | 1.0 | Classifier-free guidance scale | ## References - [SD3.5 Medium](https://huggingface.co/stabilityai/stable-diffusion-3.5-medium) - [SD3.5 Large](https://huggingface.co/stabilityai/stable-diffusion-3.5-large) - [SD3.5 Large Turbo](https://huggingface.co/stabilityai/stable-diffusion-3.5-large-turbo) - [Cache-DiT Acceleration](https://github.com/vipshop/cache-dit) --- # stepfun-ai/Step-3.5-Flash on vLLM Source: https://recipes.vllm.ai/stepfun-ai/Step-3.5-Flash Provider: StepFun Released: 2026-02-01T08:03:45.000Z Updated: 2026-04-17 Architecture: moe Parameters: 196B Context length: 262,144 Min vLLM: 0.15.1 Tasks: text Verified hardware: H200, B200 Summary: Production-grade reasoning MoE (~196B total / 11B active parameters) with hybrid attention schedules, SWA compensation, and multi-token prediction for low-latency long-context inference ## Variants - default: precision=bf16, vram_minimum_gb=470 Full precision BF16 — runs on 4xH200/H20/B200 - fp8: precision=fp8, vram_minimum_gb=235, model_id=stepfun-ai/Step-3.5-Flash-FP8 Native FP8 checkpoint (TP not supported beyond 2 — use DP4) - int4: precision=int4, vram_minimum_gb=118, model_id=stepfun-ai/Step-3.5-Flash-INT4 INT4 quantized weights - int8: precision=int8, vram_minimum_gb=235, model_id=stepfun-ai/Step-3.5-Flash-INT8 INT8 quantized weights ## Recommended base flags ```bash vllm serve 'stepfun-ai/Step-3.5-Flash' \ --trust-remote-code ``` ## Guide ## Overview [Step-3.5-Flash](https://huggingface.co/stepfun-ai/Step-3.5-Flash) is an advanced reasoning engine from [StepFun](https://www.stepfun.com/company). Highlights: - Hybrid attention schedules with compensation for sliding-window attention (SWA) - Sparse MoE structure (196B total parameters, 11B active) - Multi-token prediction mechanism for faster inference Available precisions: - [stepfun-ai/Step-3.5-Flash](https://huggingface.co/stepfun-ai/Step-3.5-Flash) (BF16) - [stepfun-ai/Step-3.5-Flash-FP8](https://huggingface.co/stepfun-ai/Step-3.5-Flash-FP8) - [stepfun-ai/Step-3.5-Flash-Int4](https://huggingface.co/stepfun-ai/Step-3.5-Flash-Int4) (not yet supported by vLLM) ## Prerequisites - **vLLM version:** latest stable - **Hardware:** 4x H200/H20/B200 ### Install vLLM ```bash uv venv source .venv/bin/activate uv pip install vllm --torch-backend auto ``` ## Launching the Server ### Tensor Parallel ```bash vllm serve stepfun-ai/Step-3.5-Flash \ --tensor-parallel-size 4 \ --reasoning-parser step3p5 \ --tool-call-parser step3p5 \ --enable-auto-tool-choice \ --trust-remote-code ``` Note: The FP8 version cannot use TP4 — use DP4 instead. ### Data Parallel + Expert Parallel (recommended for FP8) ```bash vllm serve stepfun-ai/Step-3.5-Flash \ --data-parallel-size 4 \ --enable-expert-parallel \ --reasoning-parser step3p5 \ --tool-call-parser step3p5 \ --enable-auto-tool-choice \ --trust-remote-code ``` ### Enabling MTP Speculative Decoding ```bash vllm serve stepfun-ai/Step-3.5-Flash \ --tensor-parallel-size 4 \ --reasoning-parser step3p5 \ --tool-call-parser step3p5 \ --enable-auto-tool-choice \ --trust-remote-code \ --hf-overrides '{"num_nextn_predict_layers": 1}' \ --speculative-config '{"method": "step3p5_mtp", "num_speculative_tokens": 1}' ``` ## Benchmarking ```bash vllm bench serve \ --backend vllm \ --model stepfun-ai/Step-3.5-Flash \ --endpoint /v1/completions \ --dataset-name random \ --random-input 2048 \ --random-output 1024 \ --max-concurrency 10 \ --num-prompt 100 ``` ## Troubleshooting - **MoE kernel tuning:** See [tune-moe-kernel](https://github.com/vllm-project/recipes/blob/main/Qwen/Qwen3-Next.md#tune-moe-kernel) to tune Triton kernels for your hardware. - **FP8 DeepGEMM:** For FP8, install DeepGEMM via [install_deepgemm.sh](https://github.com/vllm-project/vllm/blob/v0.16.0rc0/tools/install_deepgemm.sh). - **B200 FlashInfer FP8 MoE error:** The `routing_logits must be bfloat16` error on B200 was fixed in vLLM v0.28.0 — no workaround is needed. - **FP8 + TP4 incompatibility:** Use DP4+EP instead. ## References - [Model card](https://huggingface.co/stepfun-ai/Step-3.5-Flash) - [FP8 checkpoint](https://huggingface.co/stepfun-ai/Step-3.5-Flash-FP8) - [StepFun](https://www.stepfun.com/company) --- # tencent/HunyuanOCR on vLLM Source: https://recipes.vllm.ai/tencent/HunyuanOCR Provider: Hunyuan (Tencent) Released: 2025-11-18T04:08:56.000Z Updated: 2026-04-17 Architecture: dense Parameters: 1B Context length: 32,768 Min vLLM: 0.12.0 Tasks: multimodal Summary: Tencent Hunyuan end-to-end OCR expert VLM (~1B) for online OCR serving with an OpenAI-compatible API ## Variants - default: precision=bf16, vram_minimum_gb=2 Full precision BF16 — single-GPU deployment ## Recommended base flags ```bash vllm serve 'tencent/HunyuanOCR' \ --no-enable-prefix-caching \ --mm-processor-cache-gb \ 0 ``` ## Guide ## Overview [HunyuanOCR](https://huggingface.co/tencent/HunyuanOCR) is a leading end-to-end OCR expert VLM powered by Hunyuan's native multimodal architecture. This recipe covers online serving with the OpenAI-compatible API. ## Prerequisites - **vLLM version:** latest stable - **Hardware:** single GPU (1B model) ### Install vLLM ```bash uv venv source .venv/bin/activate uv pip install -U vllm --torch-backend auto ``` ## Launching the Server ```bash vllm serve tencent/HunyuanOCR \ --no-enable-prefix-caching \ --mm-processor-cache-gb 0 ``` ### Configuration Tips - Use greedy sampling (`temperature=0.0`) or low temperature for optimal OCR accuracy. - OCR tasks generally do not benefit from prefix caching or image reuse; disabling them (as above) removes hashing/caching overhead. - Adjust `--max-num-batched-tokens` for throughput based on your hardware. ## Client Usage ```python from openai import OpenAI client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1", timeout=3600) messages = [ {"role": "system", "content": ""}, { "role": "user", "content": [ {"type": "image_url", "image_url": {"url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/chat-ui/tools-dark.png"}}, { "type": "text", "text": ( "Extract all information from the main body of the document image " "and represent it in markdown format, ignoring headers and footers. " "Tables should be expressed in HTML format, formulas in the document " "should be represented using LaTeX format, and the parsing should be " "organized according to the reading order." ) } ] } ] response = client.chat.completions.create( model="tencent/HunyuanOCR", messages=messages, temperature=0.0, extra_body={"top_k": 1, "repetition_penalty": 1.0}, ) print(response.choices[0].message.content) ``` ## Troubleshooting - **Accuracy:** Use `temperature=0.0` and `top_k=1` for deterministic OCR output. - **Application-oriented prompts:** See the [official model card](https://huggingface.co/tencent/HunyuanOCR#%F0%9F%92%AC-application-oriented-prompts) for prompts tuned to various document parsing tasks. ## References - [Model card](https://huggingface.co/tencent/HunyuanOCR) --- # zai-org/GLM-4.5V on vLLM Source: https://recipes.vllm.ai/zai-org/GLM-4.5V Provider: GLM (Z-AI) Released: 2025-08-10T13:55:30.000Z Updated: 2026-04-17 Architecture: moe Parameters: 107B Context length: 65,536 Min vLLM: 0.12.0 Tasks: multimodal Verified hardware: H100, H200, MI300X, MI325X, MI355X Summary: GLM-4.5 vision-language MoE model (~107B parameters, BF16) with image-text-to-text capability, 64K context, expert parallelism, and native FP8 ## Variants - default: precision=bf16, vram_minimum_gb=257 Full precision BF16 — runs on 4xH100/H200 - fp8: precision=fp8, vram_minimum_gb=128, model_id=zai-org/GLM-4.5V-FP8 Native FP8 checkpoint with minimal accuracy loss — recommended for cost-efficient serving ## Recommended base flags ```bash vllm serve 'zai-org/GLM-4.5V' \ --trust-remote-code \ --enable-expert-parallel \ --allowed-local-media-path \ / \ --mm-encoder-tp-mode \ data \ --mm-processor-cache-type \ shm ``` ## Guide ## Overview GLM-4.5V is the vision-language variant of GLM-4.5. It is an MoE model with ~107B total parameters that accepts image and text inputs. FP8 models have minimal accuracy loss versus BF16 and are recommended for cost-efficient serving. GLM-4.5V supports a 64K context length (use GLM-4.6V for 128K). ## Prerequisites - **vLLM version:** >= 0.12.0 - **Hardware:** 4x H100/H200 (BF16 or FP8), or 4x MI300X/MI325X/MI355X with ROCm - **Python:** 3.10+ for NVIDIA; Python 3.12 with ROCm 7.0 and glibc >= 2.35 for AMD ROCm wheels ### Install vLLM (NVIDIA) ```bash uv venv source .venv/bin/activate uv pip install -U vllm --torch-backend auto ``` ### Install vLLM (AMD ROCm) ```bash uv venv --python 3.12 source .venv/bin/activate uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm uv pip install "transformers>=5.0.0" ``` ## Launching the Server ### Tensor Parallel + Expert Parallel (FP8 on 4 GPUs) ```bash vllm serve zai-org/GLM-4.5V-FP8 \ --tensor-parallel-size 4 \ --tool-call-parser glm45 \ --reasoning-parser glm45 \ --enable-auto-tool-choice \ --enable-expert-parallel \ --allowed-local-media-path / \ --mm-encoder-tp-mode data \ --mm-processor-cache-type shm ``` ### AMD ROCm (4xMI300x/MI325x/MI355x)) ```bash export VLLM_ROCM_USE_AITER=1 vllm serve zai-org/GLM-4.5V-FP8 \ --tensor-parallel-size 4 \ --tool-call-parser glm45 \ --reasoning-parser glm45 \ --enable-auto-tool-choice \ --enable-expert-parallel \ --allowed-local-media-path / \ --mm-encoder-tp-mode data \ --mm-processor-cache-type shm ``` ### Tuning Tips - `--max-model-len=65536` is near the model's max context (64K). - `--max-num-batched-tokens=32768` for prompt-heavy workloads. - `--gpu-memory-utilization=0.95` maximizes KV cache. - Use `uv pip install "transformers>=5.0.0"` before serving GLM-V checkpoints. - `--mm-encoder-tp-mode data` runs the vision encoder data-parallel — preferable to TP since the encoder is small and TP adds communication overhead. - `--mm-processor-cache-type shm` enables shared-memory caching for repeated image inputs. ## Client Usage ```python from openai import OpenAI client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1") resp = client.chat.completions.create( model="zai-org/GLM-4.5V-FP8", messages=[{ "role": "user", "content": [ {"type": "image_url", "image_url": {"url": "https://example.com/image.png"}}, {"type": "text", "text": "Describe the image."} ] }], max_tokens=512, ) print(resp.choices[0].message.content) ``` ## Benchmarking ```bash vllm bench serve \ --backend openai-chat \ --endpoint /v1/chat/completions \ --model zai-org/GLM-4.5V-FP8 \ --dataset-name hf \ --dataset-path lmarena-ai/VisionArena-Chat \ --num-prompts 1000 \ --request-rate 20 ``` ## Troubleshooting - **Vision encoder overhead:** Use `--mm-encoder-tp-mode data` unless TP-sharded encoder is known-good for your config. - **Context length errors:** GLM-4.5V max is 64K; use GLM-4.6V if you need 128K. ## References - [Model card](https://huggingface.co/zai-org/GLM-4.5V) - [FP8 checkpoint](https://huggingface.co/zai-org/GLM-4.5V-FP8) - [vLLM multimodal inputs guide](https://docs.vllm.ai/en/latest/features/multimodal_inputs.html) --- # zai-org/GLM-4.6V on vLLM Source: https://recipes.vllm.ai/zai-org/GLM-4.6V Provider: GLM (Z-AI) Released: 2025-12-07T07:20:45.000Z Updated: 2026-04-17 Architecture: moe Parameters: 107B Context length: 131,072 Min vLLM: 0.12.0 Tasks: multimodal Verified hardware: H100, H200, MI300X, MI325X, MI355X Summary: GLM-4.6 vision-language MoE model — image-text-to-text with 128K context, native FP8 checkpoint, and expert parallelism ## Variants - default: precision=bf16, vram_minimum_gb=257 Full precision BF16 — runs on 4xH100/H200 - fp8: precision=fp8, vram_minimum_gb=128, model_id=zai-org/GLM-4.6V-FP8 Native FP8 checkpoint with minimal accuracy loss ## Recommended base flags ```bash vllm serve 'zai-org/GLM-4.6V' \ --trust-remote-code \ --enable-expert-parallel \ --allowed-local-media-path \ / \ --mm-encoder-tp-mode \ data \ --mm-processor-cache-type \ shm ``` ## Guide ## Overview GLM-4.6V is an updated vision-language MoE model from Z-AI. It supports a 128K context length (vs 64K for GLM-4.5V). Native FP8 is recommended for cost-efficient serving, matching BF16 accuracy to within a small margin. ## Prerequisites - **vLLM version:** >= 0.12.0 - **Hardware:** 4x H100/H200 (BF16 or FP8), or 4x MI300X/MI325X/MI355X with ROCm - **Python:** 3.10+ for NVIDIA; Python 3.12 with ROCm 7.0 and glibc >= 2.35 for AMD ROCm wheels ### Install vLLM (NVIDIA) ```bash uv venv source .venv/bin/activate uv pip install -U vllm --torch-backend auto ``` ### Install vLLM (AMD ROCm) ```bash uv venv --python 3.12 source .venv/bin/activate uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm uv pip install "transformers>=5.0.0" ``` ## Launching the Server ### Tensor Parallel + Expert Parallel (FP8 on 4 GPUs) ```bash vllm serve zai-org/GLM-4.6V-FP8 \ --tensor-parallel-size 4 \ --tool-call-parser glm45 \ --reasoning-parser glm45 \ --enable-auto-tool-choice \ --enable-expert-parallel \ --allowed-local-media-path / \ --mm-encoder-tp-mode data \ --mm-processor-cache-type shm ``` ### AMD ROCm (FP8 on 4 GPUs) ```bash export SAFETENSORS_FAST_GPU=1 export VLLM_ROCM_USE_AITER=1 vllm serve zai-org/GLM-4.6V-FP8 \ --tensor-parallel-size 4 \ --tool-call-parser glm45 \ --reasoning-parser glm45 \ --enable-auto-tool-choice \ --enable-expert-parallel \ --allowed-local-media-path / \ --mm-encoder-tp-mode data \ --mm-processor-cache-type shm ``` ### Tuning Tips - `--max-model-len=65536` is a common default; you can push to 131072. - `--max-num-batched-tokens=32768` for prompt-heavy workloads. - `--mm-encoder-tp-mode data` + `--mm-processor-cache-type shm` for efficient vision processing. - Use `uv pip install "transformers>=5.0.0"` before serving GLM-V checkpoints. ## Client Usage ```python from openai import OpenAI client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1") resp = client.chat.completions.create( model="zai-org/GLM-4.6V-FP8", messages=[{ "role": "user", "content": [ {"type": "image_url", "image_url": {"url": "https://example.com/image.png"}}, {"type": "text", "text": "Describe the image."} ] }], max_tokens=512, ) print(resp.choices[0].message.content) ``` ## Benchmarking ```bash vllm bench serve \ --backend openai-chat \ --endpoint /v1/chat/completions \ --model zai-org/GLM-4.6V-FP8 \ --dataset-name hf \ --dataset-path lmarena-ai/VisionArena-Chat \ --num-prompts 1000 \ --request-rate 20 ``` ## Troubleshooting - **Vision encoder overhead:** Prefer `--mm-encoder-tp-mode data` over TP for the encoder. - **Long-context memory:** At 128K context, tune `--max-num-batched-tokens` and `--gpu-memory-utilization` to prevent OOM. ## References - [Model card](https://huggingface.co/zai-org/GLM-4.6V) - [vLLM multimodal inputs guide](https://docs.vllm.ai/en/latest/features/multimodal_inputs.html) --- # zai-org/GLM-ASR-Nano-2512 on vLLM Source: https://recipes.vllm.ai/zai-org/GLM-ASR-Nano-2512 Provider: GLM (Z-AI) Released: 2025-12-09T09:07:41.000Z Updated: 2026-04-17 Architecture: dense Parameters: 2.3B Context length: 8,192 Min vLLM: 0.14.1 Tasks: multimodal Summary: Open-source speech recognition model (~2B) with strong dialect support (Cantonese and others) and robust low-volume speech transcription ## Variants - default: precision=bf16, vram_minimum_gb=11 Full precision BF16 — single-GPU deployment ## Guide ## Overview GLM-ASR-Nano-2512 is an open-source automatic speech recognition model with 1.5B active parameters (2B total). It outperforms OpenAI Whisper V3 on multiple benchmarks while remaining compact enough for single-GPU deployment. ### Key Capabilities - **Dialect support:** Beyond standard Mandarin and English, strong on Cantonese (粤语) and other Chinese dialects. - **Low-volume speech:** Specifically trained for "whisper/quiet speech" scenarios. - **SOTA accuracy:** Lowest average error rate (4.10) among comparable open-source models, strong on Wenet Meeting, Aishell-1, and similar Chinese benchmarks. ## Prerequisites - **vLLM version:** >= 0.14.1 (with `[audio]` extras) - **Transformers:** install from source for latest ### Install Dependencies ```bash uv venv source .venv/bin/activate uv pip install git+https://github.com/huggingface/transformers.git uv pip install -U "vllm[audio]" --torch-backend auto ``` ## Launching the Server ```bash vllm serve zai-org/GLM-ASR-Nano-2512 ``` ## Client Usage ### OpenAI SDK (Audio URL) ```python import base64 import httpx from openai import OpenAI client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY") audio_url = "https://github.com/zai-org/GLM-ASR/raw/main/examples/example_en.wav" audio_data = base64.b64encode(httpx.get(audio_url).content).decode("utf-8") response = client.chat.completions.create( model="zai-org/GLM-ASR-Nano-2512", messages=[{ "role": "user", "content": [{ "type": "input_audio", "input_audio": {"data": audio_data, "format": "wav"} }] }], max_tokens=500, ) print(response.choices[0].message.content) ``` ### Transcribe Endpoint ```python import httpx from openai import OpenAI client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY") audio_file = httpx.get("https://github.com/zai-org/GLM-ASR/raw/main/examples/example_en.wav").content response = client.audio.transcriptions.create( model="zai-org/GLM-ASR-Nano-2512", file=("audio.wav", audio_file), ) print(response.text) ``` ### cURL (Transcribe) ```bash curl http://localhost:8000/v1/audio/transcriptions \ -H "Authorization: Bearer EMPTY" \ -F "model=zai-org/GLM-ASR-Nano-2512" \ -F "file=@your_audio.wav" ``` ### Local Audio File (chat API) ```python import base64 from openai import OpenAI client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY") with open("your_audio.mp3", "rb") as f: audio_data = base64.b64encode(f.read()).decode("utf-8") response = client.chat.completions.create( model="zai-org/GLM-ASR-Nano-2512", messages=[{ "role": "user", "content": [{"type": "input_audio", "input_audio": {"data": audio_data, "format": "mp3"}}] }], max_tokens=500, ) print(response.choices[0].message.content) ``` ## Troubleshooting - **Transformers version:** Requires `transformers >= 5.0.0` for best compatibility. - **Audio formats:** Supports wav, mp3, flac, and other common formats. ## References - [Model card](https://huggingface.co/zai-org/GLM-ASR-Nano-2512) - [GitHub repo](https://github.com/zai-org/GLM-ASR) --- # zai-org/GLM-Image on vLLM Source: https://recipes.vllm.ai/zai-org/GLM-Image Provider: GLM (Z-AI) Released: 2026-01-08T09:39:56.000Z Updated: 2026-04-17 Architecture: dense Parameters: 16B Context length: 4,096 Min vLLM: 0.11.0 Tasks: omni Verified hardware: H100 Summary: Hybrid autoregressive + diffusion image generation model — text-to-image and image-to-image with strong text rendering and knowledge-intensive generation ## Variants - default: precision=bf16, vram_minimum_gb=38 Single-GPU deployment (~33 GB for model weights, plus activation headroom) ## Recommended base flags ```bash vllm serve 'zai-org/GLM-Image' \ --trust-remote-code ``` ## Guide ## Overview GLM-Image is an image generation model with a hybrid architecture: - **Autoregressive Generator (9B):** initialized from GLM-4-9B-0414 with an expanded vocabulary for visual tokens. Produces a compact encoding (~256 tokens), then expands to 1K–4K tokens corresponding to 1K–2K resolution images. - **Diffusion Decoder (7B):** single-stream DiT that decodes latents into pixels. Includes a Glyph Encoder text module for accurate in-image text rendering. Served via vLLM-Omni for OpenAI-compatible online inference. ### Key Capabilities - Text-to-image and image-to-image (editing, style transfer, identity-preserving) - Exceptional text rendering inside generated images - Strong knowledge-intensive generation ## Prerequisites - **vLLM version:** latest (with `vllm-omni` extension) - **Transformers:** >= 5.0.0 (use source install for latest) - **Hardware:** single H100-class GPU (approx. 33 GB for weights) ### Install Dependencies ```bash uv venv --python 3.12 --seed source .venv/bin/activate uv pip install -U vllm --torch-backend auto uv pip install vllm-omni pip install git+https://github.com/huggingface/transformers.git pip install git+https://github.com/huggingface/diffusers.git ``` ## Online Serving ```bash vllm serve zai-org/GLM-Image --omni ``` ## Client Usage ### OpenAI SDK (Text-to-Image) ```python import base64 from openai import OpenAI client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY") response = client.chat.completions.create( model="zai-org/GLM-Image", messages=[{"role": "user", "content": "A beautiful landscape painting with mountains and a lake at sunset"}], ) image_url = response.choices[0].message.content[0].image_url.url image_data = base64.b64decode(image_url.split(",")[1]) with open("output.png", "wb") as f: f.write(image_data) ``` ### cURL (Text-to-Image) ```bash curl -s http://localhost:8000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "messages": [{"role": "user", "content": "A beautiful landscape painting"}] }' | jq -r '.choices[0].message.content[0].image_url.url' | cut -d',' -f2- | base64 -d > output.png ``` ### Image-to-Image ```python import base64 from openai import OpenAI client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY") with open("input.png", "rb") as f: image_base64 = base64.b64encode(f.read()).decode("utf-8") response = client.chat.completions.create( model="zai-org/GLM-Image", messages=[{ "role": "user", "content": [ {"type": "image_url", "image_url": {"url": f"data:image/png;base64,{image_base64}"}}, {"type": "text", "text": "Replace the background with a sunset beach scene"} ] }], ) image_url = response.choices[0].message.content[0].image_url.url image_data = base64.b64decode(image_url.split(",")[1]) with open("output.png", "wb") as f: f.write(image_data) ``` ## Offline Inference ```bash # Text to Image cd examples/offline_inference/text_to_image python3 text_to_image.py --model zai-org/GLM-Image --output t2i_output.png # Image to Image cd examples/offline_inference/image_to_image wget https://vllm-public-assets.s3.us-west-2.amazonaws.com/omni-assets/qwen-bear.png python3 image_to_image.py --model zai-org/GLM-Image --image qwen-bear.png --output i2i_output.png ``` ## Troubleshooting - **Resolution errors:** Target image dimensions must be divisible by 32. - **Text rendering:** Wrap text that should appear in the image with quotation marks in the prompt. - **Output stability:** Default `temperature=0.9`, `top_p=0.75`. Higher temperature gives more diverse outputs but may reduce stability. - **Transformers version:** Requires `transformers >= 5.0.0`. ## References - [Model card](https://huggingface.co/zai-org/GLM-Image) - [Technical blog](https://z.ai/blog/glm-image) - [GitHub repo](https://github.com/zai-org/GLM-Image) --- # zai-org/GLM-OCR on vLLM Source: https://recipes.vllm.ai/zai-org/GLM-OCR Provider: GLM (Z-AI) Released: 2026-01-30T04:24:21.000Z Updated: 2026-04-17 Architecture: dense Parameters: 0.9B Context length: 131,072 Min vLLM: 0.16.0 Tasks: multimodal Summary: GLM-OCR image-to-text model with built-in MTP speculative decoding for high-throughput OCR serving ## Variants - default: precision=bf16, vram_minimum_gb=2 Full precision BF16 — single-GPU deployment ## Guide ## Overview GLM-OCR is a vision-language model for end-to-end OCR. It includes built-in Multi-Token Prediction (MTP) layers enabling speculative decoding for higher throughput generation. ## Prerequisites - **vLLM version:** nightly recommended (or latest stable with MTP support) - **Transformers:** >= 5.0.0 (install from source for latest) ### Install vLLM ```bash uv venv source .venv/bin/activate uv pip install -U vllm --torch-backend auto ``` Or nightly: ```bash uv pip install -U vllm --pre --extra-index-url https://wheels.vllm.ai/nightly uv pip install git+https://github.com/huggingface/transformers.git ``` ## Launching the Server ### With MTP Speculative Decoding ```bash vllm serve zai-org/GLM-OCR \ --speculative-config.method mtp \ --speculative-config.num_speculative_tokens 1 ``` ## Client Usage ### OpenAI SDK ```python from openai import OpenAI client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1", timeout=3600) messages = [{ "role": "user", "content": [ {"type": "image_url", "image_url": {"url": "https://ofasys-multimodal-wlcb-3-toshanghai.oss-accelerate.aliyuncs.com/wpf272043/keepme/image/receipt.png"}}, {"type": "text", "text": "Text Recognition:"} ] }] response = client.chat.completions.create( model="zai-org/GLM-OCR", messages=messages, max_tokens=2048, temperature=0.0, ) print(response.choices[0].message.content) ``` ### cURL ```bash curl -s http://localhost:8000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "zai-org/GLM-OCR", "messages": [{ "role": "user", "content": [ {"type": "image_url", "image_url": {"url": "https://example.com/receipt.png"}}, {"type": "text", "text": "Text Recognition:"} ] }], "max_tokens": 2048, "temperature": 0.0 }' ``` ## Troubleshooting - **Greedy sampling recommended:** Use `temperature=0.0` for optimal OCR accuracy. - **Transformers version:** Requires `transformers >= 5.0.0`. ## References - [Model card](https://huggingface.co/zai-org/GLM-OCR) --- # zai-org/Glyph on vLLM Source: https://recipes.vllm.ai/zai-org/Glyph Provider: GLM (Z-AI) Released: 2025-10-25T06:19:07.000Z Updated: 2026-04-17 Architecture: dense Parameters: 10B Context length: 131,072 Min vLLM: 0.11.0 Tasks: multimodal Verified hardware: H100, MI300X, MI325X Summary: Visual-text compression framework that renders long text into images and processes them with a reasoning VLM, scaling effective context length ## Variants - default: precision=bf16, vram_minimum_gb=24 Full precision BF16 — single-GPU deployment on 1xH100 ## Recommended base flags ```bash vllm serve 'zai-org/Glyph' \ --no-enable-prefix-caching \ --mm-processor-cache-gb \ 0 \ --limit-mm-per-prompt.video \ 0 ``` ## Guide ## Overview [Glyph](https://github.com/thu-coai/Glyph) is a framework from Zhipu AI for scaling context length via visual-text compression. It renders long textual sequences into images and processes them with a vision-language model. This recipe covers the vLLM deployment of the `zai-org/Glyph` VLM as a component in that framework. Glyph is a reasoning multimodal model, so `--reasoning-parser glm45` is recommended to parse reasoning traces from outputs. ## Prerequisites - **vLLM version:** latest stable - **Hardware:** 1x H100 or 1x MI300X/MI325X ### Install vLLM (NVIDIA) ```bash uv venv source .venv/bin/activate uv pip install -U vllm --torch-backend auto ``` ### Install vLLM (AMD ROCm) ```bash uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm ``` ROCm wheel requires Python 3.12, ROCm 7.0, glibc >= 2.35. ## Launching the Server ### Single H100 GPU ```bash vllm serve zai-org/Glyph \ --no-enable-prefix-caching \ --mm-processor-cache-gb 0 \ --reasoning-parser glm45 \ --limit-mm-per-prompt.video 0 ``` ### Single MI300X / MI325X ```bash VLLM_ROCM_USE_AITER=1 \ SAFETENSORS_FAST_GPU=1 \ vllm serve zai-org/Glyph \ --no-enable-prefix-caching \ --mm-processor-cache-gb 0 \ --reasoning-parser glm45 \ --limit-mm-per-prompt.video 0 ``` ### Configuration Tips - `--no-enable-prefix-caching` and `--mm-processor-cache-gb 0` are recommended for OCR-like workloads where image reuse is uncommon; they avoid unnecessary hashing and caching overhead. - Adjust `--max-num-batched-tokens` for throughput according to your hardware. ## Benchmarking ```bash vllm bench serve \ --model zai-org/Glyph \ --dataset-name random \ --random-input-len 8192 \ --random-output-len 512 \ --request-rate 10000 \ --num-prompts 16 \ --ignore-eos ``` ## Troubleshooting - **Reasoning traces:** Use `--reasoning-parser glm45` to extract reasoning content. - **Slow first inference:** Disabling prefix caching and multimodal processor caching is intentional for Glyph's use case and trades off first-request latency for predictable throughput. ## References - [Model card](https://huggingface.co/zai-org/Glyph) - [Glyph framework on GitHub](https://github.com/thu-coai/Glyph) - [vLLM deployment section of Glyph docs](https://github.com/thu-coai/Glyph?tab=readme-ov-file#model-deployment-vllm-acceleration)