{ "hf_id": "moonshotai/Kimi-K3.1-Pro", "meta": { "title": "Kimi-K3.1-Pro", "slug": "kimi-k3.1-pro", "provider": "Moonshot AI", "description": "2.8T-parameter native multimodal MoE with Kimi Delta Attention, sparse MLA, and a 1M-token context window", "date_added": "2026-11-02", "date_updated": "2026-11-05", "tasks": [ "multimodal", "text" ], "performance_headline": "TP8 on two GB300 trays or one B300 node, or one MI355X node, for the MXFP4 checkpoint", "related_recipes": [ "moonshotai/Kimi-K3" ], "default_hardware": "gb300", "hardware": { "gb300": "verified", "vr200": "supported", "810e": "verified", "m890": "verified" } }, "recommended_command": { "hardware": "gb300", "strategy": "multi_node_tp", "variant": "default", "node_count": 2, "deploy_type": "multi_node", "env": { "GLOO_SOCKET_IFNAME": "$IFACE_NAME", "NCCL_SOCKET_IFNAME": "$IFACE_NAME", "VLLM_ENGINE_READY_TIMEOUT_S": "3600" }, "docker_image": "vllm/vllm-openai:kimi-k31", "head_command": "vllm serve moonshotai/Kimi-K3.1-Pro \\\n --trust-remote-code \\\n --tensor-parallel-size 8 \\\n --nnodes 2 \\\n --node-rank 0 \\\n --master-addr $HEAD_IP \\\n --load-format fastsafetensors \\\n --enable-auto-tool-choice \\\n --tool-call-parser kimi_k3 \\\n --reasoning-parser kimi_k3", "worker_command": "vllm serve moonshotai/Kimi-K3.1-Pro \\\n --trust-remote-code \\\n --tensor-parallel-size 8 \\\n --nnodes 2 \\\n --node-rank 1 \\\n --master-addr $HEAD_IP \\\n --headless \\\n --load-format fastsafetensors \\\n --enable-auto-tool-choice \\\n --tool-call-parser kimi_k3 \\\n --reasoning-parser kimi_k3", "worker_commands": [ "vllm serve moonshotai/Kimi-K3.1-Pro \\\n --trust-remote-code \\\n --tensor-parallel-size 8 \\\n --nnodes 2 \\\n --node-rank 1 \\\n --master-addr $HEAD_IP \\\n --headless \\\n --load-format fastsafetensors \\\n --enable-auto-tool-choice \\\n --tool-call-parser kimi_k3 \\\n --reasoning-parser kimi_k3" ], "head_argv": [ "vllm", "serve", "moonshotai/Kimi-K3.1-Pro", "--trust-remote-code", "--tensor-parallel-size", "8", "--nnodes", "2", "--node-rank", "0", "--master-addr", "$HEAD_IP", "--load-format", "fastsafetensors", "--enable-auto-tool-choice", "--tool-call-parser", "kimi_k3", "--reasoning-parser", "kimi_k3" ], "worker_argv": [ "vllm", "serve", "moonshotai/Kimi-K3.1-Pro", "--trust-remote-code", "--tensor-parallel-size", "8", "--nnodes", "2", "--node-rank", "1", "--master-addr", "$HEAD_IP", "--headless", "--load-format", "fastsafetensors", "--enable-auto-tool-choice", "--tool-call-parser", "kimi_k3", "--reasoning-parser", "kimi_k3" ], "worker_argvs": [ [ "vllm", "serve", "moonshotai/Kimi-K3.1-Pro", "--trust-remote-code", "--tensor-parallel-size", "8", "--nnodes", "2", "--node-rank", "1", "--master-addr", "$HEAD_IP", "--headless", "--load-format", "fastsafetensors", "--enable-auto-tool-choice", "--tool-call-parser", "kimi_k3", "--reasoning-parser", "kimi_k3" ] ], "head_docker_command": "docker run --gpus all \\\n --privileged --ipc=host -p 8000:8000 \\\n -v ~/.cache/huggingface:/root/.cache/huggingface \\\n -e GLOO_SOCKET_IFNAME=$IFACE_NAME \\\n -e NCCL_SOCKET_IFNAME=$IFACE_NAME \\\n -e VLLM_ENGINE_READY_TIMEOUT_S=3600 \\\n vllm/vllm-openai:kimi-k31 moonshotai/Kimi-K3.1-Pro \\\n --trust-remote-code \\\n --tensor-parallel-size 8 \\\n --nnodes 2 \\\n --node-rank 0 \\\n --master-addr $HEAD_IP \\\n --load-format fastsafetensors \\\n --enable-auto-tool-choice \\\n --tool-call-parser kimi_k3 \\\n --reasoning-parser kimi_k3", "worker_docker_command": "docker run --gpus all \\\n --privileged --ipc=host -p 8000:8000 \\\n -v ~/.cache/huggingface:/root/.cache/huggingface \\\n -e GLOO_SOCKET_IFNAME=$IFACE_NAME \\\n -e NCCL_SOCKET_IFNAME=$IFACE_NAME \\\n -e VLLM_ENGINE_READY_TIMEOUT_S=3600 \\\n vllm/vllm-openai:kimi-k31 moonshotai/Kimi-K3.1-Pro \\\n --trust-remote-code \\\n --tensor-parallel-size 8 \\\n --nnodes 2 \\\n --node-rank 1 \\\n --master-addr $HEAD_IP \\\n --headless \\\n --load-format fastsafetensors \\\n --enable-auto-tool-choice \\\n --tool-call-parser kimi_k3 \\\n --reasoning-parser kimi_k3", "worker_docker_commands": [ "docker run --gpus all \\\n --privileged --ipc=host -p 8000:8000 \\\n -v ~/.cache/huggingface:/root/.cache/huggingface \\\n -e GLOO_SOCKET_IFNAME=$IFACE_NAME \\\n -e NCCL_SOCKET_IFNAME=$IFACE_NAME \\\n -e VLLM_ENGINE_READY_TIMEOUT_S=3600 \\\n vllm/vllm-openai:kimi-k31 moonshotai/Kimi-K3.1-Pro \\\n --trust-remote-code \\\n --tensor-parallel-size 8 \\\n --nnodes 2 \\\n --node-rank 1 \\\n --master-addr $HEAD_IP \\\n --headless \\\n --load-format fastsafetensors \\\n --enable-auto-tool-choice \\\n --tool-call-parser kimi_k3 \\\n --reasoning-parser kimi_k3" ], "head_docker_argv": [ "docker", "run", "--gpus", "all", "--privileged", "--ipc=host", "-p", "8000:8000", "-v", "~/.cache/huggingface:/root/.cache/huggingface", "-e", "GLOO_SOCKET_IFNAME=$IFACE_NAME", "-e", "NCCL_SOCKET_IFNAME=$IFACE_NAME", "-e", "VLLM_ENGINE_READY_TIMEOUT_S=3600", "vllm/vllm-openai:kimi-k31", "moonshotai/Kimi-K3.1-Pro", "--trust-remote-code", "--tensor-parallel-size", "8", "--nnodes", "2", "--node-rank", "0", "--master-addr", "$HEAD_IP", "--load-format", "fastsafetensors", "--enable-auto-tool-choice", "--tool-call-parser", "kimi_k3", "--reasoning-parser", "kimi_k3" ], "worker_docker_argv": [ "docker", "run", "--gpus", "all", "--privileged", "--ipc=host", "-p", "8000:8000", "-v", "~/.cache/huggingface:/root/.cache/huggingface", "-e", "GLOO_SOCKET_IFNAME=$IFACE_NAME", "-e", "NCCL_SOCKET_IFNAME=$IFACE_NAME", "-e", "VLLM_ENGINE_READY_TIMEOUT_S=3600", "vllm/vllm-openai:kimi-k31", "moonshotai/Kimi-K3.1-Pro", "--trust-remote-code", "--tensor-parallel-size", "8", "--nnodes", "2", "--node-rank", "1", "--master-addr", "$HEAD_IP", "--headless", "--load-format", "fastsafetensors", "--enable-auto-tool-choice", "--tool-call-parser", "kimi_k3", "--reasoning-parser", "kimi_k3" ], "worker_docker_argvs": [ [ "docker", "run", "--gpus", "all", "--privileged", "--ipc=host", "-p", "8000:8000", "-v", "~/.cache/huggingface:/root/.cache/huggingface", "-e", "GLOO_SOCKET_IFNAME=$IFACE_NAME", "-e", "NCCL_SOCKET_IFNAME=$IFACE_NAME", "-e", "VLLM_ENGINE_READY_TIMEOUT_S=3600", "vllm/vllm-openai:kimi-k31", "moonshotai/Kimi-K3.1-Pro", "--trust-remote-code", "--tensor-parallel-size", "8", "--nnodes", "2", "--node-rank", "1", "--master-addr", "$HEAD_IP", "--headless", "--load-format", "fastsafetensors", "--enable-auto-tool-choice", "--tool-call-parser", "kimi_k3", "--reasoning-parser", "kimi_k3" ] ], "strategy_spec": { "name": "multi_node_tp", "deploy_type": "multi_node", "display_name": "Multi-Node Tensor Parallel", "short_label": "Multi-node TP", "orientation": "latency", "description": "Multi-node TP. Splits the model across GPUs spanning multiple nodes. TP is set to total GPU count across all nodes. Head node serves requests, worker nodes run with --headless. Requires InfiniBand interconnect.\n", "hardware_match": { "min_gpus": 2, "multi_node": true }, "parallel_flag": "--tensor-parallel-size", "env": { "GLOO_SOCKET_IFNAME": "$IFACE_NAME", "NCCL_SOCKET_IFNAME": "$IFACE_NAME" } }, "hardware_profile": { "brand": "NVIDIA", "generation": "blackwell", "compute_arch": "sm103", "display_name": "GB300 NVL4", "description": "NVIDIA GB300 Grace-Blackwell Ultra compute tray · 288 GB HBM3e/GPU · 4 GPUs (NVL72 tray unit)", "gpu_count": 4, "vram_gb": 1152, "multi_node": false }, "alternatives": { "pd_cluster": "/moonshotai/Kimi-K3.1-Pro/strategies/pd_cluster.json", "kv_store_centralized_mooncake": "/moonshotai/Kimi-K3.1-Pro/strategies/kv_store_centralized_mooncake.json", "kv_store_distributed_mooncake": "/moonshotai/Kimi-K3.1-Pro/strategies/kv_store_distributed_mooncake.json" }, "by_hardware": { "gb300": "/moonshotai/Kimi-K3.1-Pro/hw/gb300.json", "b300": "/moonshotai/Kimi-K3.1-Pro/hw/b300.json", "vr200": "/moonshotai/Kimi-K3.1-Pro/hw/vr200.json", "mi355x": "/moonshotai/Kimi-K3.1-Pro/hw/mi355x.json", "810e": "/moonshotai/Kimi-K3.1-Pro/hw/810e.json", "m890": "/moonshotai/Kimi-K3.1-Pro/hw/m890.json" } }, "model": { "model_id": "moonshotai/Kimi-K3.1-Pro", "min_vllm_version": "0.34.0", "nightly_required": true, "docker_image": { "nvidia": "vllm/vllm-openai:kimi-k31", "amd": "vllm/vllm-openai-rocm:kimi-k31" }, "architecture": "moe", "parameter_count": "2.8T", "active_parameters": "16 experts/token + shared (of 896 routed)", "context_length": 1048576, "base_args": [ "--trust-remote-code" ], "install": { "docker": { "command": "docker pull vllm/vllm-openai:kimi-k31" } } }, "features": { "tool_calling": { "description": "Enable automatic tool choice with the Kimi K3 tool-call parser.", "args": [ "--enable-auto-tool-choice", "--tool-call-parser", "kimi_k3" ] }, "reasoning": { "description": "Parse Kimi K3 reasoning output separately from the final answer.", "args": [ "--reasoning-parser", "kimi_k3" ] }, "spec_decoding": { "description": "Use DSpark speculative decoding with the Kimi-K3.1 draft.", "args": [ "--speculative-config", "{\"model\":\"Inferact/Kimi-K3.1-Pro-DSpark-preview\",\"num_speculative_tokens\":5,\"method\":\"dspark\",\"draft_sample_method\":\"probabilistic\"}" ], "hardware_overrides": { "mi355x": { "args": [ "--kv-cache-dtype", "fp8", "--speculative-config", "{\"model\":\"Inferact/Kimi-K3.1-Pro-DSpark-preview\",\"num_speculative_tokens\":5,\"method\":\"dspark\",\"attention_backend\":\"ROCM_AITER_MLA\",\"kv_cache_dtype\":\"fp8\",\"draft_sample_method\":\"probabilistic\"}" ] } } }, "text_only": { "description": "Skip the vision encoder for text-only workloads; frees its memory for KV cache.", "args": [ "--language-model-only" ] }, "low_latency_decode": { "label": "Low-latency decode", "description": "Run decode steps on the fused CuTe DSL decode path (latency_specialized), which covers the sparse attention layers. Fastest at low concurrency; needs FULL decode CUDA graphs.", "strategies": [ "single_node_tp", "multi_node_tp" ], "hardware": { "ppu": "unsupported" }, "args": [ "--additional-config", "{\"decode_backend\":\"latency_specialized\"}" ] }, "nvfp4_kv_cache": { "label": "NVFP4 KV cache", "description": "Store the MLA latent in NVFP4 instead of FP8 for about 1.5x the KV cache tokens; the indexer keys stay FP8.", "hardware": { "ppu": "unsupported" }, "args": [ "--kv-cache-dtype", "nvfp4_ds_mla" ] } }, "opt_in_features": [ "spec_decoding", "text_only", "low_latency_decode", "nvfp4_kv_cache" ], "variants": { "default": { "precision": "mxfp4", "vram_minimum_gb": 1874, "supported_hardware": [ "b300", "gb300", "vr200", "mi355x", "810e", "m890" ], "description": "MXFP4 routed-expert weights (~1.56 TB on disk); attention, shared experts and the vision tower stay in BF16.", "hardware_overrides": { "810e": { "docker_image": "egslingjun-registry.cn-wulanchabu.cr.aliyuncs.com/egslingjun-tmp/inference-xpu-pytorch:kimi-k3.1-pro-vllm0.32.0-cuda13.0-pytorch2.13.0-20261103", "docker_serve_command": "vllm serve", "install": { "pip": false } }, "m890": { "docker_image": "egslingjun-registry.cn-wulanchabu.cr.aliyuncs.com/egslingjun-tmp/inference-xpu-pytorch:kimi-k3.1-pro-vllm0.32.0-cuda13.0-pytorch2.13.0-20261103", "docker_serve_command": "vllm serve", "install": { "pip": false } }, "vr200": { "docker_image": "vllm/vllm-openai:kimi-k31-cu134", "install": { "pip": false } } } } }, "strategy_hardware": { "multi_node_tep": { "810e": "supported", "m890": "supported" }, "single_node_tp": { "810e": "unsupported", "m890": "unsupported" }, "single_node_tep": { "810e": "unsupported", "m890": "unsupported" }, "multi_node_tp": { "810e": "unsupported", "m890": "unsupported" }, "pd_cluster": { "810e": "unsupported", "m890": "unsupported" } }, "strategy_min_gpus": { "single_node_tp": 8, "single_node_tep": 8, "multi_node_tp": 8, "810e": 32, "m890": 32 }, "hardware_overrides": { "blackwell": { "extra_args": [ "--load-format", "fastsafetensors" ], "extra_env": { "VLLM_ENGINE_READY_TIMEOUT_S": "3600" } }, "rubin": { "extra_args": [ "--load-format", "fastsafetensors" ], "extra_env": { "VLLM_ENGINE_READY_TIMEOUT_S": "3600" } }, "amd": { "extra_env": { "VLLM_ROCM_USE_AITER": "1", "SAFETENSORS_FAST_GPU": "1", "VLLM_ROCM_USE_AITER_MOE_SITUV2": "1", "VLLM_USE_BREAKABLE_CUDAGRAPH": "1", "VLLM_ENGINE_READY_TIMEOUT_S": "7200" } } }, "presets": { "mi355x_agentx": { "label": "AgentX TP8", "hardware": "mi355x", "strategy": "single_node_tp", "workload": "Agentic (InferenceX AgentX)", "description": "Concurrency 48 on one MI355X node: TP8 with DCP8, the K3.1 DSpark draft, host KV offload and the reduced-precision knobs of the best-performance band.", "select": { "frontend": "python", "features": { "spec_decoding": true, "text_only": true }, "kv_offload": "simple" }, "args": [ "--load-format", "fastsafetensors", "--max-model-len", "1048576", "--enable-prefix-caching", "--kv-cache-dtype", "fp8", "--stream-interval", "10", "--decode-context-parallel-size", "8", "--dcp-comm-backend", "a2a", "--cp-kv-cache-interleave-size", "1", "--prefix-match-unit", "64", "--max-num-seqs", "48", "--max-num-batched-tokens", "10240", "--gpu-memory-utilization", "0.92", "--attention-config", "{\"mla_prefill_backend\":\"ROCM_AITER_FA\",\"indexer_kv_dtype\":\"mxfp4\"}", "--additional-config", "{\"kda_prefill_backend\":\"fused\"}", "--quantization-config", "{\"targets\":{\"*in_proj_qkvgfab\":\"fp8_per_channel\"}}", "--compilation-config", "{\"mode\":3,\"cudagraph_mode\":\"FULL_AND_PIECEWISE\",\"max_cudagraph_capture_size\":10240,\"custom_ops\":[\"+fused_rms_norm_gated\"],\"cudagraph_capture_sizes\":[6,12,18,24,30,36,42,48,54,60,66,72,78,84,90,96,102,108,114,120,126,132,138,144,150,156,162,168,174,180,186,192,198,204,210,216,222,228,234,240,246,252,258,264,270,276,282,288,320,352,384,416,448,480,512,640,768,896,1024,1280,1536,1792,2048,2560,3072,3584,4096,4608,5120,6144,6400,7168,8192,8704,10240]}" ], "env": { "VLLM_USE_V2_MODEL_RUNNER": "1", "VLLM_FASTSAFETENSORS_QUEUE_SIZE": "-1", "VLLM_ROCM_AITER_MLA_ASM_PADDING": "asm", "VLLM_ROCM_AITER_MLA_DCP_VERIFY": "asm", "VLLM_ROCM_DCP_AITER_ALLGATHER": "1", "VLLM_ROCM_USE_AITER_FLYDSL_SPARSE_MLA": "1", "AITER_FLYDSL_SPARSE_MLA": "1", "VLLM_ROCM_DSA_DCP_GLOBAL_KEY_PREFILL": "1", "VLLM_ROCM_DSA_DCP_ROWSHARD_PREFILL": "1", "VLLM_USE_DIRECT_DCP_A2A": "0", "VLLM_USE_DIRECT_DCP_Q_GATHER": "0", "VLLM_USE_DIRECT_DCP_KV_GATHER": "0", "AITER_FLYDSL_STAGE2_FP8": "1", "VLLM_ROCM_QUICK_REDUCE_QUANTIZATION": "INT4", "VLLM_ROCM_QUICK_REDUCE_MIN_SIZE_BYTES_MB": "16", "PYTHONHASHSEED": "42" }, "json_args": { "--kv-transfer-config": { "kv_connector_extra_config": { "cpu_bytes_to_use_per_rank": 224875000000 } } }, "runbook": "The single-node MI355X TP8 AgentX configuration at concurrency 48, from the K3.1 best-performance band. It is a BF16 serving config plus three reduced-precision knobs. It serves text with tool-call and reasoning parsers, using DCP8/A2A.\n\n**Serving config.**\n- **`--max-num-seqs 48` and capture sizes 6 to 10240.** At concurrency 48, decode batches are at most 48 requests × 6 rows (1 + 5 draft tokens), so the sizes 6 to 288 in steps of 6 cover every decode step exactly. The list then continues from 320 up to 10240, the value of `--max-num-batched-tokens`, to cover the prefill steps.\n- **`VLLM_USE_BREAKABLE_CUDAGRAPH=1`.** K3.1 has no `torch.compile` submodule, so vLLM resolves `FULL_AND_PIECEWISE` to `FULL_DECODE_ONLY` and captures decode only. Breakable graphs are what let it capture the prefill sizes above 288 as well, so the AMD profile turns them on. They cost graph memory, which comes out of the KV cache: with the old 2-to-576 list (`--max-num-seqs 96`) they took 63.1 GiB and the server did not start. Keep the capture list to the sizes the workload really uses.\n- **`--max-num-batched-tokens 10240`.** With prefix caching, the hybrid KV manager ends every non-final prefill chunk on a block boundary. In this preset (with the dense MLA DSpark draft sharing the block row) K3.1's packed DSA block is 1152 tokens, so after the decode rows a chunk floors to 9216 tokens (6912 at 8192).\n- **`--gpu-memory-utilization 0.92`** gives more KV. Keep at least about 6 GiB free per GPU.\n- **The indexer key cache is MXFP4** (`indexer_kv_dtype`). AITER needs padded FP4 indexer pages for it. Stock AITER asserts on the page stride at startup.\n- `ROCM_AITER_FA` is named as the MLA prefill backend, which ROCm already picks first. `VLLM_ROCM_DCP_AITER_ALLGATHER=1`, the FlyDSL sparse-MLA kernels and the DSA DCP prefill paths are on. `kda_prefill_backend: fused` prefills the KDA layers with the fused gfx950 kernel.\n- **Spec decoding** (the Spec decoding feature, on in this preset) uses the K3.1 DSpark draft (5 layers, not the K3 draft) with 5 tokens, `ROCM_AITER_MLA` attention, FP8 KV, probabilistic drafting and vLLM's default (standard) rejection sampler. The draft is [`Inferact/Kimi-K3.1-Pro-DSpark-preview`](https://huggingface.co/Inferact/Kimi-K3.1-Pro-DSpark-preview).\n- The **Simple** KV-offload option gets 224,875,000,000 bytes of host DRAM per rank. Size `cpu_bytes_to_use_per_rank` to your host.\n\n**Reduced precision (best-performance band).** The serving config keeps linear layers, MoE combine and all-reduce in BF16. This preset adds:\n- `--quantization-config '{\"targets\":{\"*in_proj_qkvgfab\":\"fp8_per_channel\"}}'`: only the KDA input projections are quantized online to FP8 per channel.\n- `AITER_FLYDSL_STAGE2_FP8=1`: the FlyDSL MoE down projection writes its per-route outputs as MXFP8 before they are summed.\n- `VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4` with `VLLM_ROCM_QUICK_REDUCE_MIN_SIZE_BYTES_MB=16`: all-reduces of 16 MiB and more use the lossy INT4 codec. To serve with BF16 all-reduce, unset both.\n\nDrop all three for BF16 precision.\n\n**Environment outside the builder.** The lane also sets `AITER_BF16_FP8_MOE_BOUND=0` (the SiTUv2 path does not read it), `VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=1200` for long agentic prefills, `PYTHONNOUSERSITE=1`, `HSA_NO_SCRATCH_RECLAIM=0`, and `HSA_SCRATCH_SINGLE_LIMIT_ASYNC=25769803776` (24 GiB), which caps a single asynchronous scratch allocation.\n\n### Throughput measurement only (synthetic acceptance)\n\nThe AgentX runs replace real verification with synthetic acceptance at the golden acceptance length for 5 draft tokens, 3.62. Only `--speculative-config` changes:\n\n```bash\n--speculative-config '{\"model\":\"Inferact/Kimi-K3.1-Pro-DSpark-preview\",\"num_speculative_tokens\":5,\"method\":\"dspark\",\"attention_backend\":\"ROCM_AITER_MLA\",\"kv_cache_dtype\":\"fp8\",\"draft_sample_method\":\"probabilistic\",\"rejection_sample_method\":\"synthetic\",\"synthetic_acceptance_length\":3.62}'\n```\n\nSynthetic acceptance skips real verification. Never use it to serve users or to evaluate accuracy.\n", "json": "/moonshotai/Kimi-K3.1-Pro/presets/mi355x_agentx.json" } }, "decode_context_parallel": { "sizes": { "b300": [ 8 ], "gb300": [ 8 ] }, "match_tp": true }, "guide": "## Overview\n\nKimi-K3.1 keeps the Kimi-K3 layout (Kimi Delta Attention, 896 routed experts with 16 active\nper token, a latent MoE projection and a vision encoder) and adds sparse\nattention (DSA, top-2048 indexer) to 24 of its MLA layers. The checkpoint ships MXFP4\nrouted-expert weights; everything else is BF16.\n\nThe baseline is the simplest working setup on NVIDIA: one tensor-parallel group of 8 GPUs,\neither two GB300 trays or one B300 node. Throughput was not tuned.\n\nVR200 (preview) takes the same TP8 layout on two trays, in the `vllm/vllm-openai:kimi-k31-cu134`\nimage (CUDA 13.4). It uses the Blackwell flags and hasn't been tuned or verified on Rubin.\n\nOn MI355X, one node takes the same TP8 layout with the AMD profile's AITER environment, in the\nday-0 `vllm/vllm-openai-rocm:kimi-k31` image. So far only the\n[AgentX TP8](#preset-mi355x_agentx) configuration has run on MI355X: the tuned single-node AgentX\nconfiguration for concurrency 48, with DCP8, the K3.1 DSpark draft and host KV offload.\n\nOn Zhenwu PPUs, the default MXFP4 checkpoint runs without speculative decoding on\ntwo 16-PPU 810E nodes with TP32 / EP32, or four 8-PPU M890 nodes with TP8 / DP4 / EP32.\nOn 810E, `ppu_deep_gemm_w4a16` loads the checkpoint with a W4A16 compute path.\nUse the PPU image specified for each hardware configuration.\n\n## Tensor + expert parallelism\n\nOn one B300 node, the **Tensor + Expert Parallel** strategy keeps attention at TP8 and runs\nthe routed experts with expert parallelism on the MegaMoE kernel. It has the lowest\nlong-prompt time to first token of the single-node layouts: a single 340K-token prompt\nprefills in about 10 s, against about 15 s at plain TP8 (8x B300, one request at a time,\npre-release checkpoint).\n\n## Long context: decode context parallelism\n\nThe **Decode Context Parallelism** row shards the MLA latent and indexer-key caches across\nthe 8 GPUs of the TP group instead of replicating them, so DCP8 holds about 7x the KV cache\ntokens of TP8 (about 18.5M vs 2.6M tokens with FP8 KV on 8x B300). Prefill is as fast as\nat TP8. Decode steps are 7–25% slower than TP8 at the same batch, so use DCP when KV\ncapacity is the limit: very long contexts or many long-lived agentic sessions.\n\n## Low-latency decode\n\nThe **Low-latency decode** feature switches decode steps to a fused CuTe DSL path tuned for\nsmall batches; prefill and mixed steps still run the default path. On 8x B300 at TP8 it cuts\nthe decode step by 1.6–1.7x at batch 1 and 1.2–1.3x at batch 32–64. Under DCP8 it brings decode\nto within 6% of TP8. It requires the V2 model runner and FULL decode CUDA graphs (the\ndefaults), TP8 or TP8 + DCP8, and no expert parallelism in this recipe.\n\n## NVFP4 KV cache\n\nThe **NVFP4 KV cache** feature stores the MLA latent in NVFP4 (352 bytes per token per layer\ninstead of 576) and holds about 1.5x the KV cache tokens of FP8. The indexer keys stay FP8,\nso the top-2048 token selection is unchanged. GSM8K matched FP8 on 8x GB300 (96.1% vs 96.2%);\nlong-context accuracy has not been measured yet.\n\n## Prefill/decode disaggregation\n\nThe **Prefill/Decode Disaggregation** strategy runs one TP8 prefill pool (with expert\nparallelism) and one DEP16 decode pool (TP1 x DP16 with expert parallelism), and decode\npulls each request's KV from prefill over NIXL. Every PD worker sets\n`VLLM_SSM_CONV_STATE_LAYOUT=DS`: NIXL transfers the Kimi Delta Attention conv state in three\nreads, which needs that layout, and the NIXL connector refuses to start without it.\n\n## KV cache offloading\n\nThe **Simple** option on the KV Offload row adds `SimpleCPUOffloadConnector`, a host-DRAM\nprefix tier for each GPU. Under Prefill/Decode Disaggregation it is added to the prefill pool\nonly, next to NIXL in a MultiConnector; decode keeps plain NIXL. The tier pins\n`cpu_bytes_to_use_per_rank` of host memory for every GPU, so size it from the host DRAM that\nis free after the engines are up. The tested GB300 prefill workers used 128 GiB per GPU\n(512 GiB per tray).\n\n## Client usage\n\n```bash\ncurl http://localhost:8000/v1/chat/completions \\\n -H \"Content-Type: application/json\" \\\n -d '{\n \"model\": \"moonshotai/Kimi-K3.1-Pro\",\n \"messages\": [{\"role\": \"user\", \"content\": \"Explain sparse attention in two sentences.\"}],\n \"temperature\": 0.7,\n \"top_p\": 0.95\n }'\n```\n\n## References\n\n- [Model card](https://huggingface.co/moonshotai/Kimi-K3.1-Pro)\n- [Kimi-K3 recipe](https://recipes.vllm.ai/moonshotai/Kimi-K3)\n", "source": { "source": "vllm", "repo": "vllm-project/vllm", "ref": "main", "sha": "56dc7b70ce535c8410b97fa39db3fd5eec379b30", "path": "recipes/models/moonshotai/Kimi-K3.1-Pro.yaml", "edit_url": "https://github.com/vllm-project/vllm/edit/main/recipes/models/moonshotai/Kimi-K3.1-Pro.yaml", "overlays": [] } }