{ "hf_id": "moonshotai/Kimi-K3", "meta": { "title": "Kimi-K3", "slug": "kimi-k3", "provider": "Moonshot AI", "description": "Pre-release 2.8T-parameter native multimodal MoE with Kimi Delta Attention, Gated MLA, Attention Residuals, and a 1M-token context window", "date_added": "2026-07-27", "date_updated": "2026-08-27", "difficulty": "hard", "tasks": [ "multimodal", "text" ], "performance_headline": "Pre-release TP8, TEP16, TP8xPP2, and disaggregated P/D profiles for the 2.8T MXFP4 checkpoint", "related_recipes": [ "moonshotai/Kimi-K2.6", "moonshotai/Kimi-K2.5" ], "default_hardware": "b300", "hardware": { "h200": "verified", "b200": "verified", "b300": "verified", "gb200": "verified", "gb300": "verified", "mi355x": "verified", "ascend_910c": "verified" } }, "recommended_command": { "hardware": "b300", "strategy": "single_node_tp", "variant": "default", "node_count": 1, "deploy_type": "single_node", "env": { "VLLM_ALLREDUCE_USE_FLASHINFER": "1", "VLLM_ENGINE_READY_TIMEOUT_S": "3600", "VLLM_USE_V2_MODEL_RUNNER": "1", "VLLM_USE_RUST_FRONTEND": "1" }, "docker_image": "vllm/vllm-openai:latest", "command": "vllm serve moonshotai/Kimi-K3 \\\n --trust-remote-code \\\n --gpu-memory-utilization 0.95 \\\n --tensor-parallel-size 8 \\\n --load-format fastsafetensors \\\n --no-enable-flashinfer-autotune \\\n --max-model-len 1048576 \\\n --kv-cache-dtype fp8 \\\n --attention-config '{\"use_prefill_query_quantization\":true,\"mla_prefill_backend\":\"flashinfer\"}' \\\n --enable-prefix-caching \\\n --prefix-match-unit 128 \\\n --enable-auto-tool-choice \\\n --tool-call-parser kimi_k3 \\\n --reasoning-parser kimi_k3", "argv": [ "vllm", "serve", "moonshotai/Kimi-K3", "--trust-remote-code", "--gpu-memory-utilization", "0.95", "--tensor-parallel-size", "8", "--load-format", "fastsafetensors", "--no-enable-flashinfer-autotune", "--max-model-len", "1048576", "--kv-cache-dtype", "fp8", "--attention-config", "{\"use_prefill_query_quantization\":true,\"mla_prefill_backend\":\"flashinfer\"}", "--enable-prefix-caching", "--prefix-match-unit", "128", "--enable-auto-tool-choice", "--tool-call-parser", "kimi_k3", "--reasoning-parser", "kimi_k3" ], "docker_command": "docker run --gpus all \\\n --privileged --ipc=host -p 8000:8000 \\\n -v ~/.cache/huggingface:/root/.cache/huggingface \\\n -e VLLM_ALLREDUCE_USE_FLASHINFER=1 \\\n -e VLLM_ENGINE_READY_TIMEOUT_S=3600 \\\n -e VLLM_USE_V2_MODEL_RUNNER=1 \\\n -e VLLM_USE_RUST_FRONTEND=1 \\\n vllm/vllm-openai:latest moonshotai/Kimi-K3 \\\n --trust-remote-code \\\n --gpu-memory-utilization 0.95 \\\n --tensor-parallel-size 8 \\\n --load-format fastsafetensors \\\n --no-enable-flashinfer-autotune \\\n --max-model-len 1048576 \\\n --kv-cache-dtype fp8 \\\n --attention-config '{\"use_prefill_query_quantization\":true,\"mla_prefill_backend\":\"flashinfer\"}' \\\n --enable-prefix-caching \\\n --prefix-match-unit 128 \\\n --enable-auto-tool-choice \\\n --tool-call-parser kimi_k3 \\\n --reasoning-parser kimi_k3", "docker_argv": [ "docker", "run", "--gpus", "all", "--privileged", "--ipc=host", "-p", "8000:8000", "-v", "~/.cache/huggingface:/root/.cache/huggingface", "-e", "VLLM_ALLREDUCE_USE_FLASHINFER=1", "-e", "VLLM_ENGINE_READY_TIMEOUT_S=3600", "-e", "VLLM_USE_V2_MODEL_RUNNER=1", "-e", "VLLM_USE_RUST_FRONTEND=1", "vllm/vllm-openai:latest", "moonshotai/Kimi-K3", "--trust-remote-code", "--gpu-memory-utilization", "0.95", "--tensor-parallel-size", "8", "--load-format", "fastsafetensors", "--no-enable-flashinfer-autotune", "--max-model-len", "1048576", "--kv-cache-dtype", "fp8", "--attention-config", "{\"use_prefill_query_quantization\":true,\"mla_prefill_backend\":\"flashinfer\"}", "--enable-prefix-caching", "--prefix-match-unit", "128", "--enable-auto-tool-choice", "--tool-call-parser", "kimi_k3", "--reasoning-parser", "kimi_k3" ], "strategy_spec": { "name": "single_node_tp", "deploy_type": "single_node", "display_name": "Tensor Parallel", "orientation": "latency", "description": "Single-node tensor parallel. Splits the model across all local GPUs. TP size is set to the GPU count at deploy time. The simplest multi-GPU strategy — works for all model architectures.\n", "hardware_match": { "min_gpus": 1, "max_gpus": 8, "multi_node": false }, "vllm_args": [], "parallel_flag": "--tensor-parallel-size" }, "hardware_profile": { "brand": "NVIDIA", "generation": "blackwell", "display_name": "B300", "description": "NVIDIA B300 SXM 268 GB HBM3e · 8-GPU HGX B300 node (Blackwell Ultra)", "gpu_count": 8, "vram_gb": 2144, "multi_node": false }, "alternatives": { "multi_node_tp": "/moonshotai/Kimi-K3/strategies/multi_node_tp.json", "multi_node_tep": "/moonshotai/Kimi-K3/strategies/multi_node_tep.json", "multi_node_tp_pp": "/moonshotai/Kimi-K3/strategies/multi_node_tp_pp.json", "multi_node_dep": "/moonshotai/Kimi-K3/strategies/multi_node_dep.json", "pd_cluster": "/moonshotai/Kimi-K3/strategies/pd_cluster.json", "kv_store_centralized_mooncake": "/moonshotai/Kimi-K3/strategies/kv_store_centralized_mooncake.json", "kv_store_distributed_mooncake": "/moonshotai/Kimi-K3/strategies/kv_store_distributed_mooncake.json" }, "by_hardware": { "b300": "/moonshotai/Kimi-K3/hw/b300.json", "h100": "/moonshotai/Kimi-K3/hw/h100.json", "h200": "/moonshotai/Kimi-K3/hw/h200.json", "b200": "/moonshotai/Kimi-K3/hw/b200.json", "gb200": "/moonshotai/Kimi-K3/hw/gb200.json", "gb300": "/moonshotai/Kimi-K3/hw/gb300.json", "mi300x": "/moonshotai/Kimi-K3/hw/mi300x.json", "mi325x": "/moonshotai/Kimi-K3/hw/mi325x.json", "mi355x": "/moonshotai/Kimi-K3/hw/mi355x.json" } }, "model": { "model_id": "moonshotai/Kimi-K3", "min_vllm_version": "0.27.1", "architecture": "moe", "parameter_count": "2.8T", "active_parameters": "16 experts/token + shared (of 896 routed)", "context_length": 1048576, "base_args": [ "--trust-remote-code", "--gpu-memory-utilization", "0.95" ], "install": { "pip": { "command": "uv venv\nsource .venv/bin/activate\nuv pip install -U vllm --torch-backend auto", "note": "Use a K3-enabled vLLM nightly after the integration lands." }, "docker": { "command": "docker pull vllm/vllm-openai:latest", "note": "Use a K3-enabled nightly image after the integration lands. CUDA 13 (cu130) build only — no -cu129 tag; the host needs an r580+ NVIDIA driver." } } }, "features": { "tool_calling": { "description": "Enable automatic tool choice with the Kimi K3 tool-call parser.", "args": [ "--enable-auto-tool-choice", "--tool-call-parser", "kimi_k3" ] }, "reasoning": { "description": "Parse Kimi K3 reasoning output separately from the final answer.", "args": [ "--reasoning-parser", "kimi_k3" ] }, "spec_decoding": { "description": "Use DSpark speculative decoding.", "strategies": [ "single_node_tp", "multi_node_tp", "multi_node_tep", "multi_node_dep", "multi_node_tp_dp", "pd_cluster" ], "args": [ "--speculative-config", "{\"model\":\"RedHatAI/Kimi-K3-speculator.dspark\", \"num_speculative_tokens\":8, \"method\": \"dspark\", \"draft_sample_method\": \"probabilistic\", \"rejection_sample_method\": \"block\"}" ], "hardware_overrides": { "amd": { "args": [ "--max-num-seqs", "128", "--speculative-config", "{\"model\":\"RedHatAI/Kimi-K3-speculator.dspark\", \"num_speculative_tokens\":8, \"method\": \"dspark\", \"draft_sample_method\": \"probabilistic\", \"rejection_sample_method\": \"block\"}" ] }, "npu": { "args": [ "--speculative-config", "{\"method\":\"dspark\",\"model\":\"RedHatAI/Kimi-K3-speculator.dspark\",\"num_speculative_tokens\":7,\"enforce_eager\":true}" ] } } }, "text_only": { "description": "Skip the vision encoder for text-only workloads. Mutually exclusive with encoder_parallel.", "args": [ "--language-model-only" ] }, "decode_context_parallelism": { "description": "Decode context parallelism (DCP): shard the decode KV cache across the tensor-parallel ranks for decode-heavy long-context serving. The DCP size must divide the tensor-parallel size, so the args below assume the TP8 single-node layout. Pairs the TOKENSPEED_MLA decode backend with TRTLLM_RAGGED MLA prefill under FP8 KV.", "strategies": [ "single_node_tp" ], "args": [ "--decode-context-parallel-size", "8", "--dcp-comm-backend", "a2a", "--attention-backend", "TOKENSPEED_MLA", "--attention-config", "{\"use_prefill_query_quantization\":true,\"mla_prefill_backend\":\"TRTLLM_RAGGED\"}" ] } }, "opt_in_features": [ "spec_decoding", "text_only", "decode_context_parallelism" ], "variants": { "default": { "precision": "mxfp4", "vram_minimum_gb": 1680, "supported_hardware": [ "h100", "h200", "b200", "b300", "gb200", "gb300", "mi300x", "mi325x", "mi355x" ], "description": "Quantization-aware-trained MXFP4 weights with MXFP8 activations; footprint is estimated until the checkpoint is published." }, "nvfp4": { "precision": "nvfp4", "vram_minimum_gb": 1650, "supported_hardware": [ "b200", "b300", "gb200", "gb300" ], "model_id": "RedHatAI/Kimi-K3-NVFP4", "description": "RedHatAI compressed-tensors NVFP4 quantization; MoE layers quantized to NVFP4, attention and embeddings at full precision", "json": "/RedHatAI/Kimi-K3-NVFP4.json" }, "ascend_w4a8": { "model_id": "Eco-Tech/Kimi-K3-w4a8", "precision": "int4", "label": "W4A8 (Ascend)", "vram_minimum_gb": 1711, "supported_hardware": [ "ascend_910c" ], "extra_args": [ "--quantization", "ascend", "--enable-prefix-caching", "--enable-expert-parallel", "--max-num-seqs", "16", "--max-model-len", "131072", "--max-num-batched-tokens", "24576", "--gpu-memory-utilization", "0.9", "--compilation-config", "{\"cudagraph_mode\":\"FULL_DECODE_ONLY\"}", "--mm-processor-cache-gb", "0", "--additional-config", "{\"enable_cpu_binding\":true, \"enable_flashcomm1\":true}", "--mm-encoder-tp-mode", "data", "--limit-mm-per-prompt", "{\"vision_chunk\": 2}", "--allowed-local-media-path", "/" ], "extra_env": { "PYTORCH_NPU_ALLOC_CONF": "expandable_segments:True", "OMP_PROC_BIND": "false", "OMP_NUM_THREADS": "1", "TASK_QUEUE_ENABLE": "1", "HCCL_BUFFSIZE": "800", "VLLM_ENGINE_READY_TIMEOUT_S": "7200" }, "hardware_overrides": { "ascend_910c": { "docker_image": "quay.io/ascend/vllm-ascend:kimi-k3-a3" } }, "description": "ModelSlim INT4 W4A8 for Ascend NPU. Requires --quantization ascend. Validated on 4× 910C mixed P/D.", "json": "/Eco-Tech/Kimi-K3-w4a8.json" } }, "strategy_hardware": { "multi_node_tp_dp": { "h200": "supported", "gb200": "supported", "gb300": "supported", "ascend_910c": "supported" }, "multi_node_dep": { "h100": "unsupported", "h200": "unsupported", "ascend_910c": "unsupported" }, "multi_node_tp": { "ascend_910c": "unsupported" }, "multi_node_tep": { "ascend_910c": "unsupported" }, "multi_node_tp_pp": { "ascend_910c": "unsupported" }, "pd_cluster": { "ascend_910c": "unsupported" } }, "strategy_min_gpus": { "single_node_tp": 8, "multi_node_tp": 8, "multi_node_tp_dp": 16, "multi_node_tep": 8, "multi_node_tp_pp": 16, "multi_node_dep": 16, "h100": 32, "ascend_910c": 64 }, "hardware_overrides": { "hopper": { "extra_args": [ "--gpu-memory-utilization", "0.97", "--max-num-seqs", "5", "--max-model-len", "32768", "--moe-backend", "marlin", "--disable-custom-all-reduce", "--no-enable-flashinfer-autotune", "--max-num-batched-tokens", "4096", "--attention-backend", "FLASHMLA" ], "extra_env": { "VLLM_ENGINE_READY_TIMEOUT_S": "3600", "VLLM_USE_V2_MODEL_RUNNER": "1", "VLLM_USE_RUST_FRONTEND": "1", "PYTORCH_CUDA_ALLOC_CONF": "expandable_segments:True" } }, "blackwell": { "extra_args": [ "--load-format", "fastsafetensors", "--no-enable-flashinfer-autotune", "--max-model-len", "1048576", "--kv-cache-dtype", "fp8", "--attention-config", "{\"use_prefill_query_quantization\":true,\"mla_prefill_backend\":\"flashinfer\"}", "--enable-prefix-caching", "--prefix-match-unit", "128" ], "extra_env": { "VLLM_ALLREDUCE_USE_FLASHINFER": "1", "VLLM_ENGINE_READY_TIMEOUT_S": "3600", "VLLM_USE_V2_MODEL_RUNNER": "1", "VLLM_USE_RUST_FRONTEND": "1" } }, "amd": { "extra_env": { "VLLM_ROCM_USE_AITER": "1", "SAFETENSORS_FAST_GPU": "1", "VLLM_ROCM_USE_AITER_MOE_SITUV2_A8W4": "1", "AITER_BF16_FP8_MOE_BOUND": "0", "VLLM_USE_BREAKABLE_CUDAGRAPH": "0" }, "extra_args": [ "--load-format", "auto", "--gpu-memory-utilization", "0.95", "--mm-encoder-tp-mode", "data", "--max-num-seqs", "128", "--reasoning-parser", "kimi_k3", "--max-num-batched-tokens", "4096", "--compilation-config", "{\"cudagraph_mode\":\"FULL_DECODE_ONLY\",\"custom_ops\":[\"+fused_rms_norm_gated\"]}" ] } }, "guide": "## Overview\n\nKimi K3 is a 2.8-trillion-parameter Mixture-of-Experts model (16 of 896 experts active per token) built on the Kimi Delta Attention (KDA) and Attention Residuals (AttnRes), with a 1M-token context window and native vision. \n\n## Prerequisites\n\n- **vLLM:** Use the vllm/vllm-openai:kimi-k3 docker\n- **CUDA:** The kimi-k3 image ships as a CUDA 13 (cu130) build only — there is no -cu129 tag, and the K3-enabled wheels are not on the cu129 nightly index. The host needs an r580+ NVIDIA driver; on a CUDA 12.9 (r575) host, upgrade the driver or build vLLM from the K3 branch against cu129 PyTorch yourself.\n- **Hardware:** At least 8x GB300. Multi-node for real production traffic.\n- **ROCm:** Use vllm/vllm-openai_rocm:kimi-k3 docker and at least 8x MI355X/MI350X hardware.\n\n## Client Usage\n\nOnce the vLLM server is running, consume it via the OpenAI-compatible API:\n\n```python\nimport time\nfrom openai import OpenAI\n\nclient = OpenAI(\n api_key=\"EMPTY\",\n base_url=\"http://localhost:8000/v1\",\n timeout=3600\n)\n\nmessages = [\n {\n \"role\": \"user\",\n \"content\": [\n {\n \"type\": \"image_url\",\n \"image_url\": {\n \"url\": \"https://ofasys-multimodal-wlcb-3-toshanghai.oss-accelerate.aliyuncs.com/wpf272043/keepme/image/receipt.png\"\n }\n },\n {\n \"type\": \"text\",\n \"text\": \"Read all the text in the image.\"\n }\n ]\n }\n]\n\nstart = time.time()\nresponse = client.chat.completions.create(\n model=\"moonshotai/Kimi-K3\",\n messages=messages,\n max_tokens=2048\n)\nprint(f\"Response costs: {time.time() - start:.2f}s\")\nprint(f\"Generated text: {response.choices[0].message.content}\")\n```\n\n## Notes\n\n- **Cross node communitcation**: Use `--all2all-backend deepep_v2` for RDMA and `--all2all-backend flashinfer_nvlink_one_sided` for NVLink.\n- **MoE backend**: Recommend to use `deep_gemm_mega_moe` for DEP deployments with cross-node NVLink e.g. GB200 and GB300. Note that DeepGEMM MegaMoE is not compatible with cross-node RDMA.\n- **Model Runner v2 and Rust Frontend**: `VLLM_USE_V2_MODEL_RUNNER=1` and `VLLM_USE_RUST_FRONTEND=1`: Model Runner v2 and Rust Frontend fully supports this model and can be enabled if needed.\n- **Tool calling**: K3 occasionally emit a tool-call format its own parser doesn't expect. Suggest to run do schema validation and retry.\n- **AMD (MI355X / MI350X, CDNA4 gfx950)**: set AITER_SITUV2_A8W4 to 0 along with AITER master flag to use aiter a16w4 MoE path. Set it to 1 to use aiter a8w4 MoE path.\n- **max-model-len**: Adjust max-model-len for different benchmark scenarios for best performance.\n- **RDMA**: If RDMA is enabled, set `UCX_TLS=\"rc,cuda_copy\"` to make sure KV Cache transfer goes through RDMA.\n- **MNNVL environments** (GB200/GB300 NVL): recommend adding `NCCL_MNNVL_ENABLE=1`, `NCCL_CUMEM_ENABLE=1`, and `NCCL_NVLS_ENABLE=1`.\n- **mlx5 dmabuf registration failures**: if engine init fails with \"NCCL error: unhandled system error\" and the log shows `mlx5dv_reg_dmabuf_mr` errno 524, the kernel/driver lacks mlx5 dmabuf support (NCCL 2.28 registers dmabuf by default). Set `NCCL_DMABUF_ENABLE=0` to fall back to `nvidia_peermem` (must be loaded on the nodes) — still GPUDirect RDMA.\n- **FP8 KV**: If FP8 KV cache is needed, please also add `--attention-config '{\"use_prefill_query_quantization\":true,\"mla_prefill_backend\":\"flashinfer\"}'` when serving vLLM. `flashinfer` is the recommended default and is what the Blackwell profile above emits, but it is not the only supported choice: vLLM 0.27.1 registers three MLA prefill backends — `FLASHINFER`, `TRTLLM_RAGGED` and `TOKENSPEED_MLA` (see `vllm/v1/attention/backends/mla/prefill/registry.py`) — and any of them may be substituted for `mla_prefill_backend` alongside `use_prefill_query_quantization`. Which one wins depends on the input-length distribution, so benchmark before switching.\n- **Prefix-match unit**: K3 is a hybrid model: its MLA attention layers and KDA (Mamba-like) layers form two KV-cache groups under the hybrid KV-cache manager. With prefix caching on, vLLM pads the attention block size up to match the Mamba state page, so both groups resolve to one (large) block size and the default prefix-cache hit boundary lands on that block — very coarse. The Blackwell profile sets `--prefix-match-unit 128`: the inflation formula always yields a multiple of 128, so 128 divides whatever block size results, and it aligns with the MLA kernel's native block boundary, giving finer prefix-hit granularity. The value is only effective when prefix caching (or a KV connector) is active; it is a no-op otherwise, and it is safe under all tensor-parallel layouts.\n- **Decode context parallelism (DCP)**: enable the `dcp` opt-in feature in the command builder to shard the decode KV cache across the tensor-parallel ranks for decode-heavy long-context serving. It emits `--decode-context-parallel-size 8 --dcp-comm-backend a2a --attention-backend TOKENSPEED_MLA` with `mla_prefill_backend: TRTLLM_RAGGED`, and is scoped to the TP8 single-node profile because the DCP size must divide the tensor-parallel size. On builds that carry them, the `VLLM_USE_DIRECT_DCP_A2A`, `VLLM_USE_DIRECT_DCP_Q_GATHER` and `VLLM_USE_DIRECT_DCP_KV_GATHER` knobs default to auto and turn the direct DCP ops on where applicable; set them to `1` only to pin that path explicitly, or `0` to disable it.\n- Ascend W4A8 checkpoint (ModelScope): https://www.modelscope.cn/models/Eco-Tech/Kimi-K3-w4a8\n- vLLM Ascend tutorial: https://docs.vllm.ai/projects/ascend/en/v0.23.0/tutorials/models/Kimi-K3.html\n" }