{ "hf_id": "moonshotai/Kimi-K3", "meta": { "title": "Kimi-K3", "slug": "kimi-k3", "provider": "Moonshot AI", "description": "Pre-release 2.8T-parameter native multimodal MoE with Kimi Delta Attention, Gated MLA, Attention Residuals, and a 1M-token context window", "date_added": "2026-07-27", "date_updated": "2026-07-25", "difficulty": "hard", "tasks": [ "multimodal", "text" ], "performance_headline": "Pre-release TP8, TEP16, and disaggregated P/D profiles for the 2.8T MXFP4 checkpoint", "related_recipes": [ "moonshotai/Kimi-K2.6", "moonshotai/Kimi-K2.5" ], "default_hardware": "b300", "hardware": { "h200": "verified", "b300": "verified", "gb300": "verified", "mi355x": "verified" } }, "recommended_command": { "hardware": "h200", "strategy": "multi_node_tp", "variant": "default", "node_count": 2, "deploy_type": "multi_node", "env": { "GLOO_SOCKET_IFNAME": "$IFACE_NAME", "NCCL_SOCKET_IFNAME": "$IFACE_NAME", "VLLM_ENABLE_K3_LATENT_MOE_TAIL_FUSION": "1" }, "docker_image": "vllm/vllm-openai:kimi-k3", "head_command": "vllm serve moonshotai/Kimi-K3 \\\n --trust-remote-code \\\n --load-format fastsafetensors \\\n --gpu-memory-utilization 0.95 \\\n -cc.pass_config.fuse_allreduce_rms=False \\\n --tensor-parallel-size 16 \\\n --nnodes 2 \\\n --node-rank 0 \\\n --master-addr $HEAD_IP \\\n --no-enable-flashinfer-autotune \\\n --moe-backend marlin \\\n --disable-custom-all-reduce \\\n --enable-auto-tool-choice \\\n --tool-call-parser kimi_k3 \\\n --reasoning-parser kimi_k3", "worker_command": "vllm serve moonshotai/Kimi-K3 \\\n --trust-remote-code \\\n --load-format fastsafetensors \\\n --gpu-memory-utilization 0.95 \\\n -cc.pass_config.fuse_allreduce_rms=False \\\n --tensor-parallel-size 16 \\\n --nnodes 2 \\\n --node-rank 1 \\\n --master-addr $HEAD_IP \\\n --headless \\\n --no-enable-flashinfer-autotune \\\n --moe-backend marlin \\\n --disable-custom-all-reduce \\\n --enable-auto-tool-choice \\\n --tool-call-parser kimi_k3 \\\n --reasoning-parser kimi_k3", "head_argv": [ "vllm", "serve", "moonshotai/Kimi-K3", "--trust-remote-code", "--load-format", "fastsafetensors", "--gpu-memory-utilization", "0.95", "-cc.pass_config.fuse_allreduce_rms=False", "--tensor-parallel-size", "16", "--nnodes", "2", "--node-rank", "0", "--master-addr", "$HEAD_IP", "--no-enable-flashinfer-autotune", "--moe-backend", "marlin", "--disable-custom-all-reduce", "--enable-auto-tool-choice", "--tool-call-parser", "kimi_k3", "--reasoning-parser", "kimi_k3" ], "worker_argv": [ "vllm", "serve", "moonshotai/Kimi-K3", "--trust-remote-code", "--load-format", "fastsafetensors", "--gpu-memory-utilization", "0.95", "-cc.pass_config.fuse_allreduce_rms=False", "--tensor-parallel-size", "16", "--nnodes", "2", "--node-rank", "1", "--master-addr", "$HEAD_IP", "--headless", "--no-enable-flashinfer-autotune", "--moe-backend", "marlin", "--disable-custom-all-reduce", "--enable-auto-tool-choice", "--tool-call-parser", "kimi_k3", "--reasoning-parser", "kimi_k3" ], "head_docker_command": "docker run --gpus all \\\n --privileged --ipc=host -p 8000:8000 \\\n -v ~/.cache/huggingface:/root/.cache/huggingface \\\n -e GLOO_SOCKET_IFNAME=$IFACE_NAME \\\n -e NCCL_SOCKET_IFNAME=$IFACE_NAME \\\n -e VLLM_ENABLE_K3_LATENT_MOE_TAIL_FUSION=1 \\\n vllm/vllm-openai:kimi-k3 moonshotai/Kimi-K3 \\\n --trust-remote-code \\\n --load-format fastsafetensors \\\n --gpu-memory-utilization 0.95 \\\n -cc.pass_config.fuse_allreduce_rms=False \\\n --tensor-parallel-size 16 \\\n --nnodes 2 \\\n --node-rank 0 \\\n --master-addr $HEAD_IP \\\n --no-enable-flashinfer-autotune \\\n --moe-backend marlin \\\n --disable-custom-all-reduce \\\n --enable-auto-tool-choice \\\n --tool-call-parser kimi_k3 \\\n --reasoning-parser kimi_k3", "worker_docker_command": "docker run --gpus all \\\n --privileged --ipc=host -p 8000:8000 \\\n -v ~/.cache/huggingface:/root/.cache/huggingface \\\n -e GLOO_SOCKET_IFNAME=$IFACE_NAME \\\n -e NCCL_SOCKET_IFNAME=$IFACE_NAME \\\n -e VLLM_ENABLE_K3_LATENT_MOE_TAIL_FUSION=1 \\\n vllm/vllm-openai:kimi-k3 moonshotai/Kimi-K3 \\\n --trust-remote-code \\\n --load-format fastsafetensors \\\n --gpu-memory-utilization 0.95 \\\n -cc.pass_config.fuse_allreduce_rms=False \\\n --tensor-parallel-size 16 \\\n --nnodes 2 \\\n --node-rank 1 \\\n --master-addr $HEAD_IP \\\n --headless \\\n --no-enable-flashinfer-autotune \\\n --moe-backend marlin \\\n --disable-custom-all-reduce \\\n --enable-auto-tool-choice \\\n --tool-call-parser kimi_k3 \\\n --reasoning-parser kimi_k3", "head_docker_argv": [ "docker", "run", "--gpus", "all", "--privileged", "--ipc=host", "-p", "8000:8000", "-v", "~/.cache/huggingface:/root/.cache/huggingface", "-e", "GLOO_SOCKET_IFNAME=$IFACE_NAME", "-e", "NCCL_SOCKET_IFNAME=$IFACE_NAME", "-e", "VLLM_ENABLE_K3_LATENT_MOE_TAIL_FUSION=1", "vllm/vllm-openai:kimi-k3", "moonshotai/Kimi-K3", "--trust-remote-code", "--load-format", "fastsafetensors", "--gpu-memory-utilization", "0.95", "-cc.pass_config.fuse_allreduce_rms=False", "--tensor-parallel-size", "16", "--nnodes", "2", "--node-rank", "0", "--master-addr", "$HEAD_IP", "--no-enable-flashinfer-autotune", "--moe-backend", "marlin", "--disable-custom-all-reduce", "--enable-auto-tool-choice", "--tool-call-parser", "kimi_k3", "--reasoning-parser", "kimi_k3" ], "worker_docker_argv": [ "docker", "run", "--gpus", "all", "--privileged", "--ipc=host", "-p", "8000:8000", "-v", "~/.cache/huggingface:/root/.cache/huggingface", "-e", "GLOO_SOCKET_IFNAME=$IFACE_NAME", "-e", "NCCL_SOCKET_IFNAME=$IFACE_NAME", "-e", "VLLM_ENABLE_K3_LATENT_MOE_TAIL_FUSION=1", "vllm/vllm-openai:kimi-k3", "moonshotai/Kimi-K3", "--trust-remote-code", "--load-format", "fastsafetensors", "--gpu-memory-utilization", "0.95", "-cc.pass_config.fuse_allreduce_rms=False", "--tensor-parallel-size", "16", "--nnodes", "2", "--node-rank", "1", "--master-addr", "$HEAD_IP", "--headless", "--no-enable-flashinfer-autotune", "--moe-backend", "marlin", "--disable-custom-all-reduce", "--enable-auto-tool-choice", "--tool-call-parser", "kimi_k3", "--reasoning-parser", "kimi_k3" ], "strategy_spec": { "name": "multi_node_tp", "deploy_type": "multi_node", "display_name": "Multi-Node Tensor Parallel", "orientation": "latency", "description": "Multi-node TP. Splits the model across GPUs spanning multiple nodes. TP is set to total GPU count across all nodes. Head node serves requests, worker nodes run with --headless. Requires InfiniBand interconnect.\n", "hardware_match": { "min_gpus": 2, "multi_node": true }, "vllm_args": [ "-cc.pass_config.fuse_allreduce_rms=False" ], "parallel_flag": "--tensor-parallel-size", "env": { "GLOO_SOCKET_IFNAME": "$IFACE_NAME", "NCCL_SOCKET_IFNAME": "$IFACE_NAME" } }, "hardware_profile": { "brand": "NVIDIA", "generation": "hopper", "display_name": "H200", "description": "NVIDIA H200 SXM 141 GB HBM3e ยท 8-GPU HGX node", "gpu_count": 8, "vram_gb": 1128, "multi_node": false }, "alternatives": { "single_node_tp": "/moonshotai/Kimi-K3/strategies/single_node_tp.json", "multi_node_tep": "/moonshotai/Kimi-K3/strategies/multi_node_tep.json", "multi_node_dep": "/moonshotai/Kimi-K3/strategies/multi_node_dep.json", "pd_cluster": "/moonshotai/Kimi-K3/strategies/pd_cluster.json" }, "by_hardware": { "h200": "/moonshotai/Kimi-K3/hw/h200.json", "h100": "/moonshotai/Kimi-K3/hw/h100.json", "b200": "/moonshotai/Kimi-K3/hw/b200.json", "gb200": "/moonshotai/Kimi-K3/hw/gb200.json", "b300": "/moonshotai/Kimi-K3/hw/b300.json", "gb300": "/moonshotai/Kimi-K3/hw/gb300.json", "mi300x": "/moonshotai/Kimi-K3/hw/mi300x.json", "mi325x": "/moonshotai/Kimi-K3/hw/mi325x.json", "mi355x": "/moonshotai/Kimi-K3/hw/mi355x.json" } }, "model": { "model_id": "moonshotai/Kimi-K3", "min_vllm_version": "0.27.0", "docker_image": { "nvidia": "vllm/vllm-openai:kimi-k3", "amd": "vllm/vllm-openai-rocm:kimi-k3" }, "nightly_required": true, "architecture": "moe", "parameter_count": "2.8T", "active_parameters": "16 experts/token + shared (of 896 routed)", "context_length": 1048576, "base_args": [ "--trust-remote-code", "--load-format", "fastsafetensors", "--moe-backend", "auto", "--gpu-memory-utilization", "0.95" ], "install": { "pip": { "command": "uv venv\nsource .venv/bin/activate\nuv pip install -U vllm --pre \\\n --extra-index-url https://wheels.vllm.ai/nightly/cu130 \\\n --extra-index-url https://download.pytorch.org/whl/cu130 \\\n --index-strategy unsafe-best-match", "note": "Use a K3-enabled vLLM nightly after the integration lands." }, "docker": { "command": "docker pull vllm/vllm-openai:kimi-k3", "note": "Use a K3-enabled nightly image after the integration lands." } } }, "features": { "tool_calling": { "description": "Enable automatic tool choice with the Kimi K3 tool-call parser.", "args": [ "--enable-auto-tool-choice", "--tool-call-parser", "kimi_k3" ] }, "reasoning": { "description": "Parse Kimi K3 reasoning output separately from the final answer.", "args": [ "--reasoning-parser", "kimi_k3" ] }, "spec_decoding": { "description": "Use Inferact trained DSpark.", "args": [ "--max-num-seqs", "32", "--speculative-config", "{\"model\":\"Inferact/Kimi-K3-DSpark\", \"num_speculative_tokens\":7, \"method\": \"dspark\", \"attention_backend\": \"FLASHINFER_MLA\", \"draft_sample_method\": \"probabilistic\", \"rejection_sample_method\": \"block\"}" ], "hardware_overrides": { "amd": { "args": [ "--max-num-seqs", "128", "--speculative-config", "{\"model\":\"Inferact/Kimi-K3-DSpark\", \"num_speculative_tokens\":7, \"method\": \"dspark\", \"attention_backend\": \"TRITON_MLA\", \"draft_sample_method\": \"probabilistic\", \"rejection_sample_method\": \"block\"}" ] } } }, "text_only": { "description": "Skip the vision encoder for text-only workloads. Mutually exclusive with encoder_parallel.", "args": [ "--language-model-only" ] } }, "opt_in_features": [ "spec_decoding", "text_only" ], "variants": { "default": { "precision": "mxfp4", "vram_minimum_gb": 1680, "description": "Quantization-aware-trained MXFP4 weights with MXFP8 activations; footprint is estimated until the checkpoint is published." } }, "strategy_min_gpus": { "single_node_tp": 8, "multi_node_tp": 8, "multi_node_tep": 8, "multi_node_dep": 16 }, "kv_cache_strategy_hardware": { "kv_store_distributed_mooncake": { "h100": "unsupported", "h200": "unsupported", "b200": "unsupported", "gb200": "unsupported", "b300": "unsupported", "gb300": "unsupported", "mi300x": "unsupported", "mi325x": "unsupported", "mi355x": "unsupported" }, "kv_store_centralized_mooncake": { "h100": "unsupported", "h200": "unsupported", "b200": "unsupported", "gb200": "unsupported", "b300": "unsupported", "gb300": "unsupported", "mi300x": "unsupported", "mi325x": "unsupported", "mi355x": "unsupported" } }, "hardware_overrides": { "hopper": { "extra_args": [ "--no-enable-flashinfer-autotune", "--moe-backend", "marlin", "--disable-custom-all-reduce" ], "extra_env": { "VLLM_ENABLE_K3_LATENT_MOE_TAIL_FUSION": "1" } }, "blackwell": { "extra_args": [ "--max-model-len", "1000000", "--kv-cache-dtype", "fp8", "--attention-config", "{\"mla_prefill_backend\":\"TRTLLM_RAGGED\",\"use_prefill_query_quantization\":true}", "--max-num-batched-tokens", "32768", "--enable-prefix-caching" ], "extra_env": { "VLLM_ENABLE_K3_LATENT_MOE_TAIL_FUSION": "1", "VLLM_ALLREDUCE_USE_FLASHINFER": "1", "VLLM_ENGINE_READY_TIMEOUT_S": "3600", "VLLM_USE_V2_MODEL_RUNNER": "1", "VLLM_USE_RUST_FRONTEND": "1" } }, "nvidia": { "extra_args": [ "--no-enable-flashinfer-autotune" ], "extra_env": { "VLLM_ENABLE_K3_LATENT_MOE_TAIL_FUSION": "1" } }, "amd": { "extra_env": { "VLLM_ROCM_USE_AITER": "1", "SAFETENSORS_FAST_GPU": "1", "AITER_SITUV2_A8W4": "1", "AITER_BF16_FP8_MOE_BOUND": "0", "VLLM_USE_BREAKABLE_CUDAGRAPH": "0" }, "extra_args": [ "--load-format", "auto", "--gpu-memory-utilization", "0.95", "--mm-encoder-tp-mode", "data", "--max-num-seqs", "128", "--reasoning-parser", "kimi_k3", "--max-num-batched-tokens", "4096" ] } }, "guide": "## Overview\n\nKimi K3 is a 2.8-trillion-parameter Mixture-of-Experts model (16 of 896 experts active per token) built on the Kimi Delta Attention (KDA) and Attention Residuals (AttnRes), with a 1M-token context window and native vision. \n\n## Prerequisites\n\n- **vLLM:** Use the vllm/vllm-openai:kimi-k3 docker\n- **Hardware:** At least 8x GB300. Multi-node for real production traffic.\n- **ROCm:** Use vllm/vllm-openai_rocm:kimi-k3 docker and at least 8x MI355X/MI350X hardware.\n\n## Client Usage\n\nOnce the vLLM server is running, consume it via the OpenAI-compatible API:\n\n```python\nimport time\nfrom openai import OpenAI\n\nclient = OpenAI(\n api_key=\"EMPTY\",\n base_url=\"http://localhost:8000/v1\",\n timeout=3600\n)\n\nmessages = [\n {\n \"role\": \"user\",\n \"content\": [\n {\n \"type\": \"image_url\",\n \"image_url\": {\n \"url\": \"https://ofasys-multimodal-wlcb-3-toshanghai.oss-accelerate.aliyuncs.com/wpf272043/keepme/image/receipt.png\"\n }\n },\n {\n \"type\": \"text\",\n \"text\": \"Read all the text in the image.\"\n }\n ]\n }\n]\n\nstart = time.time()\nresponse = client.chat.completions.create(\n model=\"moonshotai/Kimi-K3\",\n messages=messages,\n max_tokens=2048\n)\nprint(f\"Response costs: {time.time() - start:.2f}s\")\nprint(f\"Generated text: {response.choices[0].message.content}\")\n```\n\n## Notes\n\n- **Cross node communitcation**: Use `--all2all-backend deepep_v2` for RDMA and `--all2all-backend flashinfer_nvlink_one_sided` for NVLink.\n- **MoE backend**: Recommend to use deep_gemm_mega_moe for any DEP environment and flashinfer_trtllm for TP>1.\n- **Model Runner v2 and Rust Frontend**: `VLLM_USE_V2_MODEL_RUNNER=1` and `VLLM_USE_RUST_FRONTEND=1`: Model Runner v2 and Rust Frontend fully supports this model and can be enabled if needed.\n- **Tool calling**: K3 occasionally emit a tool-call format its own parser doesn't expect. Suggest to run do schema validation and retry.\n- **AMD (MI355X / MI350X, CDNA4 gfx950)**: set AITER_SITUV2_A8W4 to 0 along with AITER master flag to use aiter a16w4 MoE path. Set it to 1 to use aiter a8w4 MoE path.\n- **max-model-len**: Adjust max-model-len for different benchmark scenarios for best performance.\n- **RDMA**: If RDMA is enabled, set `UCX_TLS=\"rc,cuda_copy\"` to make sure KV Cache transfer goes through RDMA.\n- **FP8 KV**: If FP8 KV cache is needed, please also add `--attention-config '{\"use_prefill_query_quantization\":true}'` when serving vLLM.\n" }