moonshotai/Kimi-K3

Overview

Kimi K3 is a 2.8-trillion-parameter Mixture-of-Experts model (16 of 896 experts active per token) built on the Kimi Delta Attention (KDA) and Attention Residuals (AttnRes), with a 1M-token context window and native vision.

Prerequisites

• vLLM: Use the vllm/vllm-openai:kimi-k3 docker
• CUDA: The kimi-k3 image ships as a CUDA 13 (cu130) build only — there is no -cu129 tag, and the K3-enabled wheels are not on the cu129 nightly index. The host needs an r580+ NVIDIA driver; on a CUDA 12.9 (r575) host, upgrade the driver or build vLLM from the K3 branch against cu129 PyTorch yourself.
• Hardware: At least 8x GB300. Multi-node for real production traffic.
• ROCm: Use vllm/vllm-openai_rocm:kimi-k3 docker and at least 8x MI355X/MI350X hardware.

Client Usage

Once the vLLM server is running, consume it via the OpenAI-compatible API:

import time
from openai import OpenAI

client = OpenAI(
    api_key="EMPTY",
    base_url="http://localhost:8000/v1",
    timeout=3600
)

messages = [
    {
        "role": "user",
        "content": [
            {
                "type": "image_url",
                "image_url": {
                    "url": "https://ofasys-multimodal-wlcb-3-toshanghai.oss-accelerate.aliyuncs.com/wpf272043/keepme/image/receipt.png"
                }
            },
            {
                "type": "text",
                "text": "Read all the text in the image."
            }
        ]
    }
]

start = time.time()
response = client.chat.completions.create(
    model="moonshotai/Kimi-K3",
    messages=messages,
    max_tokens=2048
)
print(f"Response costs: {time.time() - start:.2f}s")
print(f"Generated text: {response.choices[0].message.content}")

Notes

• Cross node communitcation: Use --all2all-backend deepep_v2 for RDMA and --all2all-backend flashinfer_nvlink_one_sided for NVLink.
• MoE backend: Recommend to use deep_gemm_mega_moe for DEP deployments with cross-node NVLink e.g. GB200 and GB300. Note that DeepGEMM MegaMoE is not compatible with cross-node RDMA.
• Model Runner v2 and Rust Frontend: VLLM_USE_V2_MODEL_RUNNER=1 and VLLM_USE_RUST_FRONTEND=1: Model Runner v2 and Rust Frontend fully supports this model and can be enabled if needed.
• Tool calling: K3 occasionally emit a tool-call format its own parser doesn't expect. Suggest to run do schema validation and retry.
• AMD (MI355X / MI350X, CDNA4 gfx950): set AITER_SITUV2_A8W4 to 0 along with AITER master flag to use aiter a16w4 MoE path. Set it to 1 to use aiter a8w4 MoE path.
• max-model-len: Adjust max-model-len for different benchmark scenarios for best performance.
• RDMA: If RDMA is enabled, set UCX_TLS="rc,cuda_copy" to make sure KV Cache transfer goes through RDMA.
• MNNVL environments (GB200/GB300 NVL): recommend adding NCCL_MNNVL_ENABLE=1, NCCL_CUMEM_ENABLE=1, and NCCL_NVLS_ENABLE=1.
• mlx5 dmabuf registration failures: if engine init fails with "NCCL error: unhandled system error" and the log shows mlx5dv_reg_dmabuf_mr errno 524, the kernel/driver lacks mlx5 dmabuf support (NCCL 2.28 registers dmabuf by default). Set NCCL_DMABUF_ENABLE=0 to fall back to nvidia_peermem (must be loaded on the nodes) — still GPUDirect RDMA.
• FP8 KV: If FP8 KV cache is needed, please also add --attention-config '{"use_prefill_query_quantization":true,"mla_prefill_backend":"flashinfer"}' when serving vLLM. flashinfer is the recommended default and is what the Blackwell profile above emits, but it is not the only supported choice: vLLM 0.27.1 registers three MLA prefill backends — FLASHINFER, TRTLLM_RAGGED and TOKENSPEED_MLA (see vllm/v1/attention/backends/mla/prefill/registry.py) — and any of them may be substituted for mla_prefill_backend alongside use_prefill_query_quantization. Which one wins depends on the input-length distribution, so benchmark before switching.
• Prefix-match unit: K3 is a hybrid model: its MLA attention layers and KDA (Mamba-like) layers form two KV-cache groups under the hybrid KV-cache manager. With prefix caching on, vLLM pads the attention block size up to match the Mamba state page, so both groups resolve to one (large) block size and the default prefix-cache hit boundary lands on that block — very coarse. The Blackwell profile sets --prefix-match-unit 128: the inflation formula always yields a multiple of 128, so 128 divides whatever block size results, and it aligns with the MLA kernel's native block boundary, giving finer prefix-hit granularity. The value is only effective when prefix caching (or a KV connector) is active; it is a no-op otherwise, and it is safe under all tensor-parallel layouts.
• Decode context parallelism (DCP): enable the dcp opt-in feature in the command builder to shard the decode KV cache across the tensor-parallel ranks for decode-heavy long-context serving. It emits --decode-context-parallel-size 8 --dcp-comm-backend a2a --attention-backend TOKENSPEED_MLA with mla_prefill_backend: TRTLLM_RAGGED, and is scoped to the TP8 single-node profile because the DCP size must divide the tensor-parallel size. On builds that carry them, the VLLM_USE_DIRECT_DCP_A2A, VLLM_USE_DIRECT_DCP_Q_GATHER and VLLM_USE_DIRECT_DCP_KV_GATHER knobs default to auto and turn the direct DCP ops on where applicable; set them to 1 only to pin that path explicitly, or 0 to disable it.
• Ascend W4A8 checkpoint (ModelScope): https://www.modelscope.cn/models/Eco-Tech/Kimi-K3-w4a8
• vLLM Ascend tutorial: https://docs.vllm.ai/projects/ascend/en/v0.23.0/tutorials/models/Kimi-K3.html