vLLM/Recipes
Search
Moonshot AI

moonshotai/Kimi-K3

Pre-release 2.8T-parameter native multimodal MoE with Kimi Delta Attention, Gated MLA, Attention Residuals, and a 1M-token context window

Pre-release TP8, TEP16, TP8xPP2, and disaggregated P/D profiles for the 2.8T MXFP4 checkpoint

View on HuggingFace
moe2.8T / 16 experts/token + shared (of 896 routed)1,048,576 ctxvLLM 0.29.0+multimodaltext

Overview

Kimi K3 is a 2.8-trillion-parameter Mixture-of-Experts model (16 of 896 experts active per token) built on the Kimi Delta Attention (KDA) and Attention Residuals (AttnRes), with a 1M-token context window and native vision.

Prerequisites

  • vLLM: Use the vllm/vllm-openai:kimi-k3 docker
  • CUDA: The kimi-k3 image ships as a CUDA 13 (cu130) build only — there is no -cu129 tag, and the K3-enabled wheels are not on the cu129 nightly index. The host needs an r580+ NVIDIA driver; on a CUDA 12.9 (r575) host, upgrade the driver or build vLLM from the K3 branch against cu129 PyTorch yourself.
  • Hardware: At least 8x GB300. Multi-node for real production traffic.
  • ROCm: At least 8x MI355X/MI350X. On MI355X the command builder uses the ROCm 10.0 nightly image vllm/vllm-openai-rocm:nightly-rocm100; the day-0 vllm/vllm-openai-rocm:kimi-k3 image predates the DSpark and MoE fixes that path needs (see the MI355X AgentX preset).

Client Usage

Once the vLLM server is running, consume it via the OpenAI-compatible API:

import time
from openai import OpenAI

client = OpenAI(
    api_key="EMPTY",
    base_url="http://localhost:8000/v1",
    timeout=3600
)

messages = [
    {
        "role": "user",
        "content": [
            {
                "type": "image_url",
                "image_url": {
                    "url": "https://ofasys-multimodal-wlcb-3-toshanghai.oss-accelerate.aliyuncs.com/wpf272043/keepme/image/receipt.png"
                }
            },
            {
                "type": "text",
                "text": "Read all the text in the image."
            }
        ]
    }
]

start = time.time()
response = client.chat.completions.create(
    model="moonshotai/Kimi-K3",
    messages=messages,
    max_tokens=2048
)
print(f"Response costs: {time.time() - start:.2f}s")
print(f"Generated text: {response.choices[0].message.content}")

Notes

  • Cross node communitcation: Use --all2all-backend deepep_v2 for RDMA and --all2all-backend flashinfer_nvlink_one_sided for NVLink.
  • MoE backend: Recommend to use deep_gemm_mega_moe for DEP deployments with cross-node NVLink e.g. GB200 and GB300. Note that DeepGEMM MegaMoE is not compatible with cross-node RDMA.
  • Rust Frontend: Selected by default in the command builder. Switch to Python if you encounter unsupported features or compatibility issues.
  • Model Runner v2: VLLM_USE_V2_MODEL_RUNNER=1: Model Runner v2 fully supports this model and can be enabled if needed.
  • Tool calling: K3 occasionally emit a tool-call format its own parser doesn't expect. Suggest to run do schema validation and retry.
  • AMD (MI355X / MI350X, CDNA4 gfx950): VLLM_ROCM_USE_AITER_MOE_SITUV2=1 (set above) enables AITER's SiTUv2 a4w4 FlyDSL MoE path on builds that carry vllm-project/vllm#53940 (ROCm nightlies since 2026-09-16). v0.30.0 predates it: there the variable is ignored and the MoE runs the default a16w4 path. Unset it or set 0 for the default a16w4 path. On builds with #53940, VLLM_ROCM_USE_AITER_MOE_SITUV2_A8W4 is a deprecated alias for the same a4w4 path and vLLM removes AITER_SITUV2_A8W4 at startup while the toggle is on; on v0.30.0 and older, VLLM_ROCM_USE_AITER_MOE_SITUV2_A8W4=1 selects the a8w4 path instead. Do not set AITER_SITUV2_A8W4=1 by itself: vLLM does not derive its MoE weight layout from that AITER flag (vllm-project/vllm#53954 is open), so the two can disagree and corrupt output without an error.
  • max-model-len: Adjust max-model-len for different benchmark scenarios for best performance.
  • RDMA: If RDMA is enabled, set UCX_TLS="rc,cuda_copy" to make sure KV Cache transfer goes through RDMA.
  • MNNVL environments (GB200/GB300 NVL): recommend adding NCCL_MNNVL_ENABLE=1, NCCL_CUMEM_ENABLE=1, and NCCL_NVLS_ENABLE=1.
  • mlx5 dmabuf registration failures: if engine init fails with "NCCL error: unhandled system error" and the log shows mlx5dv_reg_dmabuf_mr errno 524, the kernel/driver lacks mlx5 dmabuf support (NCCL 2.28 registers dmabuf by default). Set NCCL_DMABUF_ENABLE=0 to fall back to nvidia_peermem (must be loaded on the nodes) — still GPUDirect RDMA.
  • FP8 KV: If FP8 KV cache is needed, please also add --attention-config '{"use_prefill_query_quantization":true,"mla_prefill_backend":"TOKENSPEED_MLA"}' when serving vLLM. TOKENSPEED_MLA is what the Blackwell profile above emits (paired with --attention-backend TOKENSPEED_MLA for decode), but it is not the only supported choice: vLLM 0.27.1 registers three MLA prefill backends — FLASHINFER, TRTLLM_RAGGED and TOKENSPEED_MLA (see vllm/v1/attention/backends/mla/prefill/registry.py) — and any of them may be substituted for mla_prefill_backend alongside use_prefill_query_quantization. Which one wins depends on the input-length distribution, so benchmark before switching. Those backends do not run on ROCm: on MI355X the builder pairs FP8 KV with --attention-config '{"mla_prefill_backend":"ROCM_AITER_FA"}' instead.
  • Prefix-match unit: K3 is a hybrid model: its MLA attention layers and KDA (Mamba-like) layers form two KV-cache groups under the hybrid KV-cache manager. With prefix caching on, vLLM pads the attention block size up to match the Mamba state page, so both groups resolve to one (large) block size and the default prefix-cache hit boundary lands on that block — very coarse. The Blackwell profile sets --prefix-match-unit 128: the inflation formula always yields a multiple of 128, so 128 divides whatever block size results, and it aligns with the MLA kernel's native block boundary, giving finer prefix-hit granularity. The value is only effective when prefix caching (or a KV connector) is active; it is a no-op otherwise, and it is safe under all tensor-parallel layouts.
  • Decode context parallelism (DCP): the qualified DCP layouts are part of the presets that ran them: DCP8 on the GB200 aggregated TP8 x PP2 and GB300 disaggregated TP8 pools, DCP16 on the GB200 TP16 disaggregated pools, DCP16/DCP8 at the GB300 low-latency points, and DCP8 in the MI355X AgentX cells. Each pairs --decode-context-parallel-size (a divisor of the tensor-parallel size) with --dcp-comm-backend a2a and an MLA attention backend that supports it (TOKENSPEED_MLA with mla_prefill_backend: TRTLLM_RAGGED on Blackwell, ROCM_AITER_MLA with mla_prefill_backend: ROCM_AITER_FA on MI355X). On builds that carry them, the VLLM_USE_DIRECT_DCP_A2A, VLLM_USE_DIRECT_DCP_Q_GATHER and VLLM_USE_DIRECT_DCP_KV_GATHER knobs default to auto; set them to 1 to pin the direct DCP ops or 0 to disable them, as the MI355X cells do.
  • Ascend W4A8 checkpoint (ModelScope): https://www.modelscope.cn/models/Eco-Tech/Kimi-K3-w4a8
  • vLLM Ascend tutorial: https://docs.vllm.ai/projects/ascend/en/v0.23.0/tutorials/models/Kimi-K3.html

AMD MI355X: disaggregated prefill and decode

This example uses one TP8/DCP8 prefill worker and one TP8/DCP8 decode worker on separate 8-GPU MI355X nodes. MoRIIO transfers the KV cache in READ mode; MultiConnector combines it with SimpleCPUOffloadConnector on prefill only. Both roles use the same target and DSpark draft checkpoints and FP8 KV.

These standalone commands are based on the 1P1D c48 configuration in InferenceX#4391, with real block verification for serving and accuracy evaluation. They are separate from the command builder's NIXL PD configuration.

Prerequisites

  • Make moonshotai/Kimi-K3 and Inferact/Kimi-K3-DSpark available inside every worker container. The commands below use /models/Kimi-K3 and /models/Inferact-Kimi-K3-DSpark as example container paths; adjust them to match your deployment. Use identical checkpoint revisions on all workers.
  • Use the worker and router images below, with the same worker image build on every node. The worker image contains the required vLLM support; no source patches are needed.
  • Expose all eight GPUs, /dev/infiniband, host networking, and sufficient shared memory to each worker container. Allow locked memory for RDMA. The container's verbs provider must be compatible with the host NIC and driver.
  • The prefill offload budget is 2,399 GB of host DRAM across all eight ranks, in addition to weight-loading and runtime memory. Reduce CPU_OFFLOAD_BYTES if the host cannot accommodate that allocation; the documented configurations use 2399000000000.
  • Set SOCKET_IFNAME and RDMA_DEVICES to your network interface and HCA list. ROUTER_IP and each WORKER_IP must be reachable addresses, not 0.0.0.0. Allow router TCP ports 8000 and 36367, worker HTTP port 6100, and MoRIIO handshake/notification ports 26000–26007 and 27000–27007, plus the site's RDMA traffic.
WORKER_IMAGE=vllm/vllm-openai-rocm:nightly
ROUTER_IMAGE=vllm/vllm-router:nightly

Start the router

Start one router before the workers. Discovery registers workers automatically; do not add static --prefill or --decode endpoints.

docker run --rm --network host --entrypoint vllm-router "$ROUTER_IMAGE" \
  --vllm-pd-disaggregation \
  --kv-connector moriio \
  --vllm-discovery-address 0.0.0.0:36367 \
  --host 0.0.0.0 --port 8000 \
  --worker-startup-timeout-secs 7200 \
  --policy consistent_hash \
  --prefill-policy consistent_hash \
  --decode-policy consistent_hash \
  --log-level info

Start the workers

Save the following as kimi-k3-pd.sh on each worker. Run it with Bash inside the worker image, with the devices and model paths described above.

#!/usr/bin/env bash
set -euo pipefail

ROLE=${1:?Usage: bash kimi-k3-pd.sh prefill|decode}
: "${ROUTER_IP:?Set the router reachable IP}"
: "${WORKER_IP:?Set this worker reachable IP}"
: "${SOCKET_IFNAME:?Set the host network interface}"
: "${RDMA_DEVICES:?Set the comma-separated HCA list}"
CPU_OFFLOAD_BYTES=${CPU_OFFLOAD_BYTES:-2399000000000}

export NCCL_SOCKET_IFNAME="$SOCKET_IFNAME" GLOO_SOCKET_IFNAME="$SOCKET_IFNAME"
export NCCL_IB_HCA="$RDMA_DEVICES"
export VLLM_ROCM_USE_AITER=1 AITER_BF16_FP8_MOE_BOUND=0
export AITER_FLYDSL_STAGE2_FP8=1
export VLLM_ROCM_AITER_MLA_ASM_PADDING=asm VLLM_ROCM_AITER_MLA_DCP_VERIFY=asm
export VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4
export VLLM_ROCM_QUICK_REDUCE_MAX_SIZE_BYTES_MB=256
export SAFETENSORS_FAST_GPU=1
export VLLM_SSM_CONV_STATE_LAYOUT=DS VLLM_KV_CACHE_LAYOUT=HND
export NCCL_DMABUF_ENABLE=0 HSA_ENABLE_IPC_MODE_LEGACY=1 HIP_FORCE_DEV_KERNARG=1
export PYTHONHASHSEED=42 PYTHONNOUSERSITE=1
export VLLM_USE_DIRECT_DCP_A2A=0 VLLM_USE_DIRECT_DCP_Q_GATHER=0 VLLM_USE_DIRECT_DCP_KV_GATHER=0
export VLLM_ENGINE_READY_TIMEOUT_S=7200 VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=1200
export MORI_IO_SQ_BACKOFF_TIMEOUT_US=50000
export MORI_IO_QP_MAX_SEND_WR=16384 MORI_IO_QP_MAX_CQE=32768 MORI_IO_QP_MAX_SGE=2
export MORI_IO_TC_DISABLE=0

CUSTOM_OPS='["+fused_rms_norm_gated","+quant_fp8","+grouped_topk","+sparse_attn_indexer","none"]'
case "$ROLE" in
  prefill)
    MAX_SEQS=96
    BATCHED=8192
    KV_ROLE=kv_producer
    export HSA_NO_SCRATCH_RECLAIM=0
    COMPILATION="{\"mode\":3,\"cudagraph_mode\":\"NONE\",\"max_cudagraph_capture_size\":0,\"custom_ops\":$CUSTOM_OPS}"
    ;;
  decode)
    MAX_SEQS=96
    BATCHED=512
    KV_ROLE=kv_consumer
    CAPTURES="$(seq -s, 1 128),$(seq -s, 130 10 240)"
    COMPILATION="{\"mode\":3,\"cudagraph_mode\":\"FULL_DECODE_ONLY\",\"max_cudagraph_capture_size\":240,\"cudagraph_capture_sizes\":[$CAPTURES],\"custom_ops\":$CUSTOM_OPS}"
    ;;
  *) echo "Role must be prefill or decode" >&2; exit 1 ;;
esac

MORI="{\"kv_connector\":\"MoRIIOConnector\",\"kv_role\":\"$KV_ROLE\",\"kv_load_failure_policy\":\"fail\",\"kv_connector_extra_config\":{\"backend\":\"rdma\",\"read_mode\":true,\"proxy_ip\":\"$ROUTER_IP\",\"proxy_ping_port\":\"36367\",\"host_ip\":\"$WORKER_IP\",\"http_port\":\"6100\",\"handshake_port\":\"26000\",\"notify_port\":\"27000\"}}"
KV_CONFIG=$MORI
if [ "$ROLE" = prefill ]; then
  CPU="{\"kv_connector\":\"SimpleCPUOffloadConnector\",\"kv_role\":\"kv_both\",\"kv_connector_extra_config\":{\"cpu_bytes_to_use\":$CPU_OFFLOAD_BYTES,\"lazy_offload\":false}}"
  KV_CONFIG="{\"kv_connector\":\"MultiConnector\",\"kv_role\":\"kv_both\",\"kv_load_failure_policy\":\"fail\",\"kv_connector_extra_config\":{\"connectors\":[$MORI,$CPU]}}"
fi

exec vllm serve /models/Kimi-K3 \
  --host 0.0.0.0 --port 6100 \
  --served-model-name moonshotai/Kimi-K3 \
  --trust-remote-code \
  --tensor-parallel-size 8 \
  --decode-context-parallel-size 8 \
  --cp-kv-cache-interleave-size 1 \
  --dcp-comm-backend a2a \
  --moe-backend auto \
  --load-format fastsafetensors \
  --gpu-memory-utilization 0.90 \
  --language-model-only \
  --enable-auto-tool-choice \
  --tool-call-parser kimi_k3 --reasoning-parser kimi_k3 \
  --max-model-len 1048576 \
  --stream-interval 10 \
  --enable-prefix-caching --prefix-match-unit 128 \
  --kv-cache-dtype fp8 --mamba-ssm-cache-dtype bfloat16 \
  --block-size 128 \
  --attention-backend ROCM_AITER_MLA \
  --attention-config '{"mla_prefill_backend":"ROCM_AITER_FA","use_prefill_query_quantization":true}' \
  --speculative-config '{"model":"/models/Inferact-Kimi-K3-DSpark","method":"dspark","num_speculative_tokens":4,"attention_backend":"ROCM_AITER_MLA","kv_cache_dtype":"fp8","draft_sample_method":"probabilistic","rejection_sample_method":"block"}' \
  --max-num-seqs "$MAX_SEQS" \
  --max-num-batched-tokens "$BATCHED" \
  --compilation-config "$COMPILATION" \
  --kv-transfer-config "$KV_CONFIG"

On the prefill worker, replace the example addresses, interface, and HCAs with your site's values:

export ROUTER_IP=192.0.2.10 WORKER_IP=192.0.2.11
export SOCKET_IFNAME=eth0 RDMA_DEVICES=rdma0,rdma1,rdma2,rdma3,rdma4,rdma5,rdma6,rdma7
bash kimi-k3-pd.sh prefill

On the decode worker:

export ROUTER_IP=192.0.2.10 WORKER_IP=192.0.2.12
export SOCKET_IFNAME=eth0 RDMA_DEVICES=rdma0,rdma1,rdma2,rdma3,rdma4,rdma5,rdma6,rdma7
bash kimi-k3-pd.sh decode

Prefill uses an 8,192-token batch budget without CUDA graphs; decode uses a 512-token budget with FULL_DECODE_ONLY graphs. Both roles use a 96-sequence limit and four DSpark draft tokens, with BF16 SSM. INT4 QuickReduce is a quantized collective optimization; validate accuracy for your workload and keep the same setting on all workers. The RDMA queue backoff is 50,000 microseconds (50 ms).

Prefill scratch reclamation is enabled with HSA_NO_SCRATCH_RECLAIM=0, matching the c48 configurations.

Scaling and concurrency tuning

For 1P2D or 1P3D, run the decode command on additional nodes with their own WORKER_IP. Discovery registers each worker automatically; the port ranges can be reused on separate hosts. These are router-balanced replicas, not a cross-node TP group.

For the offload configurations, let C be global client concurrency, N the number of decode workers, and K the draft length:

P max-num-seqs = 2 * C
D max-num-seqs = ceil(2 * C / N)
D graph max G = (K + 1) * C

These are tuning starting points; sequence limits provide scheduling headroom. Keep P/D SSM dtypes, DCP settings, and cache layouts matched.

TopologyGlobal concurrencyP / per-D max sequencesDraft tokensSSM stateP CPU offloadD graph dense / max
1P1D81 / 17FP32NoneExplicit [1, 8]
1P1D4896 / 964BF162399 GB128 / 240
1P2D2448 / 244BF162399 GB64 / 120
1P2D4896 / 484BF162399 GB128 / 240
1P3D2448 / 164BF162399 GB64 / 120

For c8, set both sequence limits to 1, use K7 and FP32 SSM, and remove prefill CPU offload (use MoRIIO alone). On decode only, set VLLM_USE_V2_MODEL_RUNNER=1, add --additional-config '{"decode_backend":"latency_specialized"}', and use graph sizes [1, 8] with maximum 8.

Benchmark concurrency sweeps and harness settings remain in the linked InferenceX configuration. Benchmark runs use its measured golden acceptance curve; the serving commands retain real block verification.

Verify routed inference

Wait for every worker to load and register with the router. Check each worker's /health endpoint and the router's /v1/models, then send a request through the router:

curl --fail-with-body "http://${ROUTER_IP}:8000/v1/chat/completions" \
  -H 'Content-Type: application/json' \
  -d '{"model":"moonshotai/Kimi-K3","messages":[{"role":"user","content":"What is 17 + 25?"}],"reasoning_effort":"low","temperature":0.6,"max_tokens":4096}'

Confirm that a final answer is returned, and inspect the prefill/decode logs for successful KV transfers and transfer errors. Health checks alone do not exercise RDMA. Preserve reasoning_content and tool calls when building multi-turn conversations, as described in the model card.