# moonshotai/Kimi-K3.1-Pro — vLLM serving recipe

> 2.8T-parameter native multimodal MoE with Kimi Delta Attention, sparse MLA, and a 1M-token context window

Configuration: hardware `gb300` (4× GB300 NVL4) · strategy `multi_node_tp` (Multi-Node Tensor Parallel) · 2 nodes · variant `default` (MXFP4) · features `tool_calling`, `reasoning`

- Interactive page: https://recipes.vllm.ai/moonshotai/Kimi-K3.1-Pro
- This configuration as JSON: https://recipes.vllm.ai/moonshotai/Kimi-K3.1-Pro.json?hardware=gb300

The commands below were rendered for exactly this configuration and take precedence over any example in the guide.

## Other configurations

Change the query string (the same parameters as the page URL) and fetch this URL again. Unknown or incompatible values fall back to the recipe default, so check the Configuration line.

- `hardware`: `b300`, `gb300`, `vr200`, `mi355x`, `810e`, `m890`
- `variant`: `default`
- `strategy`: `single_node_tp`, `single_node_tep`, `multi_node_tp`, `multi_node_tep`, `pd_cluster` (multi-node strategies also need `nodes=<n>`)
- `features`: comma-separated list of `tool_calling`, `reasoning`, `spec_decoding`, `text_only`, `low_latency_decode`, `nvfp4_kv_cache` (default: `tool_calling`, `reasoning`). Use `features=none` to turn them all off.
- `preset`: a tuned, evidence-backed configuration (`baseline` opts out of a default one). Each sets its own hardware and strategy:
  - `mi355x_agentx` — AgentX TP8 (mi355x · TP · Agentic (InferenceX AgentX) · opt-in): `hardware=mi355x&strategy=single_node_tp&preset=mi355x_agentx`

## Install

Requires vLLM >= 0.34.0 (not released yet — nightly wheel).

Docker image: `vllm/vllm-openai:kimi-k31`

## Serve

Run one command per node: the head (node rank 0) plus 1 worker. Each node's command differs — don't replay one on every node.

### Head (node 0)

```bash
export GLOO_SOCKET_IFNAME=$IFACE_NAME
export NCCL_SOCKET_IFNAME=$IFACE_NAME
export VLLM_ENGINE_READY_TIMEOUT_S=3600
vllm serve moonshotai/Kimi-K3.1-Pro \
  --trust-remote-code \
  --tensor-parallel-size 8 \
  --nnodes 2 \
  --node-rank 0 \
  --master-addr $HEAD_IP \
  --load-format fastsafetensors \
  --enable-auto-tool-choice \
  --tool-call-parser kimi_k3 \
  --reasoning-parser kimi_k3
```

### Node 1

```bash
export GLOO_SOCKET_IFNAME=$IFACE_NAME
export NCCL_SOCKET_IFNAME=$IFACE_NAME
export VLLM_ENGINE_READY_TIMEOUT_S=3600
vllm serve moonshotai/Kimi-K3.1-Pro \
  --trust-remote-code \
  --tensor-parallel-size 8 \
  --nnodes 2 \
  --node-rank 1 \
  --master-addr $HEAD_IP \
  --headless \
  --load-format fastsafetensors \
  --enable-auto-tool-choice \
  --tool-call-parser kimi_k3 \
  --reasoning-parser kimi_k3
```

Docker equivalent:

```bash
# Head
docker run --gpus all \
  --privileged --ipc=host -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -e GLOO_SOCKET_IFNAME=$IFACE_NAME \
  -e NCCL_SOCKET_IFNAME=$IFACE_NAME \
  -e VLLM_ENGINE_READY_TIMEOUT_S=3600 \
  vllm/vllm-openai:kimi-k31 moonshotai/Kimi-K3.1-Pro \
  --trust-remote-code \
  --tensor-parallel-size 8 \
  --nnodes 2 \
  --node-rank 0 \
  --master-addr $HEAD_IP \
  --load-format fastsafetensors \
  --enable-auto-tool-choice \
  --tool-call-parser kimi_k3 \
  --reasoning-parser kimi_k3
```

## Guide

> Background reference. Commands in it are examples and may target another configuration — use the Serve section above.

### Overview

Kimi-K3.1 keeps the Kimi-K3 layout (Kimi Delta Attention, 896 routed experts with 16 active
per token, a latent MoE projection and a vision encoder) and adds sparse
attention (DSA, top-2048 indexer) to 24 of its MLA layers. The checkpoint ships MXFP4
routed-expert weights; everything else is BF16.

The baseline is the simplest working setup on NVIDIA: one tensor-parallel group of 8 GPUs,
either two GB300 trays or one B300 node. Throughput was not tuned.

VR200 (preview) takes the same TP8 layout on two trays, in the `vllm/vllm-openai:kimi-k31-cu134`
image (CUDA 13.4). It uses the Blackwell flags and hasn't been tuned or verified on Rubin.

On MI355X, one node takes the same TP8 layout with the AMD profile's AITER environment, in the
day-0 `vllm/vllm-openai-rocm:kimi-k31` image. So far only the
[AgentX TP8](https://recipes.vllm.ai/moonshotai/Kimi-K3.1-Pro.md?hardware=mi355x&strategy=single_node_tp&preset=mi355x_agentx) configuration has run on MI355X: the tuned single-node AgentX
configuration for concurrency 48, with DCP8, the K3.1 DSpark draft and host KV offload.

On Zhenwu PPUs, the default MXFP4 checkpoint runs without speculative decoding on
two 16-PPU 810E nodes with TP32 / EP32, or four 8-PPU M890 nodes with TP8 / DP4 / EP32.
On 810E, `ppu_deep_gemm_w4a16` loads the checkpoint with a W4A16 compute path.
Use the PPU image specified for each hardware configuration.

### Tensor + expert parallelism

On one B300 node, the **Tensor + Expert Parallel** strategy keeps attention at TP8 and runs
the routed experts with expert parallelism on the MegaMoE kernel. It has the lowest
long-prompt time to first token of the single-node layouts: a single 340K-token prompt
prefills in about 10 s, against about 15 s at plain TP8 (8x B300, one request at a time,
pre-release checkpoint).

### Long context: decode context parallelism

The **Decode Context Parallelism** row shards the MLA latent and indexer-key caches across
the 8 GPUs of the TP group instead of replicating them, so DCP8 holds about 7x the KV cache
tokens of TP8 (about 18.5M vs 2.6M tokens with FP8 KV on 8x B300). Prefill is as fast as
at TP8. Decode steps are 7–25% slower than TP8 at the same batch, so use DCP when KV
capacity is the limit: very long contexts or many long-lived agentic sessions.

### Low-latency decode

The **Low-latency decode** feature switches decode steps to a fused CuTe DSL path tuned for
small batches; prefill and mixed steps still run the default path. On 8x B300 at TP8 it cuts
the decode step by 1.6–1.7x at batch 1 and 1.2–1.3x at batch 32–64. Under DCP8 it brings decode
to within 6% of TP8. It requires the V2 model runner and FULL decode CUDA graphs (the
defaults), TP8 or TP8 + DCP8, and no expert parallelism in this recipe.

### NVFP4 KV cache

The **NVFP4 KV cache** feature stores the MLA latent in NVFP4 (352 bytes per token per layer
instead of 576) and holds about 1.5x the KV cache tokens of FP8. The indexer keys stay FP8,
so the top-2048 token selection is unchanged. GSM8K matched FP8 on 8x GB300 (96.1% vs 96.2%);
long-context accuracy has not been measured yet.

### Prefill/decode disaggregation

The **Prefill/Decode Disaggregation** strategy runs one TP8 prefill pool (with expert
parallelism) and one DEP16 decode pool (TP1 x DP16 with expert parallelism), and decode
pulls each request's KV from prefill over NIXL. Every PD worker sets
`VLLM_SSM_CONV_STATE_LAYOUT=DS`: NIXL transfers the Kimi Delta Attention conv state in three
reads, which needs that layout, and the NIXL connector refuses to start without it.

### KV cache offloading

The **Simple** option on the KV Offload row adds `SimpleCPUOffloadConnector`, a host-DRAM
prefix tier for each GPU. Under Prefill/Decode Disaggregation it is added to the prefill pool
only, next to NIXL in a MultiConnector; decode keeps plain NIXL. The tier pins
`cpu_bytes_to_use_per_rank` of host memory for every GPU, so size it from the host DRAM that
is free after the engines are up. The tested GB300 prefill workers used 128 GiB per GPU
(512 GiB per tray).

### Client usage

```bash
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "moonshotai/Kimi-K3.1-Pro",
    "messages": [{"role": "user", "content": "Explain sparse attention in two sentences."}],
    "temperature": 0.7,
    "top_p": 0.95
  }'
```

### References

- [Model card](https://huggingface.co/moonshotai/Kimi-K3.1-Pro)
- [Kimi-K3 recipe](https://recipes.vllm.ai/moonshotai/Kimi-K3)