# moonshotai/Kimi-K2.7-Code — vLLM serving recipe

> Coding-focused agentic MoE built on Kimi-K2.6, tuned for long-horizon software-engineering tasks with thinking-only reasoning, tool calling, and vision-language input

Configuration: hardware `h200` (8× H200) · strategy `single_node_tp` (Tensor Parallel) · variant `default` (INT4) · features `tool_calling`, `reasoning`, `spec_decoding`, `encoder_parallel`

- Interactive page: https://recipes.vllm.ai/moonshotai/Kimi-K2.7-Code
- This configuration as JSON: https://recipes.vllm.ai/moonshotai/Kimi-K2.7-Code.json?hardware=h200

The commands below were rendered for exactly this configuration and take precedence over any example in the guide.

## Other configurations

Change the query string (the same parameters as the page URL) and fetch this URL again. Unknown or incompatible values fall back to the recipe default, so check the Configuration line.

- `hardware`: `h100`, `h200`, `b200`, `gb200`, `b300`, `gb300`, `mi300x`, `mi325x`, `mi350x`, `mi355x`
- `variant`: `default`
- `strategy`: `single_node_tp`, `single_node_tep`, `single_node_dep`, `multi_node_tp`, `multi_node_tp_pp`, `multi_node_dep`, `multi_node_tep`, `pd_cluster` (multi-node strategies also need `nodes=<n>`)
- `features`: comma-separated list of `tool_calling`, `reasoning`, `spec_decoding`, `text_only`, `encoder_parallel` (default: `tool_calling`, `reasoning`, `spec_decoding`, `encoder_parallel`)

## Install

Requires vLLM >= 0.19.1.

```bash
uv venv
source .venv/bin/activate
uv pip install -U vllm --torch-backend auto
```

Docker image: `vllm/vllm-openai:latest`

## Serve

```bash
vllm serve moonshotai/Kimi-K2.7-Code \
  --trust-remote-code \
  --tensor-parallel-size 8 \
  --tool-call-parser kimi_k2 \
  --enable-auto-tool-choice \
  --reasoning-parser kimi_k2 \
  --speculative-config '{"model":"lightseekorg/kimi-k2.7-coder-eagle3.1-mla","method":"eagle3","num_speculative_tokens":3}' \
  --mm-encoder-tp-mode data
```

Docker equivalent:

```bash
docker run --gpus all \
  --privileged --ipc=host -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai:latest moonshotai/Kimi-K2.7-Code \
  --trust-remote-code \
  --tensor-parallel-size 8 \
  --tool-call-parser kimi_k2 \
  --enable-auto-tool-choice \
  --reasoning-parser kimi_k2 \
  --speculative-config '{"model":"lightseekorg/kimi-k2.7-coder-eagle3.1-mla","method":"eagle3","num_speculative_tokens":3}' \
  --mm-encoder-tp-mode data
```

## Guide

> Background reference. Commands in it are examples and may target another configuration — use the Serve section above.

### Overview

[Kimi-K2.7-Code](https://huggingface.co/moonshotai/Kimi-K2.7-Code) is a coding-focused
agentic model built on top of Kimi-K2.6. It substantially improves real-world
long-horizon coding performance — strengthening end-to-end task completion across complex
software-engineering workflows while improving token efficiency, reducing thinking-token
usage by ~30% versus Kimi-K2.6.

It shares the same architecture as Kimi-K2.5 / Kimi-K2.6 (1T-parameter MoE, 32B activated,
DeepSeek-V3 backbone with MLA attention and a MoonViT vision encoder), so the K2.6
deployment method can be reused directly. Kimi-K2.7-Code runs in **thinking mode only**
with `preserve_thinking` forced on; keep the reasoning parser enabled.

### Prerequisites

- **vLLM version:** >= 0.19.1 (manually verified). The model is also available in the
  vLLM nightly wheel, but nightly builds are experimental.
- **Hardware (INT4):** 8x H200 GPUs (verified), or equivalent aggregate VRAM (~640 GB)

```bash
uv venv
source .venv/bin/activate
uv pip install -U vllm --torch-backend=auto
```

### Launch command

Serve on a single H200 node with TP8 (vision encoder in data-parallel mode):

```bash
vllm serve moonshotai/Kimi-K2.7-Code \
  --tensor-parallel-size 8 \
  --mm-encoder-tp-mode data \
  --trust-remote-code \
  --tool-call-parser kimi_k2 \
  --enable-auto-tool-choice \
  --reasoning-parser kimi_k2
```

**Key notes**

- `--tool-call-parser kimi_k2` — required for enabling tool calling.
- `--reasoning-parser kimi_k2` — Kimi-K2.7-Code supports thinking mode only; pass this for
  correct reasoning processing.
- `--mm-encoder-tp-mode data` — runs the small MoonViT encoder data-parallel to avoid TP
  communication overhead.

#### AMD MI300X/MI325X

On 8x MI300X or MI325X (`gfx942`), use the standard W4A16 MoE path with AITER
and INT4 QuickReduce.

```bash
export VLLM_ROCM_USE_AITER=1
export VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4

vllm serve moonshotai/Kimi-K2.7-Code \
  --host 0.0.0.0 \
  --port 8000 \
  --trust-remote-code \
  --tensor-parallel-size 8 \
  --tool-call-parser kimi_k2 \
  --enable-auto-tool-choice \
  --reasoning-parser kimi_k2 \
  --mm-encoder-tp-mode data
```

#### AMD MI350X/MI355X

On 8x MI350X or MI355X (`gfx950`), add `--moe-backend flydsl` to use the
optimized FlyDSL W4A16 MoE kernel. Keep LoRA disabled for this path.

```bash
export VLLM_ROCM_USE_AITER=1
export VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4

vllm serve moonshotai/Kimi-K2.7-Code \
  --tensor-parallel-size 8 \
  --trust-remote-code \
  --mm-encoder-tp-mode data \
  --moe-backend flydsl \
  --tool-call-parser kimi_k2 \
  --enable-auto-tool-choice \
  --reasoning-parser kimi_k2 \
  --compilation-config '{"pass_config": {"fuse_allreduce_rms": false}}'
```

Notes:
- The FlyDSL INT4 MoE path does not support expert parallelism; do not add
  `--enable-expert-parallel`.
- Keep `--compilation-config '{"pass_config": {"fuse_allreduce_rms": false}}'`;
  it is required for this FlyDSL path on MI350X / MI355X.
- vLLM has tuned MI350X/MI355X FlyDSL configs for this Kimi shape at TP=8 and TP=4.
- Keep vLLM's default block size unless you are tuning long-context
  throughput; `--block-size 64` is safe to try.

### Client Usage

Once the vLLM server is running, consume it via the OpenAI-compatible API. The recommended
sampling for thinking mode is `temperature=1.0` and `top_p=0.95`.

```python
import time
from openai import OpenAI

client = OpenAI(
    api_key="EMPTY",
    base_url="http://localhost:8000/v1",
    timeout=3600,
)

messages = [
    {
        "role": "user",
        "content": [
            {
                "type": "image_url",
                "image_url": {
                    "url": "https://huggingface.co/moonshotai/Kimi-K2.7-Code/resolve/main/figures/kimi-logo.png"
                },
            },
            {"type": "text", "text": "Describe this image in detail."},
        ],
    }
]

start = time.time()
response = client.chat.completions.create(
    model="moonshotai/Kimi-K2.7-Code",
    messages=messages,
    max_tokens=4096,
    temperature=1.0,
    top_p=0.95,
)
print(f"Response costs: {time.time() - start:.2f}s")
print(f"Reasoning: {response.choices[0].message.reasoning}")
print(f"Response: {response.choices[0].message.content}")
```

### Troubleshooting

- **OOM errors:** Lower `--gpu-memory-utilization` or adjust TP/EP to match your GPU count.
- **Vision encoder performance:** Use `--mm-encoder-tp-mode data` to run the vision encoder
  in data-parallel mode. The encoder is small, so TP adds communication overhead with little gain.
- **Unique multimodal inputs:** Pass `--mm-processor-cache-gb 0` to avoid caching overhead.
  For repeated inputs, `--mm-processor-cache-type shm` uses host shared memory for better
  performance at high TP settings.
- **Text-only workloads:** Pass `--language-model-only` to skip loading MoonViT and free
  VRAM for KV cache.

### References

- [Kimi-K2.7-Code on Hugging Face](https://huggingface.co/moonshotai/Kimi-K2.7-Code)
- [Kimi-K2.7-Code deployment guide](https://huggingface.co/moonshotai/Kimi-K2.7-Code/blob/main/docs/deploy_guidance.md)
- [Kimi-K2.6 recipe](https://recipes.vllm.ai/moonshotai/Kimi-K2.6)
- [vLLM multimodal inputs guide](https://docs.vllm.ai/en/latest/features/multimodal_inputs.html)