# zai-org/GLM-4.7 — vLLM serving recipe

> GLM-4.7 MoE language model (~358B total parameters) with MTP speculative decoding, updated tool call parser, and reasoning support

Configuration: hardware `h200` (8× H200) · strategy `single_node_tp` (Tensor Parallel) · variant `default` (BF16) · features `tool_calling`, `reasoning` · frontend `rust`

- Interactive page: https://recipes.vllm.ai/zai-org/GLM-4.7
- This configuration as JSON: https://recipes.vllm.ai/zai-org/GLM-4.7.json?hardware=h200

The commands below were rendered for exactly this configuration and take precedence over any example in the guide.

## Other configurations

Change the query string (the same parameters as the page URL) and fetch this URL again. Unknown or incompatible values fall back to the recipe default, so check the Configuration line.

- `hardware`: `h100`, `h200`, `b200`, `gb200`, `b300`, `gb300`, `mi300x`, `mi325x`, `mi350x`, `mi355x`
- `variant`: `default`, `fp8`, `nvfp4`
- `strategy`: `single_node_tp`, `single_node_tep`, `single_node_dep`, `multi_node_tp`, `multi_node_tp_pp`, `multi_node_dep`, `multi_node_tep`, `pd_cluster` (multi-node strategies also need `nodes=<n>`)
- `features`: comma-separated list of `tool_calling`, `reasoning`, `spec_decoding` (default: `tool_calling`, `reasoning`)

## Install

Requires vLLM >= 0.24.0.

```bash
uv venv
source .venv/bin/activate
uv pip install -U vllm --torch-backend auto
```

Docker image: `vllm/vllm-openai:latest`

Extra dependencies:

- GLM-4.7 requires the nightly vllm wheel

```bash
uv pip install -U vllm --pre --extra-index-url https://wheels.vllm.ai/nightly
```

- transformers from source — GLM-4.7 tokenizer is newer than any release

```bash
uv pip install git+https://github.com/huggingface/transformers.git
```

## Serve

```bash
export VLLM_USE_RUST_FRONTEND=1
vllm serve zai-org/GLM-4.7 \
  --trust-remote-code \
  --tensor-parallel-size 8 \
  --tool-call-parser glm47 \
  --enable-auto-tool-choice \
  --reasoning-parser glm47
```

Docker equivalent:

```bash
docker run --gpus all \
  --privileged --ipc=host -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -e VLLM_USE_RUST_FRONTEND=1 \
  vllm/vllm-openai:latest zai-org/GLM-4.7 \
  --trust-remote-code \
  --tensor-parallel-size 8 \
  --tool-call-parser glm47 \
  --enable-auto-tool-choice \
  --reasoning-parser glm47
```

## Guide

> Background reference. Commands in it are examples and may target another configuration — use the Serve section above.

### Overview

GLM-4.7 is the latest GLM-4.X MoE release from Z-AI. It introduces the `glm47`
tool call parser and uses the matching `glm47` reasoning parser. Built-in
Multi-Token Prediction (MTP) layers enable speculative decoding for throughput
gains on decode-heavy workloads.

A smaller `zai-org/GLM-4.7-Flash` variant is also available for lower-latency
scenarios.

### Prerequisites

- **vLLM version:** nightly recommended for GLM-4.7 (until packaged in a stable release)
- **Hardware:** 4x-8x H200 (FP8), AMD MI300X / MI325X / MI355X for ROCm
- **Python:** 3.10 - 3.13 (3.12 required for ROCm wheels)

#### Install vLLM (NVIDIA, nightly)

```bash
uv venv
source .venv/bin/activate
uv pip install -U vllm --pre --extra-index-url https://wheels.vllm.ai/nightly
uv pip install git+https://github.com/huggingface/transformers.git
```

#### Install vLLM (AMD ROCm)

```bash
uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm
```

### Launching the Server

#### Tensor Parallel + MTP (FP8 on 4xH200)

```bash
vllm serve zai-org/GLM-4.7-FP8 \
    --tensor-parallel-size 4 \
    --speculative-config.method mtp \
    --speculative-config.num_speculative_tokens 1 \
    --tool-call-parser glm47 \
    --reasoning-parser glm47 \
    --enable-auto-tool-choice
```

#### AMD ROCm

```bash
SAFETENSORS_FAST_GPU=1 \
vllm serve zai-org/GLM-4.7 \
    --tensor-parallel-size 8 \
    --gpu-memory-utilization 0.9 \
    --no-enable-prefix-caching \
    --trust-remote-code
```

#### Tuning Tips

- `--max-model-len=65536` is a sensible default; max is 128K.
- `--max-num-batched-tokens=32768` for prompt-heavy workloads; reduce to 8K-16K for latency-sensitive.
- Use `--gpu-memory-utilization=0.95` to maximize KV cache.

### Client Usage

```python
from openai import OpenAI

client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1")
resp = client.chat.completions.create(
    model="zai-org/GLM-4.7-FP8",
    messages=[{"role": "user", "content": "Hello!"}],
    max_tokens=512,
)
print(resp.choices[0].message.content)
```

### Benchmarking

```bash
vllm bench serve \
  --model zai-org/GLM-4.7-FP8 \
  --dataset-name random \
  --random-input-len 8000 \
  --random-output-len 1000 \
  --request-rate 10000 \
  --num-prompts 16 \
  --ignore-eos
```

### Troubleshooting

- **Parser mismatch:** GLM-4.7 uses `--tool-call-parser glm47` and `--reasoning-parser glm47`.
- **MTP acceptance:** 1 speculative token gives ~90%+ acceptance and best throughput.

### References

- [Model card](https://huggingface.co/zai-org/GLM-4.7)
- [FP8 checkpoint](https://huggingface.co/zai-org/GLM-4.7-FP8)
- [GLM-4.7-Flash](https://huggingface.co/zai-org/GLM-4.7-Flash)
- [vLLM docs](https://docs.vllm.ai/)