# zai-org/GLM-4.5 — vLLM serving recipe

> GLM-4.5 MoE language model (~358B total parameters, BF16) with built-in MTP layers for speculative decoding and native tool calling

Configuration: hardware `h200` (8× H200) · strategy `single_node_tp` (Tensor Parallel) · variant `default` (BF16) · features `tool_calling`, `reasoning` · frontend `rust`

- Interactive page: https://recipes.vllm.ai/zai-org/GLM-4.5
- This configuration as JSON: https://recipes.vllm.ai/zai-org/GLM-4.5.json?hardware=h200

The commands below were rendered for exactly this configuration and take precedence over any example in the guide.

## Other configurations

Change the query string (the same parameters as the page URL) and fetch this URL again. Unknown or incompatible values fall back to the recipe default, so check the Configuration line.

- `hardware`: `h100`, `h200`, `b200`, `gb200`, `b300`, `gb300`, `mi300x`, `mi325x`, `mi350x`, `mi355x`
- `variant`: `default`, `fp8`
- `strategy`: `single_node_tp`, `single_node_tep`, `single_node_dep`, `multi_node_tp`, `multi_node_tp_pp`, `multi_node_dep`, `multi_node_tep`, `pd_cluster` (multi-node strategies also need `nodes=<n>`)
- `features`: comma-separated list of `tool_calling`, `reasoning`, `spec_decoding` (default: `tool_calling`, `reasoning`)

## Install

Requires vLLM >= 0.11.0.

```bash
uv venv
source .venv/bin/activate
uv pip install -U vllm --torch-backend auto
```

Docker image: `vllm/vllm-openai:latest`

## Serve

```bash
export VLLM_USE_RUST_FRONTEND=1
vllm serve zai-org/GLM-4.5 \
  --trust-remote-code \
  --tensor-parallel-size 8 \
  --tool-call-parser glm45 \
  --enable-auto-tool-choice \
  --reasoning-parser glm45
```

Docker equivalent:

```bash
docker run --gpus all \
  --privileged --ipc=host -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -e VLLM_USE_RUST_FRONTEND=1 \
  vllm/vllm-openai:latest zai-org/GLM-4.5 \
  --trust-remote-code \
  --tensor-parallel-size 8 \
  --tool-call-parser glm45 \
  --enable-auto-tool-choice \
  --reasoning-parser glm45
```

## Guide

> Background reference. Commands in it are examples and may target another configuration — use the Serve section above.

### Overview

GLM-4.5 is a Mixture-of-Experts language model from Z-AI with ~358B total parameters.
The checkpoint ships in both BF16 and native FP8 formats. FP8 models have minimal
accuracy loss, so unless you need strict reproducibility for benchmarking, FP8 is the
recommended precision for lower-cost serving. All GLM-4.X models include built-in
Multi-Token Prediction (MTP) layers that enable speculative decoding for higher
generation throughput.

### Prerequisites

- **vLLM version:** >= 0.11.0 (latest stable recommended)
- **Hardware:** 8x H200 (BF16) or 4x-8x H200 (FP8), AMD MI300X / MI325X / MI355X for ROCm
- **Python:** 3.10 - 3.13 (3.12 required for ROCm wheels)

#### Install vLLM (NVIDIA)

```bash
uv venv
source .venv/bin/activate
uv pip install -U vllm --torch-backend auto
```

#### Install vLLM (AMD ROCm)

```bash
uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm
```

The vLLM wheel for ROCm requires Python 3.12, ROCm 7.0, and glibc >= 2.35.

### Launching the Server

#### Tensor Parallel (FP8 on 8 GPUs)

```bash
vllm serve zai-org/GLM-4.5-FP8 \
    --tensor-parallel-size 8 \
    --tool-call-parser glm45 \
    --reasoning-parser glm45 \
    --enable-auto-tool-choice
```

#### Enabling MTP Speculative Decoding

```bash
vllm serve zai-org/GLM-4.5-FP8 \
    --tensor-parallel-size 4 \
    --speculative-config.method mtp \
    --speculative-config.num_speculative_tokens 1 \
    --tool-call-parser glm45 \
    --reasoning-parser glm45 \
    --enable-auto-tool-choice
```

Use `--speculative-config.num_speculative_tokens 1` for optimal throughput. Higher
values increase mean acceptance length but drop acceptance rate significantly.

#### Tuning Tips

- `--max-model-len=65536` works well for most scenarios; max is 128K.
- `--max-num-batched-tokens=32768` is a good default for prompt-heavy workloads.
  Reduce to 16K/8K to cut activation memory and latency.
- Set `--gpu-memory-utilization=0.95` to maximize KV cache headroom.

### Client Usage

```python
from openai import OpenAI

client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1")
resp = client.chat.completions.create(
    model="zai-org/GLM-4.5-FP8",
    messages=[{"role": "user", "content": "Explain MTP speculative decoding."}],
    max_tokens=512,
)
print(resp.choices[0].message.content)
```

### Benchmarking

Disable prefix caching with `--no-enable-prefix-caching` on the server command,
then:

```bash
vllm bench serve \
  --model zai-org/GLM-4.5-FP8 \
  --dataset-name random \
  --random-input-len 8000 \
  --random-output-len 1000 \
  --request-rate 10000 \
  --num-prompts 16 \
  --ignore-eos
```

### Troubleshooting

- **Tool calling not firing:** Ensure `--tool-call-parser glm45 --enable-auto-tool-choice`
  are both present.
- **MTP memory overhead:** MTP adds memory for draft computations. Monitor GPU memory
  and reduce `--max-model-len` or `--max-num-batched-tokens` if you OOM.
- **Low MTP acceptance:** If acceptance rate is below ~90%, drop speculative tokens to 1.

### References

- [Model card](https://huggingface.co/zai-org/GLM-4.5)
- [FP8 checkpoint](https://huggingface.co/zai-org/GLM-4.5-FP8)
- [vLLM docs](https://docs.vllm.ai/)