# zai-org/GLM-5 — vLLM serving recipe

> GLM-5 frontier-scale MoE language model (~744B total parameters, 28.5T training tokens) with asynchronous RL infrastructure for reasoning, coding, and agentic tasks

Configuration: hardware `h200` (8× H200) · strategy `multi_node_tp` (Multi-Node Tensor Parallel) · 2 nodes · variant `default` (BF16) · features `tool_calling`, `reasoning` · frontend `rust`

- Interactive page: https://recipes.vllm.ai/zai-org/GLM-5
- This configuration as JSON: https://recipes.vllm.ai/zai-org/GLM-5.json?hardware=h200

The commands below were rendered for exactly this configuration and take precedence over any example in the guide.

## Other configurations

Change the query string (the same parameters as the page URL) and fetch this URL again. Unknown or incompatible values fall back to the recipe default, so check the Configuration line.

- `hardware`: `h100`, `h200`, `b200`, `gb200`, `b300`, `gb300`, `mi300x`, `mi325x`, `mi350x`, `mi355x`
- `variant`: `default`, `fp8`, `nvfp4`
- `strategy`: `single_node_tp`, `single_node_tep`, `multi_node_tp`, `multi_node_tp_pp`, `multi_node_tep`, `multi_node_dep`, `pd_cluster` (multi-node strategies also need `nodes=<n>`)
- `features`: comma-separated list of `tool_calling`, `reasoning`, `spec_decoding` (default: `tool_calling`, `reasoning`)

## Install

Requires vLLM >= 0.24.0.

```bash
uv venv
source .venv/bin/activate
uv pip install -U vllm --torch-backend auto
```

Docker image: `vllm/vllm-openai:latest`

## Serve

Run one command per node: the head (node rank 0) plus 1 worker. Each node's command differs — don't replay one on every node.

### Head (node 0)

```bash
export GLOO_SOCKET_IFNAME=$IFACE_NAME
export NCCL_SOCKET_IFNAME=$IFACE_NAME
export VLLM_USE_RUST_FRONTEND=1
vllm serve zai-org/GLM-5 \
  --trust-remote-code \
  --chat-template-content-format=string \
  --tensor-parallel-size 16 \
  --nnodes 2 \
  --node-rank 0 \
  --master-addr $HEAD_IP \
  --tool-call-parser glm47 \
  --enable-auto-tool-choice \
  --reasoning-parser glm47
```

### Node 1

```bash
export GLOO_SOCKET_IFNAME=$IFACE_NAME
export NCCL_SOCKET_IFNAME=$IFACE_NAME
export VLLM_USE_RUST_FRONTEND=1
vllm serve zai-org/GLM-5 \
  --trust-remote-code \
  --chat-template-content-format=string \
  --tensor-parallel-size 16 \
  --nnodes 2 \
  --node-rank 1 \
  --master-addr $HEAD_IP \
  --headless \
  --tool-call-parser glm47 \
  --enable-auto-tool-choice \
  --reasoning-parser glm47
```

Docker equivalent:

```bash
# Head
docker run --gpus all \
  --privileged --ipc=host -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -e GLOO_SOCKET_IFNAME=$IFACE_NAME \
  -e NCCL_SOCKET_IFNAME=$IFACE_NAME \
  -e VLLM_USE_RUST_FRONTEND=1 \
  vllm/vllm-openai:latest zai-org/GLM-5 \
  --trust-remote-code \
  --chat-template-content-format=string \
  --tensor-parallel-size 16 \
  --nnodes 2 \
  --node-rank 0 \
  --master-addr $HEAD_IP \
  --tool-call-parser glm47 \
  --enable-auto-tool-choice \
  --reasoning-parser glm47
```

## Guide

> Background reference. Commands in it are examples and may target another configuration — use the Serve section above.

### Overview

GLM-5 is a significantly scaled-up language model with 744B parameters, trained on
28.5T tokens using novel asynchronous RL infrastructure. It delivers best-in-class
open-source performance on reasoning, coding, and agentic tasks, rivaling frontier
closed-source models. GLM-5 is available in both BF16 and native FP8 precisions.

Thinking mode is enabled by default; disable it by passing
`"chat_template_kwargs": {"enable_thinking": false}` in request extras.

### Prerequisites

- **vLLM version:** >= 0.24.0 for the `glm47` reasoning parser
- **Hardware (FP8):** 8xH200 or 8xH20 (141GB × 8)
- **DeepGEMM (FP8, NVIDIA):** install via `install_deepgemm.sh` from the vLLM repo
- **Hardware (FP8, AMD):** 8x MI325X (256GB × 8) or 8x MI350X (288GB × 8), `vllm/vllm-openai-rocm:nightly-0cbac6cd1305f710e12193596b27488397bcb205`; uses AITER, not DeepGEMM

#### Using Docker

```bash
docker run --gpus all \
  -p 8000:8000 \
  --ipc=host \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai:latest zai-org/GLM-5-FP8 \
    --tensor-parallel-size 8 \
    --tool-call-parser glm47 \
    --reasoning-parser glm47 \
    --enable-auto-tool-choice \
    --chat-template-content-format=string
```

Use `vllm/vllm-openai:latest-cu129` for CUDA 12.x.

#### Install vLLM from Source

```bash
uv venv
source .venv/bin/activate
uv pip install "vllm==0.24.0" --torch-backend=auto
uv pip install "transformers>=5.4.0"
```

On NVIDIA, install DeepGEMM using `install_deepgemm.sh` from the vLLM tools directory.

### Launching the Server

#### FP8 on 8xH200 with MTP

```bash
vllm serve zai-org/GLM-5-FP8 \
     --tensor-parallel-size 8 \
     --speculative-config.method mtp \
     --speculative-config.num_speculative_tokens 3 \
     --tool-call-parser glm47 \
     --reasoning-parser glm47 \
     --enable-auto-tool-choice \
     --chat-template-content-format=string \
     --served-model-name glm-5-fp8
```

#### FP8 on AMD MI325X and MI350X

Select **MI325X** or **MI350X** and the **FP8** variant in the command builder above. The
generated command adds `VLLM_ROCM_USE_AITER=1`, `VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS=1`,
`--linear-backend aiter` and `--moe-backend aiter`, and raises `VLLM_ENGINE_READY_TIMEOUT_S`
because loading the weights can take longer than the Rust frontend's 600 s default. The
Docker command uses the nightly build
`vllm/vllm-openai-rocm:nightly-0cbac6cd1305f710e12193596b27488397bcb205`.

On 8x MI325X that command, with MTP enabled, serves the full 202,752-token context with a
1,292,544-token KV cache (6.38x maximum concurrency) and an MTP mean acceptance length of
2.74 with 3 draft tokens on a ~380-token greedy generation.

On 8x MI350X the same command serves the full context with a 1,587,312-token KV cache (7.83x
maximum concurrency) and an MTP mean acceptance length of 2.95 with 3 draft tokens on a
512-token greedy generation.

### Client Usage

```python
from openai import OpenAI

client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1")

# Thinking ON (default)
resp_on = client.chat.completions.create(
    model="glm-5-fp8",
    messages=[{"role": "user", "content": "Summarize GLM-5 in one sentence."}],
    temperature=1,
    max_tokens=4096,
)
print("thinking=on, think content:\n", resp_on.choices[0].message.reasoning)

# Thinking OFF
resp_off = client.chat.completions.create(
    model="glm-5-fp8",
    messages=[{"role": "user", "content": "Summarize GLM-5 in one sentence."}],
    temperature=1,
    max_tokens=4096,
    extra_body={"chat_template_kwargs": {"enable_thinking": False}},
)
print("thinking=off:\n", resp_off.choices[0].message.reasoning)
```

### Benchmarking

```bash
vllm bench serve \
  --model zai-org/GLM-5-FP8 \
  --dataset-name random \
  --random-input 8000 \
  --random-output 1024 \
  --request-rate 10 \
  --num-prompts 32 \
  --ignore-eos
```

The MTP acceptance rate can be relatively low in pure benchmarks; measured throughput
may underestimate real-world speed.

### Troubleshooting

- **Accuracy drift:** Prefer the 0.19.0 stable release over nightly for best accuracy.
- **Tool calling + MTP:** If you need both, use the latest vLLM main branch.
- **FP8 installation (NVIDIA):** DeepGEMM is required for FP8 performance.

### References

- [Model card](https://huggingface.co/zai-org/GLM-5)
- [FP8 checkpoint](https://huggingface.co/zai-org/GLM-5-FP8)
- [DeepGEMM install script](https://github.com/vllm-project/vllm/blob/v0.16.0rc0/tools/install_deepgemm.sh)