# zai-org/GLM-5.1 — vLLM serving recipe

> GLM-5.1 refreshed version of GLM-5 — frontier-scale MoE language model (~744B total parameters) with MTP speculative decoding and thinking mode

Configuration: hardware `h200` (8× H200) · strategy `multi_node_tp` (Multi-Node Tensor Parallel) · 2 nodes · variant `default` (BF16) · features `tool_calling`, `reasoning` · frontend `rust`

- Interactive page: https://recipes.vllm.ai/zai-org/GLM-5.1
- This configuration as JSON: https://recipes.vllm.ai/zai-org/GLM-5.1.json?hardware=h200

The commands below were rendered for exactly this configuration and take precedence over any example in the guide.

## Other configurations

Change the query string (the same parameters as the page URL) and fetch this URL again. Unknown or incompatible values fall back to the recipe default, so check the Configuration line.

- `hardware`: `h100`, `h200`, `b200`, `gb200`, `b300`, `gb300`, `mi300x`, `mi325x`, `mi350x`, `mi355x`
- `variant`: `default`, `fp8`, `nvfp4`, `mxfp4`
- `strategy`: `single_node_tp`, `single_node_tep`, `multi_node_tp`, `multi_node_tp_pp`, `multi_node_tep`, `multi_node_dep`, `pd_cluster` (multi-node strategies also need `nodes=<n>`)
- `features`: comma-separated list of `tool_calling`, `reasoning`, `spec_decoding` (default: `tool_calling`, `reasoning`)

## Install

Requires vLLM >= 0.24.0.

```bash
uv venv
source .venv/bin/activate
uv pip install -U vllm --torch-backend auto
```

Docker image: `vllm/vllm-openai:latest`

## Serve

Run one command per node: the head (node rank 0) plus 1 worker. Each node's command differs — don't replay one on every node.

### Head (node 0)

```bash
export GLOO_SOCKET_IFNAME=$IFACE_NAME
export NCCL_SOCKET_IFNAME=$IFACE_NAME
export VLLM_USE_RUST_FRONTEND=1
vllm serve zai-org/GLM-5.1 \
  --trust-remote-code \
  --chat-template-content-format=string \
  --tensor-parallel-size 16 \
  --nnodes 2 \
  --node-rank 0 \
  --master-addr $HEAD_IP \
  --tool-call-parser glm47 \
  --enable-auto-tool-choice \
  --reasoning-parser glm47
```

### Node 1

```bash
export GLOO_SOCKET_IFNAME=$IFACE_NAME
export NCCL_SOCKET_IFNAME=$IFACE_NAME
export VLLM_USE_RUST_FRONTEND=1
vllm serve zai-org/GLM-5.1 \
  --trust-remote-code \
  --chat-template-content-format=string \
  --tensor-parallel-size 16 \
  --nnodes 2 \
  --node-rank 1 \
  --master-addr $HEAD_IP \
  --headless \
  --tool-call-parser glm47 \
  --enable-auto-tool-choice \
  --reasoning-parser glm47
```

Docker equivalent:

```bash
# Head
docker run --gpus all \
  --privileged --ipc=host -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -e GLOO_SOCKET_IFNAME=$IFACE_NAME \
  -e NCCL_SOCKET_IFNAME=$IFACE_NAME \
  -e VLLM_USE_RUST_FRONTEND=1 \
  vllm/vllm-openai:latest zai-org/GLM-5.1 \
  --trust-remote-code \
  --chat-template-content-format=string \
  --tensor-parallel-size 16 \
  --nnodes 2 \
  --node-rank 0 \
  --master-addr $HEAD_IP \
  --tool-call-parser glm47 \
  --enable-auto-tool-choice \
  --reasoning-parser glm47
```

## Guide

> Background reference. Commands in it are examples and may target another configuration — use the Serve section above.

### Overview

GLM-5.1 is a refreshed version of GLM-5, the 744B parameter frontier MoE model from Z-AI.
It keeps the asynchronous RL training recipe and delivers best-in-class open-source
performance on reasoning, coding, and agentic benchmarks. Both BF16 and native FP8
checkpoints are published.

Thinking mode is enabled by default; disable it by passing
`"chat_template_kwargs": {"enable_thinking": false}` in request extras.

### Prerequisites

- **vLLM version:** >= 0.24.0 for the `glm47` reasoning parser.
  Use the latest main branch if you need tool calling + MTP simultaneously.
- **Hardware (FP8):** 8xH200 or 8xH20 (141GB × 8)
- **DeepGEMM (FP8, NVIDIA):** install via `install_deepgemm.sh` from vLLM repo
- **Hardware (FP8, AMD):** 8x MI325X (256GB × 8) or 8x MI350X (288GB × 8), `vllm/vllm-openai-rocm:nightly-92044241a02f05de51420d654daf669de20b4691`; uses AITER, not DeepGEMM
- **Hardware (MXFP4):** 8x MI355X (gfx950), vLLM >= 0.24.0, `vllm/vllm-openai-rocm:latest`

#### Using Docker

```bash
docker run --gpus all \
  -p 8000:8000 \
  --ipc=host \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai:latest zai-org/GLM-5.1-FP8 \
    --tensor-parallel-size 8 \
    --tool-call-parser glm47 \
    --reasoning-parser glm47 \
    --enable-auto-tool-choice \
    --chat-template-content-format=string \
    --served-model-name glm-5.1-fp8
```

Use `vllm/vllm-openai:latest-cu129` for CUDA 12.x.

#### Install vLLM from Source

```bash
uv venv
source .venv/bin/activate
uv pip install "vllm==0.24.0" --torch-backend=auto
uv pip install "transformers>=5.4.0"
```

### Launching the Server

#### FP8 on 8xH200 with MTP

```bash
vllm serve zai-org/GLM-5.1-FP8 \
     --tensor-parallel-size 8 \
     --speculative-config.method mtp \
     --speculative-config.num_speculative_tokens 3 \
     --tool-call-parser glm47 \
     --reasoning-parser glm47 \
     --enable-auto-tool-choice \
     --chat-template-content-format=string \
     --served-model-name glm-5.1-fp8
```

#### FP8 on AMD MI325X and MI350X

Select **MI325X** or **MI350X** and the **FP8** variant in the command builder above. The
generated command adds `VLLM_ROCM_USE_AITER=1`, `VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS=1`,
`--linear-backend aiter` and `--moe-backend aiter`, and raises `VLLM_ENGINE_READY_TIMEOUT_S`
because loading the weights can take longer than the Rust frontend's 600 s default. The
Docker command uses the nightly build
`vllm/vllm-openai-rocm:nightly-92044241a02f05de51420d654daf669de20b4691`.

On 8x MI325X that command, with MTP enabled, serves the full 202,752-token context with a
1,292,544-token KV cache (6.38x maximum concurrency) and an MTP mean acceptance length of
2.72 with 3 draft tokens on a ~380-token greedy generation.

On 8x MI350X the same command serves the full context with a 1,587,312-token KV cache (7.83x
maximum concurrency) and an MTP mean acceptance length of 3.13 with 3 draft tokens on a
512-token greedy generation.

### Client Usage

```python
from openai import OpenAI

client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1")

# Thinking ON (default)
resp_on = client.chat.completions.create(
    model="glm-5.1-fp8",
    messages=[{"role": "user", "content": "Summarize GLM-5.1 in one sentence."}],
    temperature=1,
    max_tokens=4096,
)
print(resp_on.choices[0].message.reasoning)

# Thinking OFF
resp_off = client.chat.completions.create(
    model="glm-5.1-fp8",
    messages=[{"role": "user", "content": "Summarize GLM-5.1 in one sentence."}],
    temperature=1,
    max_tokens=4096,
    extra_body={"chat_template_kwargs": {"enable_thinking": False}},
)
```

#### cURL (Thinking ON)

```bash
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "glm-5.1-fp8",
    "messages": [
      {"role": "user", "content": "Summarize GLM-5.1 in one sentence."}
    ],
    "temperature": 1,
    "max_tokens": 4096
  }'
```

### Benchmarking

```bash
vllm bench serve \
  --model zai-org/GLM-5.1-FP8 \
  --dataset-name random \
  --random-input 8000 \
  --random-output 1024 \
  --request-rate 10 \
  --num-prompts 32 \
  --ignore-eos
```

### Troubleshooting

- **Accuracy drift:** Prefer the 0.19.0 stable release for best accuracy.
- **Tool calling + MTP:** If both are needed, use the latest vLLM main branch.
- **FP8 installation (NVIDIA):** DeepGEMM required for FP8 performance.
- **MXFP4 (MI355X):** MTP speculative decoding isn't yet supported on this checkpoint/kernel path. Mooncake and LMCache KV-store offload are still untested on MI355X — worth a check before leaning on them in production.

### References

- [Model card](https://huggingface.co/zai-org/GLM-5.1)
- [FP8 checkpoint](https://huggingface.co/zai-org/GLM-5.1-FP8)
- [MXFP4 checkpoint](https://huggingface.co/amd/GLM-5.1-MXFP4)
- [DeepGEMM install script](https://github.com/vllm-project/vllm/blob/v0.16.0rc0/tools/install_deepgemm.sh)