# zai-org/GLM-4.6 — vLLM serving recipe

> GLM-4.6 MoE language model (~357B total parameters, BF16) with MTP speculative decoding, native tool calling and reasoning

Configuration: hardware `h200` (8× H200) · strategy `single_node_tp` (Tensor Parallel) · variant `default` (BF16) · features `tool_calling`, `reasoning` · frontend `rust`

- Interactive page: https://recipes.vllm.ai/zai-org/GLM-4.6
- This configuration as JSON: https://recipes.vllm.ai/zai-org/GLM-4.6.json?hardware=h200

The commands below were rendered for exactly this configuration and take precedence over any example in the guide.

## Other configurations

Change the query string (the same parameters as the page URL) and fetch this URL again. Unknown or incompatible values fall back to the recipe default, so check the Configuration line.

- `hardware`: `h100`, `h200`, `b200`, `gb200`, `b300`, `gb300`, `mi300x`, `mi325x`, `mi350x`, `mi355x`
- `variant`: `default`, `fp8`, `redhat_fp8`, `redhat_fp8_plain`, `nvfp4`
- `strategy`: `single_node_tp`, `single_node_tep`, `single_node_dep`, `multi_node_tp`, `multi_node_tp_pp`, `multi_node_dep`, `multi_node_tep`, `pd_cluster` (multi-node strategies also need `nodes=<n>`)
- `features`: comma-separated list of `tool_calling`, `reasoning`, `spec_decoding` (default: `tool_calling`, `reasoning`)

## Install

Requires vLLM >= 0.11.0.

```bash
uv venv
source .venv/bin/activate
uv pip install -U vllm --torch-backend auto
```

Docker image: `vllm/vllm-openai:latest`

## Serve

```bash
export VLLM_USE_RUST_FRONTEND=1
vllm serve zai-org/GLM-4.6 \
  --trust-remote-code \
  --tensor-parallel-size 8 \
  --tool-call-parser glm45 \
  --enable-auto-tool-choice \
  --reasoning-parser glm45
```

Docker equivalent:

```bash
docker run --gpus all \
  --privileged --ipc=host -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -e VLLM_USE_RUST_FRONTEND=1 \
  vllm/vllm-openai:latest zai-org/GLM-4.6 \
  --trust-remote-code \
  --tensor-parallel-size 8 \
  --tool-call-parser glm45 \
  --enable-auto-tool-choice \
  --reasoning-parser glm45
```

## Guide

> Background reference. Commands in it are examples and may target another configuration — use the Serve section above.

### Overview

GLM-4.6 is the successor to GLM-4.5 with ~357B total parameters. It retains the
MoE architecture and built-in Multi-Token Prediction (MTP) layers used for
speculative decoding. FP8 is the recommended precision for cost-efficient
serving with minimal accuracy loss relative to BF16.

### Prerequisites

- **vLLM version:** >= 0.11.0 (latest stable recommended)
- **Hardware:** 8x H200 (BF16) or 4x-8x H200 (FP8), AMD MI300X / MI325X / MI355X for ROCm
- **Python:** 3.10 - 3.13 (3.12 required for ROCm wheels)

#### Install vLLM (NVIDIA)

```bash
uv venv
source .venv/bin/activate
uv pip install -U vllm --torch-backend auto
```

#### Install vLLM (AMD ROCm)

```bash
uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm
```

### Launching the Server

#### Tensor Parallel (FP8)

```bash
vllm serve zai-org/GLM-4.6-FP8 \
    --tensor-parallel-size 8 \
    --tool-call-parser glm45 \
    --reasoning-parser glm45 \
    --enable-auto-tool-choice
```

#### Enabling MTP Speculative Decoding

```bash
vllm serve zai-org/GLM-4.6-FP8 \
    --tensor-parallel-size 4 \
    --speculative-config.method mtp \
    --speculative-config.num_speculative_tokens 1 \
    --tool-call-parser glm45 \
    --reasoning-parser glm45 \
    --enable-auto-tool-choice
```

#### Tuning Tips

- `--max-model-len=65536` works well for most scenarios; max is 128K.
- `--max-num-batched-tokens=32768` is a good default for prompt-heavy workloads.
- `--gpu-memory-utilization=0.95` maximizes KV cache headroom.

### Client Usage

```python
from openai import OpenAI

client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1")
resp = client.chat.completions.create(
    model="zai-org/GLM-4.6-FP8",
    messages=[{"role": "user", "content": "Summarize MTP speculative decoding."}],
    max_tokens=512,
)
print(resp.choices[0].message.content)
```

### Benchmarking

```bash
vllm bench serve \
  --model zai-org/GLM-4.6-FP8 \
  --dataset-name random \
  --random-input-len 8000 \
  --random-output-len 1000 \
  --request-rate 10000 \
  --num-prompts 16 \
  --ignore-eos
```

### Troubleshooting

- **MTP memory overhead:** Monitor GPU memory and tune batch size when enabling MTP.
- **Tool calling not firing:** Ensure `--tool-call-parser glm45 --enable-auto-tool-choice`
  are both present.

### References

- [Model card](https://huggingface.co/zai-org/GLM-4.6)
- [FP8 checkpoint](https://huggingface.co/zai-org/GLM-4.6-FP8)
- [vLLM docs](https://docs.vllm.ai/)