# zai-org/GLM-5.3 — vLLM serving recipe

> GLM-5.3 — Frontier Coding with Emergent Cyber Capabilities

Configuration: hardware `b300` (8× B300) · strategy `single_node_tp` (Tensor Parallel) · variant `default` (FP8) · features `tool_calling`, `reasoning` · frontend `rust`

- Interactive page: https://recipes.vllm.ai/zai-org/GLM-5.3
- This configuration as JSON: https://recipes.vllm.ai/zai-org/GLM-5.3.json?hardware=b300

The commands below were rendered for exactly this configuration and take precedence over any example in the guide.

## Other configurations

Change the query string (the same parameters as the page URL) and fetch this URL again. Unknown or incompatible values fall back to the recipe default, so check the Configuration line.

- `hardware`: `h100`, `h200`, `b200`, `gb200`, `b300`, `gb300`, `dgx_station_gb300`, `mi300x`, `mi325x`, `mi350x`, `mi355x`, `ascend_950pr_8x`
- `variant`: `default`, `nvfp4`, `w4a8`, `bf16`
- `strategy`: `single_node_tp`, `single_node_tep`, `multi_node_tp`, `multi_node_tp_pp`, `multi_node_tep`, `multi_node_dep`, `pd_cluster` (multi-node strategies also need `nodes=<n>`)
- `features`: comma-separated list of `tool_calling`, `reasoning`, `spec_decoding` (default: `tool_calling`, `reasoning`)

## Install

Requires vLLM >= 0.29.0.

Native GLM5Next rendering requires a build containing vllm-project/vllm#62424, including its Rust binary. The minimum version alone does not include this support.

```bash
uv venv
source .venv/bin/activate
uv pip install -U vllm --torch-backend auto
```

Docker image: `vllm/vllm-openai:latest`

## Serve

```bash
export VLLM_USE_RUST_FRONTEND=1
vllm serve zai-org/GLM-5.3 \
  --tokenizer-mode glm5_next \
  --kv-cache-dtype fp8_e4m3 \
  --tensor-parallel-size 8 \
  --tool-call-parser glm47 \
  --enable-auto-tool-choice \
  --reasoning-parser glm47
```

Docker equivalent:

```bash
docker run --gpus all \
  --privileged --ipc=host -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -e VLLM_USE_RUST_FRONTEND=1 \
  vllm/vllm-openai:latest zai-org/GLM-5.3 \
  --tokenizer-mode glm5_next \
  --kv-cache-dtype fp8_e4m3 \
  --tensor-parallel-size 8 \
  --tool-call-parser glm47 \
  --enable-auto-tool-choice \
  --reasoning-parser glm47
```

## Guide

> Background reference. Commands in it are examples and may target another configuration — use the Serve section above.

### Overview

**GLM-5.3** is the newest model in the GLM-5 series — a ~743B-parameter MoE (39B active) from **Z-AI**.
The **default `zai-org/GLM-5.3` checkpoint is native FP8** — BF16 weights live under the suffixed `zai-org/GLM-5.3-BF16` repo.

The FP8 checkpoint fits on a single 8xH200 / 8xH20 node and — with FP8 KV cache — reaches the full 1M-token context on 8xB200.
An NVFP4 checkpoint for Blackwell is published by Inferact (`Inferact/GLM-5.3-NVFP4`).

### Prerequisites

- **vLLM:** a build containing [native GLM5Next support (#62424)](https://github.com/vllm-project/vllm/pull/62424), including the matching Rust frontend binary.
- **Transformers:** 5.15.0 or newer.
- **GPU:** 8xH200 or 8xH20 (141 GB each) for single-node FP8; 8xB200 (180 GB each) for the full 1M context.

### Installation

Use a wheel or container built with #62424 for the Rust commands below.
The older release and hardware-specific image pins record prior model validation;
they need an updated build to use `--tokenizer-mode glm5_next`.

### Frontends

The command builder defaults to **Rust** and adds `--tokenizer-mode glm5_next`.
With `VLLM_USE_RUST_FRONTEND=1`, this selects native GLM5Next chat rendering for
GLM-5.3, including tool calls and reasoning options.

Select **Python** in the command builder, or set `VLLM_USE_RUST_FRONTEND=0`,
to use the Hugging Face renderer on the same build. On older builds that do not
recognize `glm5_next`, also replace `--tokenizer-mode glm5_next` with
`--tokenizer-mode auto`.

### Launching the server

#### FP8 on 8xH200 (standard)

```bash
VLLM_USE_RUST_FRONTEND=1 vllm serve zai-org/GLM-5.3 \
  --tokenizer-mode glm5_next \
  --kv-cache-dtype fp8 \
  --tensor-parallel-size 8 \
  --speculative-config.method mtp \
  --speculative-config.num_speculative_tokens 5 \
  --tool-call-parser glm47 \
  --reasoning-parser glm47 \
  --enable-auto-tool-choice \
  --served-model-name glm-5.3
```

#### FP8 on AMD MI300X/MI325X/MI350X/MI355X

```bash
VLLM_ROCM_USE_AITER=1 \
VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS=1 \
VLLM_USE_RUST_FRONTEND=1 \
vllm serve zai-org/GLM-5.3 \
  --tokenizer-mode glm5_next \
  --kv-cache-dtype fp8_e4m3 \
  --tensor-parallel-size 8 \
  --speculative-config.method mtp \
  --speculative-config.num_speculative_tokens 5 \
  --tool-call-parser glm47 \
  --enable-auto-tool-choice \
  --reasoning-parser glm47 \
  --gpu-memory-utilization 0.80 \
  --max-model-len 524288 \
  --max-num-seqs 32 \
  --linear-backend aiter \
  --moe-backend aiter
```

Earlier validation on 8x MI325X with `vllm/vllm-openai-rocm:v0.29.0` served
`--max-model-len 524288` at `--gpu-memory-utilization 0.80`, reporting a 1,477,168-token
KV cache and 2.82x maximum concurrency. MTP is healthy there: a mean acceptance length of
2.77 with 5 tokens drafted per step on a ~500-token greedy generation.

On 8x MI350X with the same image and settings, it reports a 2,013,072-token KV cache and
3.84x maximum concurrency, and MTP's mean acceptance length is 3.19 with 5 tokens drafted
per step on a 512-token greedy generation.

#### FP8 on 8xB200 (full 1M context)

GLM-5.3 has a native 1M-token window. Whether the full window fits is a KV-cache VRAM
question, so the lever is `--max-num-seqs` — it bounds how many sequences share the KV
budget at once. **Start at 32** and scale with your node's VRAM. FP8 KV cache
(`--kv-cache-dtype fp8_e4m3`, already in the base flags) roughly halves that budget.

```bash
VLLM_USE_RUST_FRONTEND=1 vllm serve zai-org/GLM-5.3 \
  --tokenizer-mode glm5_next \
  --kv-cache-dtype fp8_e4m3 \
  --tensor-parallel-size 8 \
  --speculative-config.method mtp \
  --speculative-config.num_speculative_tokens 5 \
  --max-num-seqs 32 \
  --tool-call-parser glm47 \
  --reasoning-parser glm47 \
  --enable-auto-tool-choice \
  --served-model-name glm-5.3
```

- **`--max-num-seqs 32`** — the single knob for fitting 1M context; start at 32 and tune it
  to your VRAM (up on headroom, down on OOM).
- **BF16** weights are served from `zai-org/GLM-5.3-BF16` and need multi-node deployment.

#### NVFP4 on Blackwell (B200/B300)

The `Inferact/GLM-5.3-NVFP4` variant is Inferact's NVFP4 re-quantization : only the MoE expert linears drop to NVFP4 while shared experts,
attention, embeddings, and the early dense layers stay BF16. The ~465 GB checkpoint fits comfortably on Blackwell
Select the **NVFP4** variant above (Blackwell-only) or run:

```bash
VLLM_USE_RUST_FRONTEND=1 vllm serve Inferact/GLM-5.3-NVFP4 \
  --tokenizer-mode glm5_next \
  --tensor-parallel-size 8 \
  --enable-expert-parallel \
  --reasoning-parser glm47 \
  --tool-call-parser glm47 \
  --enable-auto-tool-choice \
  --kv-cache-dtype fp8_e4m3 \
  --served-model-name glm-5.3-nvfp4
```

#### DSpark (opt-in)

DSpark is an external confidence-scheduled drafter. Serve `zai-org/GLM-5.3` and attach
[RedHatAI/GLM-5.3-speculator.dspark](https://huggingface.co/RedHatAI/GLM-5.3-speculator.dspark)
with eight speculative tokens:

```bash
VLLM_USE_RUST_FRONTEND=1 vllm serve zai-org/GLM-5.3 \
  --tokenizer-mode glm5_next \
  --tensor-parallel-size 8 \
  --speculative-config '{"method":"dspark","model":"RedHatAI/GLM-5.3-speculator.dspark","num_speculative_tokens":8}'
```

### Reasoning modes

Thinking is always on — the generation prompt opens a `<think>` block unconditionally.
GLM-5.3 offers **three reasoning effort levels** driven by the `reasoning_effort`
field; the default is **`max`**:

| Mode | How to request | Behavior |
|------|----------------|----------|
| **Think Max** (default) | omit `reasoning_effort`, or set `"max"` | Deepest reasoning — hard math, multi-step planning, agentic tasks. Highest token cost. |
| **Think High** | `"reasoning_effort": "high"` | Balanced depth and latency. |
| **Think Low** | `"reasoning_effort": "low"` | Lightest reasoning — simple Q&A, lowest latency and token cost. |

The chat template resolves effort to **`max` unless `reasoning_effort` is explicitly
`"low"` or `"high"`** (any other value falls back to `max`), then injects
`Reasoning Effort: Low|High|Max` into the system prompt. Pass it through
`chat_template_kwargs` or the top-level OpenAI `reasoning_effort` field.

```python
from openai import OpenAI

client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1")
msgs = [{"role": "user", "content": "Summarize GLM-5.3 in one sentence."}]

# Think Max (default) — just omit reasoning_effort
client.chat.completions.create(model="glm-5.3", messages=msgs, max_tokens=4096)

# Think High / Think Low — explicitly request the effort level
client.chat.completions.create(
    model="glm-5.3",
    messages=msgs,
    max_tokens=4096,
    extra_body={"chat_template_kwargs": {"reasoning_effort": "low"}},
)
```

```bash
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "glm-5.3",
    "messages": [{"role": "user", "content": "Summarize GLM-5.3 in one sentence."}],
    "temperature": 1,
    "max_tokens": 4096,
    "chat_template_kwargs": {"reasoning_effort": "high"}
  }'
```

### Client usage

```python
from openai import OpenAI

client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1")
response = client.chat.completions.create(
    model="glm-5.3",
    messages=[{"role": "user", "content": "Summarize GLM-5.3 in one sentence."}],
    temperature=1.0,
    max_tokens=256,
)
print(response.choices[0].message.content)
```

### Benchmarking

Add `--no-enable-prefix-caching` to the server command for a clean measurement.

```bash
vllm bench serve \
  --model zai-org/GLM-5.3 \
  --dataset-name random \
  --random-input-len 8000 \
  --random-output-len 1024 \
  --request-rate 10 \
  --num-prompts 32 \
  --ignore-eos
```

> **Note:** pure throughput benchmarks tend to under-report real speed, because MTP's
> acceptance rate is usually low in synthetic runs.

### Troubleshooting

- **FP8 performance:** DeepGEMM is required — install via `install_deepgemm.sh`.

### References

- [Model card](https://huggingface.co/zai-org/GLM-5.3)
- [BF16 checkpoint](https://huggingface.co/zai-org/GLM-5.3-BF16)
- [NVFP4 checkpoint (Inferact)](https://huggingface.co/Inferact/GLM-5.3-NVFP4)
- [DSpark draft](https://huggingface.co/RedHatAI/GLM-5.3-speculator.dspark)