{ "hf_id": "zai-org/GLM-5", "meta": { "title": "GLM-5", "slug": "glm-5", "provider": "GLM (Z-AI)", "description": "GLM-5 frontier-scale MoE language model (~744B total parameters, 28.5T training tokens) with asynchronous RL infrastructure for reasoning, coding, and agentic tasks", "date_added": "2026-02-09", "date_updated": "2026-10-05", "tasks": [ "text" ], "performance_headline": "Frontier-scale MoE with 744B parameters, best-in-class open-source performance on reasoning/coding/agents", "related_recipes": [], "hardware": { "h200": "verified", "mi325x": "verified", "mi350x": "verified" } }, "recommended_command": { "hardware": "h200", "strategy": "multi_node_tp", "variant": "default", "node_count": 2, "deploy_type": "multi_node", "env": { "GLOO_SOCKET_IFNAME": "$IFACE_NAME", "NCCL_SOCKET_IFNAME": "$IFACE_NAME", "VLLM_USE_RUST_FRONTEND": "1" }, "docker_image": "vllm/vllm-openai:latest", "head_command": "vllm serve zai-org/GLM-5 \\\n --trust-remote-code \\\n --chat-template-content-format=string \\\n --tensor-parallel-size 16 \\\n --nnodes 2 \\\n --node-rank 0 \\\n --master-addr $HEAD_IP \\\n --tool-call-parser glm47 \\\n --enable-auto-tool-choice \\\n --reasoning-parser glm47", "worker_command": "vllm serve zai-org/GLM-5 \\\n --trust-remote-code \\\n --chat-template-content-format=string \\\n --tensor-parallel-size 16 \\\n --nnodes 2 \\\n --node-rank 1 \\\n --master-addr $HEAD_IP \\\n --headless \\\n --tool-call-parser glm47 \\\n --enable-auto-tool-choice \\\n --reasoning-parser glm47", "worker_commands": [ "vllm serve zai-org/GLM-5 \\\n --trust-remote-code \\\n --chat-template-content-format=string \\\n --tensor-parallel-size 16 \\\n --nnodes 2 \\\n --node-rank 1 \\\n --master-addr $HEAD_IP \\\n --headless \\\n --tool-call-parser glm47 \\\n --enable-auto-tool-choice \\\n --reasoning-parser glm47" ], "head_argv": [ "vllm", "serve", "zai-org/GLM-5", "--trust-remote-code", "--chat-template-content-format=string", "--tensor-parallel-size", "16", "--nnodes", "2", "--node-rank", "0", "--master-addr", "$HEAD_IP", "--tool-call-parser", "glm47", "--enable-auto-tool-choice", "--reasoning-parser", "glm47" ], "worker_argv": [ "vllm", "serve", "zai-org/GLM-5", "--trust-remote-code", "--chat-template-content-format=string", "--tensor-parallel-size", "16", "--nnodes", "2", "--node-rank", "1", "--master-addr", "$HEAD_IP", "--headless", "--tool-call-parser", "glm47", "--enable-auto-tool-choice", "--reasoning-parser", "glm47" ], "worker_argvs": [ [ "vllm", "serve", "zai-org/GLM-5", "--trust-remote-code", "--chat-template-content-format=string", "--tensor-parallel-size", "16", "--nnodes", "2", "--node-rank", "1", "--master-addr", "$HEAD_IP", "--headless", "--tool-call-parser", "glm47", "--enable-auto-tool-choice", "--reasoning-parser", "glm47" ] ], "head_docker_command": "docker run --gpus all \\\n --privileged --ipc=host -p 8000:8000 \\\n -v ~/.cache/huggingface:/root/.cache/huggingface \\\n -e GLOO_SOCKET_IFNAME=$IFACE_NAME \\\n -e NCCL_SOCKET_IFNAME=$IFACE_NAME \\\n -e VLLM_USE_RUST_FRONTEND=1 \\\n vllm/vllm-openai:latest zai-org/GLM-5 \\\n --trust-remote-code \\\n --chat-template-content-format=string \\\n --tensor-parallel-size 16 \\\n --nnodes 2 \\\n --node-rank 0 \\\n --master-addr $HEAD_IP \\\n --tool-call-parser glm47 \\\n --enable-auto-tool-choice \\\n --reasoning-parser glm47", "worker_docker_command": "docker run --gpus all \\\n --privileged --ipc=host -p 8000:8000 \\\n -v ~/.cache/huggingface:/root/.cache/huggingface \\\n -e GLOO_SOCKET_IFNAME=$IFACE_NAME \\\n -e NCCL_SOCKET_IFNAME=$IFACE_NAME \\\n -e VLLM_USE_RUST_FRONTEND=1 \\\n vllm/vllm-openai:latest zai-org/GLM-5 \\\n --trust-remote-code \\\n --chat-template-content-format=string \\\n --tensor-parallel-size 16 \\\n --nnodes 2 \\\n --node-rank 1 \\\n --master-addr $HEAD_IP \\\n --headless \\\n --tool-call-parser glm47 \\\n --enable-auto-tool-choice \\\n --reasoning-parser glm47", "worker_docker_commands": [ "docker run --gpus all \\\n --privileged --ipc=host -p 8000:8000 \\\n -v ~/.cache/huggingface:/root/.cache/huggingface \\\n -e GLOO_SOCKET_IFNAME=$IFACE_NAME \\\n -e NCCL_SOCKET_IFNAME=$IFACE_NAME \\\n -e VLLM_USE_RUST_FRONTEND=1 \\\n vllm/vllm-openai:latest zai-org/GLM-5 \\\n --trust-remote-code \\\n --chat-template-content-format=string \\\n --tensor-parallel-size 16 \\\n --nnodes 2 \\\n --node-rank 1 \\\n --master-addr $HEAD_IP \\\n --headless \\\n --tool-call-parser glm47 \\\n --enable-auto-tool-choice \\\n --reasoning-parser glm47" ], "head_docker_argv": [ "docker", "run", "--gpus", "all", "--privileged", "--ipc=host", "-p", "8000:8000", "-v", "~/.cache/huggingface:/root/.cache/huggingface", "-e", "GLOO_SOCKET_IFNAME=$IFACE_NAME", "-e", "NCCL_SOCKET_IFNAME=$IFACE_NAME", "-e", "VLLM_USE_RUST_FRONTEND=1", "vllm/vllm-openai:latest", "zai-org/GLM-5", "--trust-remote-code", "--chat-template-content-format=string", "--tensor-parallel-size", "16", "--nnodes", "2", "--node-rank", "0", "--master-addr", "$HEAD_IP", "--tool-call-parser", "glm47", "--enable-auto-tool-choice", "--reasoning-parser", "glm47" ], "worker_docker_argv": [ "docker", "run", "--gpus", "all", "--privileged", "--ipc=host", "-p", "8000:8000", "-v", "~/.cache/huggingface:/root/.cache/huggingface", "-e", "GLOO_SOCKET_IFNAME=$IFACE_NAME", "-e", "NCCL_SOCKET_IFNAME=$IFACE_NAME", "-e", "VLLM_USE_RUST_FRONTEND=1", "vllm/vllm-openai:latest", "zai-org/GLM-5", "--trust-remote-code", "--chat-template-content-format=string", "--tensor-parallel-size", "16", "--nnodes", "2", "--node-rank", "1", "--master-addr", "$HEAD_IP", "--headless", "--tool-call-parser", "glm47", "--enable-auto-tool-choice", "--reasoning-parser", "glm47" ], "worker_docker_argvs": [ [ "docker", "run", "--gpus", "all", "--privileged", "--ipc=host", "-p", "8000:8000", "-v", "~/.cache/huggingface:/root/.cache/huggingface", "-e", "GLOO_SOCKET_IFNAME=$IFACE_NAME", "-e", "NCCL_SOCKET_IFNAME=$IFACE_NAME", "-e", "VLLM_USE_RUST_FRONTEND=1", "vllm/vllm-openai:latest", "zai-org/GLM-5", "--trust-remote-code", "--chat-template-content-format=string", "--tensor-parallel-size", "16", "--nnodes", "2", "--node-rank", "1", "--master-addr", "$HEAD_IP", "--headless", "--tool-call-parser", "glm47", "--enable-auto-tool-choice", "--reasoning-parser", "glm47" ] ], "strategy_spec": { "name": "multi_node_tp", "deploy_type": "multi_node", "display_name": "Multi-Node Tensor Parallel", "short_label": "Multi-node TP", "orientation": "latency", "description": "Multi-node TP. Splits the model across GPUs spanning multiple nodes. TP is set to total GPU count across all nodes. Head node serves requests, worker nodes run with --headless. Requires InfiniBand interconnect.\n", "hardware_match": { "min_gpus": 2, "multi_node": true }, "parallel_flag": "--tensor-parallel-size", "env": { "GLOO_SOCKET_IFNAME": "$IFACE_NAME", "NCCL_SOCKET_IFNAME": "$IFACE_NAME" } }, "hardware_profile": { "brand": "NVIDIA", "generation": "hopper", "compute_arch": "sm90", "display_name": "H200", "description": "NVIDIA H200 SXM 141 GB HBM3e · 8-GPU HGX node", "gpu_count": 8, "vram_gb": 1128, "multi_node": false }, "alternatives": { "single_node_tp": "/zai-org/GLM-5/strategies/single_node_tp.json", "single_node_tep": "/zai-org/GLM-5/strategies/single_node_tep.json", "multi_node_tp_pp": "/zai-org/GLM-5/strategies/multi_node_tp_pp.json", "multi_node_tep": "/zai-org/GLM-5/strategies/multi_node_tep.json", "multi_node_dep": "/zai-org/GLM-5/strategies/multi_node_dep.json", "pd_cluster": "/zai-org/GLM-5/strategies/pd_cluster.json", "kv_store_centralized_mooncake": "/zai-org/GLM-5/strategies/kv_store_centralized_mooncake.json", "kv_store_distributed_mooncake": "/zai-org/GLM-5/strategies/kv_store_distributed_mooncake.json" }, "by_hardware": { "h200": "/zai-org/GLM-5/hw/h200.json", "h100": "/zai-org/GLM-5/hw/h100.json", "b200": "/zai-org/GLM-5/hw/b200.json", "gb200": "/zai-org/GLM-5/hw/gb200.json", "b300": "/zai-org/GLM-5/hw/b300.json", "gb300": "/zai-org/GLM-5/hw/gb300.json", "mi300x": "/zai-org/GLM-5/hw/mi300x.json", "mi325x": "/zai-org/GLM-5/hw/mi325x.json", "mi350x": "/zai-org/GLM-5/hw/mi350x.json", "mi355x": "/zai-org/GLM-5/hw/mi355x.json" } }, "model": { "model_id": "zai-org/GLM-5", "min_vllm_version": "0.24.0", "default_frontend": "rust", "docker_image": { "amd": "vllm/vllm-openai-rocm:nightly-0cbac6cd1305f710e12193596b27488397bcb205" }, "architecture": "moe", "parameter_count": "744B", "active_parameters": "40B", "context_length": 202752, "base_args": [ "--trust-remote-code", "--chat-template-content-format=string" ], "base_env": {}, "install": { "pip": { "command": "uv venv\nsource .venv/bin/activate\nuv pip install -U vllm --torch-backend auto" }, "docker": { "command": "docker pull vllm/vllm-openai:latest" } } }, "features": { "tool_calling": { "description": "GLM-4.7 tool call parser with automatic tool choice", "args": [ "--tool-call-parser", "glm47", "--enable-auto-tool-choice" ] }, "reasoning": { "description": "GLM-4.7 reasoning parser — thinking mode enabled by default on requests", "args": [ "--reasoning-parser", "glm47" ] }, "spec_decoding": { "description": "Multi-Token Prediction speculative decoding (3 draft tokens)", "args": [ "--speculative-config.method", "mtp", "--speculative-config.num_speculative_tokens", "3" ] } }, "opt_in_features": [ "spec_decoding" ], "variants": { "default": { "precision": "bf16", "vram_minimum_gb": 1786, "description": "Full precision BF16 — requires multi-node deployment" }, "fp8": { "model_id": "zai-org/GLM-5-FP8", "precision": "fp8", "vram_minimum_gb": 893, "description": "Native FP8 checkpoint — 8xH200/H20 (141GB x 8) single-node serving", "json": "/zai-org/GLM-5-FP8.json" }, "nvfp4": { "model_id": "nvidia/GLM-5-NVFP4", "precision": "nvfp4", "vram_minimum_gb": 446, "description": "NVIDIA NVFP4 quantized weights for Blackwell GPUs", "extra_args": [ "--kv-cache-dtype", "fp8" ], "json": "/nvidia/GLM-5-NVFP4.json" } }, "hardware_overrides": { "amd": { "extra_args": [ "--linear-backend", "aiter", "--moe-backend", "aiter" ], "extra_env": { "VLLM_ROCM_USE_AITER": "1", "VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS": "1", "VLLM_ENGINE_READY_TIMEOUT_S": "3600" } } }, "guide": "## Overview\n\nGLM-5 is a significantly scaled-up language model with 744B parameters, trained on\n28.5T tokens using novel asynchronous RL infrastructure. It delivers best-in-class\nopen-source performance on reasoning, coding, and agentic tasks, rivaling frontier\nclosed-source models. GLM-5 is available in both BF16 and native FP8 precisions.\n\nThinking mode is enabled by default; disable it by passing\n`\"chat_template_kwargs\": {\"enable_thinking\": false}` in request extras.\n\n## Prerequisites\n\n- **vLLM version:** >= 0.24.0 for the `glm47` reasoning parser\n- **Hardware (FP8):** 8xH200 or 8xH20 (141GB × 8)\n- **DeepGEMM (FP8, NVIDIA):** install via `install_deepgemm.sh` from the vLLM repo\n- **Hardware (FP8, AMD):** 8x MI325X (256GB × 8) or 8x MI350X (288GB × 8), `vllm/vllm-openai-rocm:nightly-0cbac6cd1305f710e12193596b27488397bcb205`; uses AITER, not DeepGEMM\n\n### Using Docker\n\n```bash\ndocker run --gpus all \\\n -p 8000:8000 \\\n --ipc=host \\\n -v ~/.cache/huggingface:/root/.cache/huggingface \\\n vllm/vllm-openai:latest zai-org/GLM-5-FP8 \\\n --tensor-parallel-size 8 \\\n --tool-call-parser glm47 \\\n --reasoning-parser glm47 \\\n --enable-auto-tool-choice \\\n --chat-template-content-format=string\n```\n\nUse `vllm/vllm-openai:latest-cu129` for CUDA 12.x.\n\n### Install vLLM from Source\n\n```bash\nuv venv\nsource .venv/bin/activate\nuv pip install \"vllm==0.24.0\" --torch-backend=auto\nuv pip install \"transformers>=5.4.0\"\n```\n\nOn NVIDIA, install DeepGEMM using `install_deepgemm.sh` from the vLLM tools directory.\n\n## Launching the Server\n\n### FP8 on 8xH200 with MTP\n\n```bash\nvllm serve zai-org/GLM-5-FP8 \\\n --tensor-parallel-size 8 \\\n --speculative-config.method mtp \\\n --speculative-config.num_speculative_tokens 3 \\\n --tool-call-parser glm47 \\\n --reasoning-parser glm47 \\\n --enable-auto-tool-choice \\\n --chat-template-content-format=string \\\n --served-model-name glm-5-fp8\n```\n\n### FP8 on AMD MI325X and MI350X\n\nSelect **MI325X** or **MI350X** and the **FP8** variant in the command builder above. The\ngenerated command adds `VLLM_ROCM_USE_AITER=1`, `VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS=1`,\n`--linear-backend aiter` and `--moe-backend aiter`, and raises `VLLM_ENGINE_READY_TIMEOUT_S`\nbecause loading the weights can take longer than the Rust frontend's 600 s default. The\nDocker command uses the nightly build\n`vllm/vllm-openai-rocm:nightly-0cbac6cd1305f710e12193596b27488397bcb205`.\n\nOn 8x MI325X that command, with MTP enabled, serves the full 202,752-token context with a\n1,292,544-token KV cache (6.38x maximum concurrency) and an MTP mean acceptance length of\n2.74 with 3 draft tokens on a ~380-token greedy generation.\n\nOn 8x MI350X the same command serves the full context with a 1,587,312-token KV cache (7.83x\nmaximum concurrency) and an MTP mean acceptance length of 2.95 with 3 draft tokens on a\n512-token greedy generation.\n\n## Client Usage\n\n```python\nfrom openai import OpenAI\n\nclient = OpenAI(api_key=\"EMPTY\", base_url=\"http://localhost:8000/v1\")\n\n# Thinking ON (default)\nresp_on = client.chat.completions.create(\n model=\"glm-5-fp8\",\n messages=[{\"role\": \"user\", \"content\": \"Summarize GLM-5 in one sentence.\"}],\n temperature=1,\n max_tokens=4096,\n)\nprint(\"thinking=on, think content:\\n\", resp_on.choices[0].message.reasoning)\n\n# Thinking OFF\nresp_off = client.chat.completions.create(\n model=\"glm-5-fp8\",\n messages=[{\"role\": \"user\", \"content\": \"Summarize GLM-5 in one sentence.\"}],\n temperature=1,\n max_tokens=4096,\n extra_body={\"chat_template_kwargs\": {\"enable_thinking\": False}},\n)\nprint(\"thinking=off:\\n\", resp_off.choices[0].message.reasoning)\n```\n\n## Benchmarking\n\n```bash\nvllm bench serve \\\n --model zai-org/GLM-5-FP8 \\\n --dataset-name random \\\n --random-input 8000 \\\n --random-output 1024 \\\n --request-rate 10 \\\n --num-prompts 32 \\\n --ignore-eos\n```\n\nThe MTP acceptance rate can be relatively low in pure benchmarks; measured throughput\nmay underestimate real-world speed.\n\n## Troubleshooting\n\n- **Accuracy drift:** Prefer the 0.19.0 stable release over nightly for best accuracy.\n- **Tool calling + MTP:** If you need both, use the latest vLLM main branch.\n- **FP8 installation (NVIDIA):** DeepGEMM is required for FP8 performance.\n\n## References\n\n- [Model card](https://huggingface.co/zai-org/GLM-5)\n- [FP8 checkpoint](https://huggingface.co/zai-org/GLM-5-FP8)\n- [DeepGEMM install script](https://github.com/vllm-project/vllm/blob/v0.16.0rc0/tools/install_deepgemm.sh)\n", "source": { "source": "vllm", "repo": "vllm-project/vllm", "ref": "main", "sha": "564ac0de1b228568cafcbfbe8ccaccc82d6023eb", "path": "recipes/models/zai-org/GLM-5.yaml", "edit_url": "https://github.com/vllm-project/vllm/edit/main/recipes/models/zai-org/GLM-5.yaml", "overlays": [] } }