{ "hf_id": "zai-org/GLM-4.5", "meta": { "title": "GLM-4.5", "slug": "glm-4.5", "provider": "GLM (Z-AI)", "description": "GLM-4.5 MoE language model (~358B total parameters, BF16) with built-in MTP layers for speculative decoding and native tool calling", "date_added": "2025-08-12", "date_updated": "2026-09-14", "tasks": [ "text" ], "performance_headline": "GLM-4.X series MoE model with native FP8 and BF16 support and MTP speculative decoding", "related_recipes": [], "hardware": { "h200": "verified", "mi300x": "verified", "mi325x": "verified", "mi355x": "verified" } }, "recommended_command": { "hardware": "h200", "strategy": "single_node_tp", "variant": "default", "node_count": 1, "deploy_type": "single_node", "env": { "VLLM_USE_RUST_FRONTEND": "1" }, "docker_image": "vllm/vllm-openai:latest", "command": "vllm serve zai-org/GLM-4.5 \\\n --trust-remote-code \\\n --tensor-parallel-size 8 \\\n --tool-call-parser glm45 \\\n --enable-auto-tool-choice \\\n --reasoning-parser glm45", "argv": [ "vllm", "serve", "zai-org/GLM-4.5", "--trust-remote-code", "--tensor-parallel-size", "8", "--tool-call-parser", "glm45", "--enable-auto-tool-choice", "--reasoning-parser", "glm45" ], "docker_command": "docker run --gpus all \\\n --privileged --ipc=host -p 8000:8000 \\\n -v ~/.cache/huggingface:/root/.cache/huggingface \\\n -e VLLM_USE_RUST_FRONTEND=1 \\\n vllm/vllm-openai:latest zai-org/GLM-4.5 \\\n --trust-remote-code \\\n --tensor-parallel-size 8 \\\n --tool-call-parser glm45 \\\n --enable-auto-tool-choice \\\n --reasoning-parser glm45", "docker_argv": [ "docker", "run", "--gpus", "all", "--privileged", "--ipc=host", "-p", "8000:8000", "-v", "~/.cache/huggingface:/root/.cache/huggingface", "-e", "VLLM_USE_RUST_FRONTEND=1", "vllm/vllm-openai:latest", "zai-org/GLM-4.5", "--trust-remote-code", "--tensor-parallel-size", "8", "--tool-call-parser", "glm45", "--enable-auto-tool-choice", "--reasoning-parser", "glm45" ], "strategy_spec": { "name": "single_node_tp", "deploy_type": "single_node", "display_name": "Tensor Parallel", "short_label": "TP", "orientation": "latency", "description": "Single-node tensor parallel. Splits the model across all local GPUs. TP size is set to the GPU count at deploy time. The simplest multi-GPU strategy — works for all model architectures.\n", "hardware_match": { "min_gpus": 1, "max_gpus": 8, "multi_node": false }, "vllm_args": [], "parallel_flag": "--tensor-parallel-size" }, "hardware_profile": { "brand": "NVIDIA", "generation": "hopper", "compute_arch": "sm90", "display_name": "H200", "description": "NVIDIA H200 SXM 141 GB HBM3e · 8-GPU HGX node", "gpu_count": 8, "vram_gb": 1128, "multi_node": false }, "alternatives": { "single_node_tep": "/zai-org/GLM-4.5/strategies/single_node_tep.json", "single_node_dep": "/zai-org/GLM-4.5/strategies/single_node_dep.json", "multi_node_tp": "/zai-org/GLM-4.5/strategies/multi_node_tp.json", "multi_node_tp_pp": "/zai-org/GLM-4.5/strategies/multi_node_tp_pp.json", "multi_node_dep": "/zai-org/GLM-4.5/strategies/multi_node_dep.json", "multi_node_tep": "/zai-org/GLM-4.5/strategies/multi_node_tep.json", "pd_cluster": "/zai-org/GLM-4.5/strategies/pd_cluster.json", "kv_store_centralized_mooncake": "/zai-org/GLM-4.5/strategies/kv_store_centralized_mooncake.json", "kv_store_distributed_mooncake": "/zai-org/GLM-4.5/strategies/kv_store_distributed_mooncake.json" }, "by_hardware": { "h200": "/zai-org/GLM-4.5/hw/h200.json", "h100": "/zai-org/GLM-4.5/hw/h100.json", "b200": "/zai-org/GLM-4.5/hw/b200.json", "gb200": "/zai-org/GLM-4.5/hw/gb200.json", "b300": "/zai-org/GLM-4.5/hw/b300.json", "gb300": "/zai-org/GLM-4.5/hw/gb300.json", "mi300x": "/zai-org/GLM-4.5/hw/mi300x.json", "mi325x": "/zai-org/GLM-4.5/hw/mi325x.json", "mi350x": "/zai-org/GLM-4.5/hw/mi350x.json", "mi355x": "/zai-org/GLM-4.5/hw/mi355x.json" } }, "model": { "model_id": "zai-org/GLM-4.5", "min_vllm_version": "0.11.0", "default_frontend": "rust", "architecture": "moe", "parameter_count": "358B", "active_parameters": "32B", "context_length": 131072, "base_args": [ "--trust-remote-code" ], "base_env": {}, "install": { "pip": { "command": "uv venv\nsource .venv/bin/activate\nuv pip install -U vllm --torch-backend auto" }, "docker": { "command": "docker pull vllm/vllm-openai:latest" } } }, "features": { "tool_calling": { "description": "GLM-4.5 tool call parser with automatic tool choice", "args": [ "--tool-call-parser", "glm45", "--enable-auto-tool-choice" ] }, "reasoning": { "description": "GLM-4.5 reasoning parser for chain-of-thought extraction", "args": [ "--reasoning-parser", "glm45" ] }, "spec_decoding": { "description": "Multi-Token Prediction speculative decoding using the model's built-in MTP layers", "args": [ "--speculative-config.method", "mtp", "--speculative-config.num_speculative_tokens", "1" ] } }, "opt_in_features": [ "spec_decoding" ], "variants": { "default": { "precision": "bf16", "vram_minimum_gb": 859, "description": "Full precision BF16 on 8xH200 or equivalent" }, "fp8": { "model_id": "zai-org/GLM-4.5-FP8", "precision": "fp8", "vram_minimum_gb": 430, "description": "Native FP8 checkpoint with minimal accuracy loss — recommended for cost-efficient serving", "json": "/zai-org/GLM-4.5-FP8.json" } }, "kv_offload_support": { "offloading_cpu": "verified", "offloading_fs": "verified" }, "hardware_overrides": {}, "guide": "## Overview\n\nGLM-4.5 is a Mixture-of-Experts language model from Z-AI with ~358B total parameters.\nThe checkpoint ships in both BF16 and native FP8 formats. FP8 models have minimal\naccuracy loss, so unless you need strict reproducibility for benchmarking, FP8 is the\nrecommended precision for lower-cost serving. All GLM-4.X models include built-in\nMulti-Token Prediction (MTP) layers that enable speculative decoding for higher\ngeneration throughput.\n\n## Prerequisites\n\n- **vLLM version:** >= 0.11.0 (latest stable recommended)\n- **Hardware:** 8x H200 (BF16) or 4x-8x H200 (FP8), AMD MI300X / MI325X / MI355X for ROCm\n- **Python:** 3.10 - 3.13 (3.12 required for ROCm wheels)\n\n### Install vLLM (NVIDIA)\n\n```bash\nuv venv\nsource .venv/bin/activate\nuv pip install -U vllm --torch-backend auto\n```\n\n### Install vLLM (AMD ROCm)\n\n```bash\nuv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm\n```\n\nThe vLLM wheel for ROCm requires Python 3.12, ROCm 7.0, and glibc >= 2.35.\n\n## Launching the Server\n\n### Tensor Parallel (FP8 on 8 GPUs)\n\n```bash\nvllm serve zai-org/GLM-4.5-FP8 \\\n --tensor-parallel-size 8 \\\n --tool-call-parser glm45 \\\n --reasoning-parser glm45 \\\n --enable-auto-tool-choice\n```\n\n### Enabling MTP Speculative Decoding\n\n```bash\nvllm serve zai-org/GLM-4.5-FP8 \\\n --tensor-parallel-size 4 \\\n --speculative-config.method mtp \\\n --speculative-config.num_speculative_tokens 1 \\\n --tool-call-parser glm45 \\\n --reasoning-parser glm45 \\\n --enable-auto-tool-choice\n```\n\nUse `--speculative-config.num_speculative_tokens 1` for optimal throughput. Higher\nvalues increase mean acceptance length but drop acceptance rate significantly.\n\n### Tuning Tips\n\n- `--max-model-len=65536` works well for most scenarios; max is 128K.\n- `--max-num-batched-tokens=32768` is a good default for prompt-heavy workloads.\n Reduce to 16K/8K to cut activation memory and latency.\n- Set `--gpu-memory-utilization=0.95` to maximize KV cache headroom.\n\n## Client Usage\n\n```python\nfrom openai import OpenAI\n\nclient = OpenAI(api_key=\"EMPTY\", base_url=\"http://localhost:8000/v1\")\nresp = client.chat.completions.create(\n model=\"zai-org/GLM-4.5-FP8\",\n messages=[{\"role\": \"user\", \"content\": \"Explain MTP speculative decoding.\"}],\n max_tokens=512,\n)\nprint(resp.choices[0].message.content)\n```\n\n## Benchmarking\n\nDisable prefix caching with `--no-enable-prefix-caching` on the server command,\nthen:\n\n```bash\nvllm bench serve \\\n --model zai-org/GLM-4.5-FP8 \\\n --dataset-name random \\\n --random-input-len 8000 \\\n --random-output-len 1000 \\\n --request-rate 10000 \\\n --num-prompts 16 \\\n --ignore-eos\n```\n\n## Troubleshooting\n\n- **Tool calling not firing:** Ensure `--tool-call-parser glm45 --enable-auto-tool-choice`\n are both present.\n- **MTP memory overhead:** MTP adds memory for draft computations. Monitor GPU memory\n and reduce `--max-model-len` or `--max-num-batched-tokens` if you OOM.\n- **Low MTP acceptance:** If acceptance rate is below ~90%, drop speculative tokens to 1.\n\n## References\n\n- [Model card](https://huggingface.co/zai-org/GLM-4.5)\n- [FP8 checkpoint](https://huggingface.co/zai-org/GLM-4.5-FP8)\n- [vLLM docs](https://docs.vllm.ai/)\n", "source": { "source": "vllm", "repo": "vllm-project/vllm", "ref": "main", "sha": "564ac0de1b228568cafcbfbe8ccaccc82d6023eb", "path": "recipes/models/zai-org/GLM-4.5.yaml", "edit_url": "https://github.com/vllm-project/vllm/edit/main/recipes/models/zai-org/GLM-4.5.yaml", "overlays": [] } }