zai-org/GLM-5.3-Flash

Overview

GLM-5.3-Flash is a multimodal mixture-of-experts model with approximately 321B total parameters and 18B active parameters per token. Its 45-layer language model combines KDA linear-attention layers with NoPE sparse MLA layers, routes each token through 8 of 288 experts, and supports image and video inputs. The checkpoint declares a maximum context length of 1,048,576 tokens and includes one MTP draft layer.

The current vLLM implementation supports NVIDIA Hopper and newer GPUs only. It can scale with tensor, pipeline, expert, or data+expert parallelism.

Prerequisites

• Weights: about 306 GiB for the default native FP8 checkpoint before runtime and KV-cache overhead; the BF16 variant requires roughly twice the weight memory
• vLLM: use docker before the integration is included in the public repo
• FlashInfer: 0.6.17 or newer is required for NoPE sparse MLA

The default model ID, zai-org/GLM-5.3-Flash, is the FP8 checkpoint. Select the BF16 variant to serve zai-org/GLM-5.3-Flash-BF16 instead.

Launching the server

FP8 with TP4 and MTP on one GB200 tray

vllm serve zai-org/GLM-5.3-Flash \
  --tensor-parallel-size 4 \
  --kv-cache-dtype fp8 \
  --speculative-config '{"method":"mtp","num_speculative_tokens":5}' \
  --tool-call-parser glm47 \
  --reasoning-parser glm45 \
  --enable-auto-tool-choice \
  --served-model-name zai-org/GLM-5.3-Flash

Prefill/Decode disaggregation (single node, TP4 + TP4)

Split one 8-GPU node into a prefill pool (GPUs 0-3) and a decode pool (GPUs 4-7), bridged by NIXL KV transfer. The KDA conv-state and KV-cache layouts must be pinned identically on both pools, and MTP drafts run on both sides. On Blackwell you can add --kv-cache-dtype fp8 to both pools; Hopper does not support FP8 KV cache for this model and must run BF16 KV.

Note: num_speculative_tokens must be the same in prefill and decode instance

# Prefill pool
CUDA_VISIBLE_DEVICES=0,1,2,3 \
VLLM_SSM_CONV_STATE_LAYOUT=DS \
VLLM_KV_CACHE_LAYOUT=HND \
UCX_NET_DEVICES=all \
VLLM_NIXL_SIDE_CHANNEL_PORT=5557 \
vllm serve zai-org/GLM-5.3-Flash \
  --port 8001 \
  --tensor-parallel-size 4 \
  --kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_producer"}' \
  --speculative-config '{"method":"mtp","num_speculative_tokens":5}' \
  --compilation-config '{"cudagraph_mm_encoder": true}' \
  --enforce-eager \
  --no-disable-hybrid-kv-cache-manager \
  --served-model-name zai-org/GLM-5.3-Flash

# Decode pool
CUDA_VISIBLE_DEVICES=4,5,6,7 \
VLLM_SSM_CONV_STATE_LAYOUT=DS \
VLLM_KV_CACHE_LAYOUT=HND \
UCX_NET_DEVICES=all \
VLLM_NIXL_SIDE_CHANNEL_PORT=5558 \
vllm serve zai-org/GLM-5.3-Flash \
  --port 8002 \
  --tensor-parallel-size 4 \
  --kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_consumer"}' \
  --speculative-config '{"method":"mtp","num_speculative_tokens":5}' \
  --max-num-seqs 512 \
  --no-disable-hybrid-kv-cache-manager \
  --served-model-name zai-org/GLM-5.3-Flash

# Router
vllm-router --policy round_robin --vllm-pd-disaggregation \
  --prefill http://127.0.0.1:8001 \
  --decode http://127.0.0.1:8002 \
  --host 0.0.0.0 --port 8000 \
  --intra-node-data-parallel-size 1

Client usage

from openai import OpenAI

client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1")
response = client.chat.completions.create(
    model="zai-org/GLM-5.3-Flash",
    messages=[{"role": "user", "content": "Summarize sparse attention in one sentence."}],
    temperature=1.0,
    max_tokens=256,
)
print(response.choices[0].message.content)

Benchmarking

vllm bench serve \
  --backend vllm \
  --model zai-org/GLM-5.3-Flash \
  --served-model-name zai-org/GLM-5.3-Flash \
  --dataset-name random \
  --random-input-len 8192 \
  --random-output-len 1024 \
  --max-concurrency 16 \
  --num-prompts 64

Troubleshooting

• Sparse-MLA initialization error: verify that the image contains FlashInfer 0.6.18 or newer.

References

• Model card
• vLLM