vLLM/Recipes
Hunyuan (Tencent)

tencent/Hy4-preview

Tencent Hunyuan Hy4-preview — scaled-up MoE language model (770B total / 49B active) with a 10B MTP layer for speculative decoding, 1M context, and hy_v4 tool/reasoning parsers

Hunyuan Hy4-preview MoE — 770B/49B on 16xB200, 8xB300 with MTP

View on HuggingFace
moe770B / 49B1,048,576 ctxvLLM 0.29.0+text
Guide

Hy4-preview Usage Guide

Hy4-preview is Tencent Hunyuan's new-generation open-source Mixture-of-Experts language model. Its backbone has 770B total parameters with 49B activated per token across 78 layers. The first layer uses a dense FFN; the other 77 layers use 256 routed experts (top-8) and 1 shared expert. A native MTP layer adds 10B total parameters (0.7B activated) for speculative decoding.

On the architecture side, inspired by DeepSeek and GLM, the attention module employs Gated DeepSeek Sparse Attention (Gated DSA) with IndexCache for cross-layer sparse index reuse. The residual pathway uses iHC (identity Hyper-Connections) to expand inter-layer information flow.

vLLM Setup

Choose one of the following setup methods.

Using Docker

docker run --gpus all \
  -p 8000:8000 \
  --ipc=host \
  -e VLLM_ENABLE_HPC_OPS=1 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai:hy4-preview tencent/Hy4-preview-FP8 \
    --tensor-parallel-size 8 \
    --speculative-config '{"num_speculative_tokens":3,"method":"mtp"}' \
    --attention-backend FLASHMLA_SPARSE \
    --tool-call-parser hy_v4 \
    --reasoning-parser hy_v4 \
    --enable-auto-tool-choice \
    --port 8000 \
    --served-model-name hy4-preview

Build vLLM from source:

uv venv --python 3.12 --seed --managed-python
source .venv/bin/activate
git clone https://github.com/vllm-project/vllm.git
cd vllm
uv pip install --editable . --torch-backend=auto

vLLM Deployment

Start the FP8 model with MTP enabled:

export VLLM_ENABLE_HPC_OPS=1

vllm serve tencent/Hy4-preview-FP8 \
  --tensor-parallel-size 8 \
  --speculative-config '{"num_speculative_tokens":3,"method":"mtp"}' \
  --attention-backend FLASHMLA_SPARSE \
  --tool-call-parser hy_v4 \
  --reasoning-parser hy_v4 \
  --enable-auto-tool-choice \
  --port 8000 \
  --served-model-name hy4-preview

OpenAI Client Example

Tencent recommends temperature=0.9 and top_p=1.0. Reasoning defaults to high, which is suitable for math, coding, and other complex tasks.

from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:8000/v1", api_key="EMPTY")

response = client.chat.completions.create(
    model="hy4-preview",
    messages=[
        {"role": "user", "content": "Hello! Can you briefly introduce yourself?"},
    ],
    temperature=0.9,
    top_p=1.0,
)
output_msg = response.choices[0].message
print(output_msg.reasoning_content)  # chain-of-thought
print(output_msg.content)             # final answer

For a direct response without deep reasoning, pass reasoning_effort="no_think":

response = client.chat.completions.create(
    model="hy4-preview",
    messages=[
        {"role": "user", "content": "Hello! Can you briefly introduce yourself?"},
    ],
    temperature=0.9,
    top_p=1.0,
    extra_body={"chat_template_kwargs": {"reasoning_effort": "no_think"}},
)
print(response.choices[0].message.content)

cURL Usage

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "hy4-preview",
    "messages": [
      {"role": "system", "content": "You are a helpful assistant."},
      {"role": "user", "content": "Hello! Can you briefly introduce yourself?"}
    ],
    "temperature": 0.9,
    "top_p": 1.0
  }'

References