vLLM/Recipes
Search
Hunyuan (Tencent)

tencent/Hy4

Tencent HY4 flagship multimodal MoE — 774B total / 49B active, 1M context, hy_v4 tool and reasoning parsers, DFly2 speculative decoding

moe774B / 49B1,048,576 ctxvLLM 0.35.0+nightlyMultimodal
Default configuration: 8× B300 · Tensor Parallel · FP8
export VLLM_USE_RUST_FRONTEND=1
vllm serve tencent/Hy4-FP8 \
  --dtype bfloat16 \
  --max-model-len 131072 \
  --max-num-seqs 32 \
  --max-num-batched-tokens 8192 \
  --attention-backend FLASHMLA_SPARSE \
  --limit-mm-per-prompt '{"image":1,"video":0}' \
  --tensor-parallel-size 8 \
  --enable-auto-tool-choice \
  --tool-call-parser hy_v4 \
  --reasoning-parser hy_v4 \
  --speculative-config '{"num_speculative_tokens":5,"method":"dfly"}'

The interactive builder loads with JavaScript. Append .md to this page's URL, query string included, for any configuration's command as Markdown — e.g. /tencent/Hy4.md.

Guide

Overview

HY4 is the Tencent Hy Team's flagship Mixture-of-Experts model for text and image understanding, with up to 1M tokens of context. Two checkpoints share one architecture (HYV4VLForConditionalGeneration):

  • HY4 FP8 — the default variant, and the checkpoint the model card serves
  • HY4 BF16 — the BF16 variant

The language backbone has 770B parameters, 49B of them active per token, across 78 layers. The first layer has a dense FFN; the other 77 use 256 routed experts (top-8) and one shared expert. Attention is gated sparse MLA with cross-layer index reuse, and iHC connects the residual streams. A 40-layer vision encoder (1.23B) and the five-layer DFly2 draft (3.3B) bring the stored total to about 774B.

HY4 is not HY4 Flash or the text-only Hy4-preview. Those are separate checkpoints with their own recipes.

Prerequisites

  • Hardware: one 8-GPU NVIDIA node. FP8 needs about 944 GB of aggregate GPU memory and BF16 about 1,859 GB (8x B300). AMD Instinct is not supported: vLLM's ROCm port has neither the vision encoder nor the DFly2 drafter.
  • vLLM: the vllm/vllm-openai:hy4 image, or a nightly wheel. v0.34.0 does not register the architecture; 0.35.0 will be the first release with it.
  • Host memory: enough RAM to load the checkpoint — 786 GB (FP8) or 1,549 GB (BF16) on disk, 245 safetensors shards each.

Serving configuration

The defaults follow the model card's launch line: FP8 checkpoint, TP8, Rust frontend, FLASHMLA_SPARSE attention, BF16 compute, tool calling, reasoning parsing, and DFly2 with five speculative tokens.

  • Parsers: HY4 pairs its tool-call and reasoning parsers (hy_v4). Tool calling sets both and adds automatic tool choice.
  • DFly2: the draft ships inside both checkpoints and shares the target's KV cache and sparse indices, so it needs no extra checkpoint or KV memory. It requires data, pipeline, prefill-context and decode-context parallel sizes of 1, which is why the recipe offers single-node TP only. The checkpoints have no MTP layers (num_nextn_predict_layers=0), so mtp is not an alternative method.
  • FP8: blockwise W8A8 compressed-tensors quantization; lm_head, the vision tower, the first layer's attention and the indexer projections stay unquantized. vLLM reads the scheme from the checkpoint, so no quantization flag is needed. It is not the MXFP8 format of Hy4-preview-FP8.
  • Multimodal: one image per prompt, video off, as on the model card. Raise --limit-mm-per-prompt to allow more.

Context and capacity

The checkpoints advertise 1,048,576 positions. The recipe starts from the evaluated window: 131,072 tokens, 32 concurrent sequences and 8,192 batched tokens. The model card's launch line sets none of the three, which serves the full 1M window; raise them to match your memory budget. A full 1M-token request has not been validated for this recipe.

Client usage

The model name is the selected checkpoint ID: tencent/Hy4-FP8 by default, tencent/Hy4 for the BF16 variant. Tencent recommends temperature=0.9 and top_p=1.0.

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
response = client.chat.completions.create(
    model="tencent/Hy4-FP8",
    messages=[{
        "role": "user",
        "content": [
            {"type": "image_url", "image_url": {"url": "https://example.com/chart.png"}},
            {"type": "text", "text": "Summarize this chart."},
        ],
    }],
    temperature=0.9,
    top_p=1.0,
)
print(response.choices[0].message.reasoning)  # chain of thought
print(response.choices[0].message.content)    # final answer

reasoning_effort accepts high (the default, for math, coding and other complex tasks) and no_think for a direct answer:

response = client.chat.completions.create(
    model="tencent/Hy4-FP8",
    messages=[{"role": "user", "content": "What is 17 multiplied by 23?"}],
    temperature=0.9,
    top_p=1.0,
    extra_body={"chat_template_kwargs": {"reasoning_effort": "no_think"}},
)
print(response.choices[0].message.content)

References