tencent/Hy4
Tencent HY4 flagship multimodal MoE — 774B total / 49B active, 1M context, hy_v4 tool and reasoning parsers, DFly2 speculative decoding
export VLLM_USE_RUST_FRONTEND=1
vllm serve tencent/Hy4-FP8 \
--dtype bfloat16 \
--max-model-len 131072 \
--max-num-seqs 32 \
--max-num-batched-tokens 8192 \
--attention-backend FLASHMLA_SPARSE \
--limit-mm-per-prompt '{"image":1,"video":0}' \
--tensor-parallel-size 8 \
--enable-auto-tool-choice \
--tool-call-parser hy_v4 \
--reasoning-parser hy_v4 \
--speculative-config '{"num_speculative_tokens":5,"method":"dfly"}'The interactive builder loads with JavaScript. Append .md to this page's URL, query string included, for any configuration's command as Markdown — e.g. /tencent/Hy4.md.
Overview
HY4 is the Tencent Hy Team's flagship Mixture-of-Experts model for text and
image understanding, with up to 1M tokens of context. Two checkpoints share
one architecture (HYV4VLForConditionalGeneration):
The language backbone has 770B parameters, 49B of them active per token, across 78 layers. The first layer has a dense FFN; the other 77 use 256 routed experts (top-8) and one shared expert. Attention is gated sparse MLA with cross-layer index reuse, and iHC connects the residual streams. A 40-layer vision encoder (1.23B) and the five-layer DFly2 draft (3.3B) bring the stored total to about 774B.
HY4 is not HY4 Flash or the text-only Hy4-preview. Those are separate checkpoints with their own recipes.
Prerequisites
- Hardware: one 8-GPU NVIDIA node. FP8 needs about 944 GB of aggregate GPU memory and BF16 about 1,859 GB (8x B300). AMD Instinct is not supported: vLLM's ROCm port has neither the vision encoder nor the DFly2 drafter.
- vLLM: the
vllm/vllm-openai:hy4image, or a nightly wheel. v0.34.0 does not register the architecture; 0.35.0 will be the first release with it. - Host memory: enough RAM to load the checkpoint — 786 GB (FP8) or 1,549 GB (BF16) on disk, 245 safetensors shards each.
Serving configuration
The defaults follow the model card's launch line: FP8 checkpoint, TP8, Rust
frontend, FLASHMLA_SPARSE attention, BF16 compute, tool calling, reasoning
parsing, and DFly2 with five speculative tokens.
- Parsers: HY4 pairs its tool-call and reasoning parsers (
hy_v4). Tool calling sets both and adds automatic tool choice. - DFly2: the draft ships inside both checkpoints and shares the target's
KV cache and sparse indices, so it needs no extra checkpoint or KV memory.
It requires data, pipeline, prefill-context and decode-context parallel
sizes of 1, which is why the recipe offers single-node TP only. The
checkpoints have no MTP layers (
num_nextn_predict_layers=0), somtpis not an alternative method. - FP8: blockwise W8A8 compressed-tensors quantization;
lm_head, the vision tower, the first layer's attention and the indexer projections stay unquantized. vLLM reads the scheme from the checkpoint, so no quantization flag is needed. It is not the MXFP8 format of Hy4-preview-FP8. - Multimodal: one image per prompt, video off, as on the model card.
Raise
--limit-mm-per-promptto allow more.
Context and capacity
The checkpoints advertise 1,048,576 positions. The recipe starts from the evaluated window: 131,072 tokens, 32 concurrent sequences and 8,192 batched tokens. The model card's launch line sets none of the three, which serves the full 1M window; raise them to match your memory budget. A full 1M-token request has not been validated for this recipe.
Client usage
The model name is the selected checkpoint ID: tencent/Hy4-FP8 by default,
tencent/Hy4 for the BF16 variant. Tencent recommends temperature=0.9 and
top_p=1.0.
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
response = client.chat.completions.create(
model="tencent/Hy4-FP8",
messages=[{
"role": "user",
"content": [
{"type": "image_url", "image_url": {"url": "https://example.com/chart.png"}},
{"type": "text", "text": "Summarize this chart."},
],
}],
temperature=0.9,
top_p=1.0,
)
print(response.choices[0].message.reasoning) # chain of thought
print(response.choices[0].message.content) # final answer
reasoning_effort accepts high (the default, for math, coding and other
complex tasks) and no_think for a direct answer:
response = client.chat.completions.create(
model="tencent/Hy4-FP8",
messages=[{"role": "user", "content": "What is 17 multiplied by 23?"}],
temperature=0.9,
top_p=1.0,
extra_body={"chat_template_kwargs": {"reasoning_effort": "no_think"}},
)
print(response.choices[0].message.content)