# tencent/Hy4 — vLLM serving recipe

> Tencent HY4 flagship multimodal MoE — 774B total / 49B active, 1M context, hy_v4 tool and reasoning parsers, DFly2 speculative decoding

Configuration: hardware `b300` (8× B300) · strategy `single_node_tp` (Tensor Parallel) · variant `default` (FP8) · features `tool_calling`, `reasoning`, `spec_decoding` · frontend `rust`

- Interactive page: https://recipes.vllm.ai/tencent/Hy4
- This configuration as JSON: https://recipes.vllm.ai/tencent/Hy4.json?hardware=b300

The commands below were rendered for exactly this configuration and take precedence over any example in the guide.

## Other configurations

Change the query string (the same parameters as the page URL) and fetch this URL again. Unknown or incompatible values fall back to the recipe default, so check the Configuration line.

- `hardware`: `h100`, `h200`, `b200`, `gb200`, `b300`, `gb300`
- `variant`: `default`, `bf16`
- `strategy`: `single_node_tp` (multi-node strategies also need `nodes=<n>`)
- `features`: comma-separated list of `tool_calling`, `reasoning`, `spec_decoding` (default: `tool_calling`, `reasoning`, `spec_decoding`). Use `features=none` to turn them all off.

## Install

Requires vLLM >= 0.35.0 (not released yet — nightly wheel).

v0.34.0 does not register HY4; nightly wheels built from 2026-11-20 on do

```bash
uv venv
source .venv/bin/activate
uv pip install -U vllm --pre \
  --extra-index-url https://wheels.vllm.ai/nightly/cu130 \
  --extra-index-url https://download.pytorch.org/whl/cu130 \
  --index-strategy unsafe-best-match
```

Docker image: `vllm/vllm-openai:hy4`

## Serve

```bash
export VLLM_USE_RUST_FRONTEND=1
vllm serve tencent/Hy4-FP8 \
  --dtype bfloat16 \
  --max-model-len 131072 \
  --max-num-seqs 32 \
  --max-num-batched-tokens 8192 \
  --attention-backend FLASHMLA_SPARSE \
  --limit-mm-per-prompt '{"image":1,"video":0}' \
  --tensor-parallel-size 8 \
  --enable-auto-tool-choice \
  --tool-call-parser hy_v4 \
  --reasoning-parser hy_v4 \
  --speculative-config '{"num_speculative_tokens":5,"method":"dfly"}'
```

Docker equivalent:

```bash
docker run --gpus all \
  --privileged --ipc=host -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -e VLLM_USE_RUST_FRONTEND=1 \
  vllm/vllm-openai:hy4 tencent/Hy4-FP8 \
  --dtype bfloat16 \
  --max-model-len 131072 \
  --max-num-seqs 32 \
  --max-num-batched-tokens 8192 \
  --attention-backend FLASHMLA_SPARSE \
  --limit-mm-per-prompt '{"image":1,"video":0}' \
  --tensor-parallel-size 8 \
  --enable-auto-tool-choice \
  --tool-call-parser hy_v4 \
  --reasoning-parser hy_v4 \
  --speculative-config '{"num_speculative_tokens":5,"method":"dfly"}'
```

## Guide

> Background reference. Commands in it are examples and may target another configuration — use the Serve section above.

### Overview

HY4 is the Tencent Hy Team's flagship Mixture-of-Experts model for text and
image understanding, with up to 1M tokens of context. Two checkpoints share
one architecture (`HYV4VLForConditionalGeneration`):

- [HY4 FP8](https://huggingface.co/tencent/Hy4-FP8) — the default variant,
  and the checkpoint the model card serves
- [HY4 BF16](https://huggingface.co/tencent/Hy4) — the BF16 variant

The language backbone has 770B parameters, 49B of them active per token,
across 78 layers. The first layer has a dense FFN; the other 77 use 256
routed experts (top-8) and one shared expert. Attention is gated sparse MLA
with cross-layer index reuse, and iHC connects the residual streams. A
40-layer vision encoder (1.23B) and the five-layer DFly2 draft (3.3B) bring
the stored total to about 774B.

HY4 is not [HY4 Flash](https://recipes.vllm.ai/tencent/Hy4-flash) or the
text-only [Hy4-preview](https://recipes.vllm.ai/tencent/Hy4-preview). Those
are separate checkpoints with their own recipes.

### Prerequisites

- **Hardware**: one 8-GPU NVIDIA node. FP8 needs about 944 GB of aggregate GPU
  memory and BF16 about 1,859 GB (8x B300). AMD Instinct is not supported:
  vLLM's ROCm port has neither the vision encoder nor the DFly2 drafter.
- **vLLM**: the `vllm/vllm-openai:hy4` image, or a nightly wheel. v0.34.0
  does not register the architecture; 0.35.0 will be the first release
  with it.
- **Host memory**: enough RAM to load the checkpoint — 786 GB (FP8) or
  1,549 GB (BF16) on disk, 245 safetensors shards each.

### Serving configuration

The defaults follow the model card's launch line: FP8 checkpoint, TP8, Rust
frontend, `FLASHMLA_SPARSE` attention, BF16 compute, tool calling, reasoning
parsing, and DFly2 with five speculative tokens.

- **Parsers**: HY4 pairs its tool-call and reasoning parsers (`hy_v4`). Tool
  calling sets both and adds automatic tool choice.
- **DFly2**: the draft ships inside both checkpoints and shares the target's
  KV cache and sparse indices, so it needs no extra checkpoint or KV memory.
  It requires data, pipeline, prefill-context and decode-context parallel
  sizes of 1, which is why the recipe offers single-node TP only. The
  checkpoints have no MTP layers (`num_nextn_predict_layers=0`), so `mtp`
  is not an alternative method.
- **FP8**: blockwise W8A8 compressed-tensors quantization; `lm_head`, the
  vision tower, the first layer's attention and the indexer projections stay
  unquantized. vLLM reads the scheme from the checkpoint, so no quantization
  flag is needed. It is not the MXFP8 format of Hy4-preview-FP8.
- **Multimodal**: one image per prompt, video off, as on the model card.
  Raise `--limit-mm-per-prompt` to allow more.

### Context and capacity

The checkpoints advertise 1,048,576 positions. The recipe starts from the
evaluated window: 131,072 tokens, 32 concurrent sequences and 8,192 batched
tokens. The model card's launch line sets none of the three, which serves the
full 1M window; raise them to match your memory budget. A full 1M-token
request has not been validated for this recipe.

### Client usage

The model name is the selected checkpoint ID: `tencent/Hy4-FP8` by default,
`tencent/Hy4` for the BF16 variant. Tencent recommends `temperature=0.9` and
`top_p=1.0`.

```python
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
response = client.chat.completions.create(
    model="tencent/Hy4-FP8",
    messages=[{
        "role": "user",
        "content": [
            {"type": "image_url", "image_url": {"url": "https://example.com/chart.png"}},
            {"type": "text", "text": "Summarize this chart."},
        ],
    }],
    temperature=0.9,
    top_p=1.0,
)
print(response.choices[0].message.reasoning)  # chain of thought
print(response.choices[0].message.content)    # final answer
```

`reasoning_effort` accepts `high` (the default, for math, coding and other
complex tasks) and `no_think` for a direct answer:

```python
response = client.chat.completions.create(
    model="tencent/Hy4-FP8",
    messages=[{"role": "user", "content": "What is 17 multiplied by 23?"}],
    temperature=0.9,
    top_p=1.0,
    extra_body={"chat_template_kwargs": {"reasoning_effort": "no_think"}},
)
print(response.choices[0].message.content)
```

### References

- [HY4 research page](https://hy.tencent.ai/research/hy4)
- [BF16 checkpoint](https://huggingface.co/tencent/Hy4)
- [FP8 checkpoint](https://huggingface.co/tencent/Hy4-FP8)
- [GitHub: Tencent-Hunyuan/Hy4](https://github.com/Tencent-Hunyuan/Hy4)
- [AngelSpec, the DFly2 training toolkit](https://github.com/Tencent/AngelSpec)
