# tencent/Hy4-flash — vLLM serving recipe

> Tencent HY4 Flash multimodal MoE for everyday productivity and agent workflows — 220B total / 12B active, 1M context, DFly speculative decoding

Configuration: hardware `h200` (8× H200) · strategy `single_node_tp` (Tensor Parallel) · variant `default` (BF16) · features `tool_calling`, `reasoning`, `spec_decoding` · frontend `rust`

- Interactive page: https://recipes.vllm.ai/tencent/Hy4-flash
- This configuration as JSON: https://recipes.vllm.ai/tencent/Hy4-flash.json?hardware=h200

The commands below were rendered for exactly this configuration and take precedence over any example in the guide.

## Other configurations

Change the query string (the same parameters as the page URL) and fetch this URL again. Unknown or incompatible values fall back to the recipe default, so check the Configuration line.

- `hardware`: `h100`, `h200`, `b200`, `gb200`, `b300`, `gb300`
- `variant`: `default`, `fp8`
- `strategy`: `single_node_tp`, `single_node_tep` (multi-node strategies also need `nodes=<n>`)
- `features`: comma-separated list of `tool_calling`, `reasoning`, `spec_decoding` (default: `tool_calling`, `reasoning`, `spec_decoding`). Use `features=none` to turn them all off.

## Install

Docker image: `vllm/vllm-openai:hy4-flash`

## Serve

```bash
export VLLM_USE_RUST_FRONTEND=1
vllm serve tencent/Hy4-flash \
  --dtype bfloat16 \
  --max-model-len 8192 \
  --max-num-seqs 4 \
  --max-num-batched-tokens 8192 \
  --limit-mm-per-prompt '{"image":1,"video":0}' \
  --tensor-parallel-size 8 \
  --enable-auto-tool-choice \
  --reasoning-parser hy_v4 \
  --tool-call-parser hy_v4 \
  --speculative-config '{"num_speculative_tokens":5,"method":"dfly"}'
```

Docker equivalent:

```bash
docker run --gpus all \
  --privileged --ipc=host -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -e VLLM_USE_RUST_FRONTEND=1 \
  vllm/vllm-openai:hy4-flash tencent/Hy4-flash \
  --dtype bfloat16 \
  --max-model-len 8192 \
  --max-num-seqs 4 \
  --max-num-batched-tokens 8192 \
  --limit-mm-per-prompt '{"image":1,"video":0}' \
  --tensor-parallel-size 8 \
  --enable-auto-tool-choice \
  --reasoning-parser hy_v4 \
  --tool-call-parser hy_v4 \
  --speculative-config '{"num_speculative_tokens":5,"method":"dfly"}'
```

## Guide

> Background reference. Commands in it are examples and may target another configuration — use the Serve section above.

### Overview

HY4 Flash is Tencent's multimodal Mixture-of-Experts model for everyday
productivity and agent workflows: 220B total parameters (including 64B of
Engram embeddings), 12B active per token, and up to 1M tokens of context.

The model has 42 decoder layers with hybrid sliding-window and compressed
sparse attention, Engram embeddings, and a five-layer DFly draft head. Both
checkpoints run with BF16 compute:

- [HY4 Flash BF16](https://huggingface.co/tencent/Hy4-flash) — the default variant
- [HY4 Flash FP8](https://huggingface.co/tencent/Hy4-flash-FP8) — the FP8 variant

DFly speculative decoding is on by default; turn off speculative decoding to
serve the target model alone. MTP is not supported by this implementation.

### Prerequisites

- **Hardware**: Hopper or Blackwell data-center GPUs (H100 / H200 / B200 /
  GB200 / B300 / GB300). AMD Instinct is not supported yet: no ROCm image
  ships HY4 Flash.
- **vLLM**: the dedicated `vllm/vllm-openai:hy4-flash` Docker image. No
  released vLLM wheel registers `HYV4FlashVLForConditionalGeneration` or the
  DFly drafter yet, so pip installs are not supported.
- **Host memory**: Engram tables are offloaded to host RAM by default, so
  provision host memory in addition to GPU memory.
- **Access**: the checkpoints may require Hugging Face authentication with
  access to the selected repository.

### Supported features

- Tensor parallelism, with optional expert parallelism (TP / TEP strategies).
- Data parallelism, pipeline parallelism, context parallelism, EPLB, DBO
  and LoRA are not supported.
- HY4 uses a paired tool-call / reasoning parser (`hy_v4`): turning on either
  Tool calling or Reasoning configures both parsers; Tool calling additionally
  adds `--enable-auto-tool-choice`.

### Context and memory

The checkpoint advertises 1,048,576 positions. The recipe starts with an
8,192-token context, four concurrent sequences and one image (no video) per
prompt; raise `--max-model-len`, `--max-num-seqs` and `--limit-mm-per-prompt`
to match your memory budget and workload.

The VRAM hints (529 GB BF16, 345 GB FP8) are the checkpoint download sizes
(440.5 GB and 287.1 GB) × 1.2. They are conservative aggregate estimates, not
measured GPU minima: Engram storage offloaded to the CPU lowers the actual GPU
footprint.

### Client usage

Use the selected checkpoint ID as the model name (`tencent/Hy4-flash` for the
default variant, `tencent/Hy4-flash-FP8` for FP8):

```python
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
resp = client.chat.completions.create(
    model="tencent/Hy4-flash",
    messages=[{
        "role": "user",
        "content": [
            {"type": "image_url", "image_url": {"url": "https://example.com/chart.png"}},
            {"type": "text", "text": "Summarize this chart."},
        ],
    }],
)
print(resp.choices[0].message.content)
```

### References

- [HY4 Flash research page](https://hy.tencent.ai/research/hy4-flash?langVersion=en)
- [BF16 checkpoint](https://huggingface.co/tencent/Hy4-flash)
- [FP8 checkpoint](https://huggingface.co/tencent/Hy4-flash-FP8)
