{
  "hf_id": "tencent/Hy4",
  "meta": {
    "title": "HY4",
    "slug": "hy4",
    "provider": "Hunyuan (Tencent)",
    "description": "Tencent HY4 flagship multimodal MoE — 774B total / 49B active, 1M context, hy_v4 tool and reasoning parsers, DFly2 speculative decoding",
    "date_added": "2026-11-20",
    "date_updated": "2026-11-20",
    "tasks": [
      "text",
      "multimodal"
    ],
    "related_recipes": [
      "tencent/Hy4-flash",
      "tencent/Hy4-preview"
    ],
    "default_hardware": "b300",
    "hardware": {
      "mi300x": "unsupported",
      "mi325x": "unsupported",
      "mi350x": "unsupported",
      "mi355x": "unsupported"
    }
  },
  "recommended_command": {
    "hardware": "b300",
    "strategy": "single_node_tp",
    "variant": "default",
    "node_count": 1,
    "deploy_type": "single_node",
    "env": {
      "VLLM_USE_RUST_FRONTEND": "1"
    },
    "docker_image": "vllm/vllm-openai:hy4",
    "command": "vllm serve tencent/Hy4-FP8 \\\n  --dtype bfloat16 \\\n  --max-model-len 131072 \\\n  --max-num-seqs 32 \\\n  --max-num-batched-tokens 8192 \\\n  --attention-backend FLASHMLA_SPARSE \\\n  --limit-mm-per-prompt '{\"image\":1,\"video\":0}' \\\n  --tensor-parallel-size 8 \\\n  --enable-auto-tool-choice \\\n  --tool-call-parser hy_v4 \\\n  --reasoning-parser hy_v4 \\\n  --speculative-config '{\"num_speculative_tokens\":5,\"method\":\"dfly\"}'",
    "argv": [
      "vllm",
      "serve",
      "tencent/Hy4-FP8",
      "--dtype",
      "bfloat16",
      "--max-model-len",
      "131072",
      "--max-num-seqs",
      "32",
      "--max-num-batched-tokens",
      "8192",
      "--attention-backend",
      "FLASHMLA_SPARSE",
      "--limit-mm-per-prompt",
      "{\"image\":1,\"video\":0}",
      "--tensor-parallel-size",
      "8",
      "--enable-auto-tool-choice",
      "--tool-call-parser",
      "hy_v4",
      "--reasoning-parser",
      "hy_v4",
      "--speculative-config",
      "{\"num_speculative_tokens\":5,\"method\":\"dfly\"}"
    ],
    "docker_command": "docker run --gpus all \\\n  --privileged --ipc=host -p 8000:8000 \\\n  -v ~/.cache/huggingface:/root/.cache/huggingface \\\n  -e VLLM_USE_RUST_FRONTEND=1 \\\n  vllm/vllm-openai:hy4 tencent/Hy4-FP8 \\\n  --dtype bfloat16 \\\n  --max-model-len 131072 \\\n  --max-num-seqs 32 \\\n  --max-num-batched-tokens 8192 \\\n  --attention-backend FLASHMLA_SPARSE \\\n  --limit-mm-per-prompt '{\"image\":1,\"video\":0}' \\\n  --tensor-parallel-size 8 \\\n  --enable-auto-tool-choice \\\n  --tool-call-parser hy_v4 \\\n  --reasoning-parser hy_v4 \\\n  --speculative-config '{\"num_speculative_tokens\":5,\"method\":\"dfly\"}'",
    "docker_argv": [
      "docker",
      "run",
      "--gpus",
      "all",
      "--privileged",
      "--ipc=host",
      "-p",
      "8000:8000",
      "-v",
      "~/.cache/huggingface:/root/.cache/huggingface",
      "-e",
      "VLLM_USE_RUST_FRONTEND=1",
      "vllm/vllm-openai:hy4",
      "tencent/Hy4-FP8",
      "--dtype",
      "bfloat16",
      "--max-model-len",
      "131072",
      "--max-num-seqs",
      "32",
      "--max-num-batched-tokens",
      "8192",
      "--attention-backend",
      "FLASHMLA_SPARSE",
      "--limit-mm-per-prompt",
      "{\"image\":1,\"video\":0}",
      "--tensor-parallel-size",
      "8",
      "--enable-auto-tool-choice",
      "--tool-call-parser",
      "hy_v4",
      "--reasoning-parser",
      "hy_v4",
      "--speculative-config",
      "{\"num_speculative_tokens\":5,\"method\":\"dfly\"}"
    ],
    "strategy_spec": {
      "name": "single_node_tp",
      "deploy_type": "single_node",
      "display_name": "Tensor Parallel",
      "short_label": "TP",
      "orientation": "latency",
      "description": "Single-node tensor parallel. Splits the model across all local GPUs. TP size is set to the GPU count at deploy time. The simplest multi-GPU strategy — works for all model architectures.\n",
      "hardware_match": {
        "min_gpus": 1,
        "max_gpus": 8,
        "multi_node": false
      },
      "vllm_args": [],
      "parallel_flag": "--tensor-parallel-size"
    },
    "hardware_profile": {
      "brand": "NVIDIA",
      "generation": "blackwell",
      "compute_arch": "sm103",
      "display_name": "B300",
      "description": "NVIDIA B300 SXM 268 GB HBM3e · 8-GPU HGX B300 node (Blackwell Ultra)",
      "gpu_count": 8,
      "vram_gb": 2144,
      "multi_node": false
    },
    "alternatives": {
      "kv_store_centralized_mooncake": "/tencent/Hy4/strategies/kv_store_centralized_mooncake.json",
      "kv_store_distributed_mooncake": "/tencent/Hy4/strategies/kv_store_distributed_mooncake.json"
    },
    "by_hardware": {
      "b300": "/tencent/Hy4/hw/b300.json",
      "h100": "/tencent/Hy4/hw/h100.json",
      "h200": "/tencent/Hy4/hw/h200.json",
      "b200": "/tencent/Hy4/hw/b200.json",
      "gb200": "/tencent/Hy4/hw/gb200.json",
      "gb300": "/tencent/Hy4/hw/gb300.json"
    }
  },
  "model": {
    "model_id": "tencent/Hy4",
    "min_vllm_version": "0.35.0",
    "nightly_required": true,
    "docker_image": "vllm/vllm-openai:hy4",
    "architecture": "moe",
    "parameter_count": "774B",
    "active_parameters": "49B",
    "context_length": 1048576,
    "base_args": [
      "--dtype",
      "bfloat16",
      "--max-model-len",
      "131072",
      "--max-num-seqs",
      "32",
      "--max-num-batched-tokens",
      "8192",
      "--attention-backend",
      "FLASHMLA_SPARSE",
      "--limit-mm-per-prompt",
      "{\"image\":1,\"video\":0}"
    ],
    "default_frontend": "rust",
    "install": {
      "pip": {
        "command": "uv venv\nsource .venv/bin/activate\nuv pip install -U vllm --pre \\\n  --extra-index-url https://wheels.vllm.ai/nightly/cu130 \\\n  --extra-index-url https://download.pytorch.org/whl/cu130 \\\n  --index-strategy unsafe-best-match",
        "note": "v0.34.0 does not register HY4; nightly wheels built from 2026-11-20 on do"
      },
      "docker": {
        "command": "docker pull vllm/vllm-openai:hy4",
        "note": "Official HY4 image from the model card (amd64 and arm64)"
      }
    }
  },
  "features": {
    "tool_calling": {
      "description": "HY4 tool call parser with automatic tool choice (also sets the paired hy_v4 reasoning parser)",
      "args": [
        "--enable-auto-tool-choice",
        "--tool-call-parser",
        "hy_v4",
        "--reasoning-parser",
        "hy_v4"
      ]
    },
    "reasoning": {
      "description": "HY4 reasoning parser — returns the chain of thought separately from the final answer",
      "args": [
        "--reasoning-parser",
        "hy_v4"
      ]
    },
    "spec_decoding": {
      "description": "DFly2 speculative decoding with the checkpoint's built-in five-layer draft (5 speculative tokens)",
      "args": [
        "--speculative-config",
        "{\"num_speculative_tokens\":5,\"method\":\"dfly\"}"
      ]
    }
  },
  "variants": {
    "default": {
      "model_id": "tencent/Hy4-FP8",
      "precision": "fp8",
      "vram_minimum_gb": 944,
      "description": "Block-FP8 checkpoint (tencent/Hy4-FP8), BF16 compute — 786 GB of weights",
      "json": "/tencent/Hy4-FP8.json"
    },
    "bf16": {
      "precision": "bf16",
      "vram_minimum_gb": 1859,
      "description": "BF16 checkpoint (tencent/Hy4) — 1,549 GB of weights; needs an 8x B300 node"
    }
  },
  "notes": [
    {
      "title": "Host memory for the BF16 checkpoint",
      "when": {
        "variants": [
          "bf16"
        ]
      },
      "body": "The BF16 checkpoint is 1,549 GB across 245 shards. Provision host RAM and\npage cache for loading it; the evaluation host had 2 TiB.\n"
    }
  ],
  "guide": "## Overview\n\nHY4 is the Tencent Hy Team's flagship Mixture-of-Experts model for text and\nimage understanding, with up to 1M tokens of context. Two checkpoints share\none architecture (`HYV4VLForConditionalGeneration`):\n\n- [HY4 FP8](https://huggingface.co/tencent/Hy4-FP8) — the default variant,\n  and the checkpoint the model card serves\n- [HY4 BF16](https://huggingface.co/tencent/Hy4) — the BF16 variant\n\nThe language backbone has 770B parameters, 49B of them active per token,\nacross 78 layers. The first layer has a dense FFN; the other 77 use 256\nrouted experts (top-8) and one shared expert. Attention is gated sparse MLA\nwith cross-layer index reuse, and iHC connects the residual streams. A\n40-layer vision encoder (1.23B) and the five-layer DFly2 draft (3.3B) bring\nthe stored total to about 774B.\n\nHY4 is not [HY4 Flash](https://recipes.vllm.ai/tencent/Hy4-flash) or the\ntext-only [Hy4-preview](https://recipes.vllm.ai/tencent/Hy4-preview). Those\nare separate checkpoints with their own recipes.\n\n## Prerequisites\n\n- **Hardware**: one 8-GPU NVIDIA node. FP8 needs about 944 GB of aggregate GPU\n  memory and BF16 about 1,859 GB (8x B300). AMD Instinct is not supported:\n  vLLM's ROCm port has neither the vision encoder nor the DFly2 drafter.\n- **vLLM**: the `vllm/vllm-openai:hy4` image, or a nightly wheel. v0.34.0\n  does not register the architecture; 0.35.0 will be the first release\n  with it.\n- **Host memory**: enough RAM to load the checkpoint — 786 GB (FP8) or\n  1,549 GB (BF16) on disk, 245 safetensors shards each.\n\n## Serving configuration\n\nThe defaults follow the model card's launch line: FP8 checkpoint, TP8, Rust\nfrontend, `FLASHMLA_SPARSE` attention, BF16 compute, tool calling, reasoning\nparsing, and DFly2 with five speculative tokens.\n\n- **Parsers**: HY4 pairs its tool-call and reasoning parsers (`hy_v4`). Tool\n  calling sets both and adds automatic tool choice.\n- **DFly2**: the draft ships inside both checkpoints and shares the target's\n  KV cache and sparse indices, so it needs no extra checkpoint or KV memory.\n  It requires data, pipeline, prefill-context and decode-context parallel\n  sizes of 1, which is why the recipe offers single-node TP only. The\n  checkpoints have no MTP layers (`num_nextn_predict_layers=0`), so `mtp`\n  is not an alternative method.\n- **FP8**: blockwise W8A8 compressed-tensors quantization; `lm_head`, the\n  vision tower, the first layer's attention and the indexer projections stay\n  unquantized. vLLM reads the scheme from the checkpoint, so no quantization\n  flag is needed. It is not the MXFP8 format of Hy4-preview-FP8.\n- **Multimodal**: one image per prompt, video off, as on the model card.\n  Raise `--limit-mm-per-prompt` to allow more.\n\n## Context and capacity\n\nThe checkpoints advertise 1,048,576 positions. The recipe starts from the\nevaluated window: 131,072 tokens, 32 concurrent sequences and 8,192 batched\ntokens. The model card's launch line sets none of the three, which serves the\nfull 1M window; raise them to match your memory budget. A full 1M-token\nrequest has not been validated for this recipe.\n\n## Client usage\n\nThe model name is the selected checkpoint ID: `tencent/Hy4-FP8` by default,\n`tencent/Hy4` for the BF16 variant. Tencent recommends `temperature=0.9` and\n`top_p=1.0`.\n\n```python\nfrom openai import OpenAI\n\nclient = OpenAI(base_url=\"http://localhost:8000/v1\", api_key=\"EMPTY\")\nresponse = client.chat.completions.create(\n    model=\"tencent/Hy4-FP8\",\n    messages=[{\n        \"role\": \"user\",\n        \"content\": [\n            {\"type\": \"image_url\", \"image_url\": {\"url\": \"https://example.com/chart.png\"}},\n            {\"type\": \"text\", \"text\": \"Summarize this chart.\"},\n        ],\n    }],\n    temperature=0.9,\n    top_p=1.0,\n)\nprint(response.choices[0].message.reasoning)  # chain of thought\nprint(response.choices[0].message.content)    # final answer\n```\n\n`reasoning_effort` accepts `high` (the default, for math, coding and other\ncomplex tasks) and `no_think` for a direct answer:\n\n```python\nresponse = client.chat.completions.create(\n    model=\"tencent/Hy4-FP8\",\n    messages=[{\"role\": \"user\", \"content\": \"What is 17 multiplied by 23?\"}],\n    temperature=0.9,\n    top_p=1.0,\n    extra_body={\"chat_template_kwargs\": {\"reasoning_effort\": \"no_think\"}},\n)\nprint(response.choices[0].message.content)\n```\n\n## References\n\n- [HY4 research page](https://hy.tencent.ai/research/hy4)\n- [BF16 checkpoint](https://huggingface.co/tencent/Hy4)\n- [FP8 checkpoint](https://huggingface.co/tencent/Hy4-FP8)\n- [GitHub: Tencent-Hunyuan/Hy4](https://github.com/Tencent-Hunyuan/Hy4)\n- [AngelSpec, the DFly2 training toolkit](https://github.com/Tencent/AngelSpec)\n",
  "source": {
    "source": "vllm",
    "repo": "vllm-project/vllm",
    "ref": "main",
    "sha": "cf0a814985c8107e905cde10ae43d8f7bc0c0675",
    "path": "recipes/models/tencent/Hy4.yaml",
    "edit_url": "https://github.com/vllm-project/vllm/edit/main/recipes/models/tencent/Hy4.yaml",
    "overlays": []
  }
}