Hy3 Usage Guide

Hy3 is Tencent's latest open-source Mixture-of-Experts language model: 295B total parameters with 21B activated per token, plus a 3.8B MTP layer for speculative decoding. 80 transformer layers, 192 routed experts (top-8) + 1 shared expert, GQA with 64 heads over 8 KV heads, 256K context.

Availability

Hy3 uses the same model code as Hy3-preview, so any vLLM build that serves Hy3-preview also serves Hy3. The latest optimizations — Tencent's HPC-Ops attention and MoE backends — require a newer vLLM that includes #47433 plus the separately-built HPC-Ops kernels (see the HPC-Ops backends section below).

Setup

Choose one of the following setup methods.

Using Docker

# Switch to the trtllm all-reduce backend to work around an mnnvl workspace-size issue.
docker run --gpus all \
  -p 8000:8000 \
  --ipc=host \
  -e VLLM_FLASHINFER_ALLREDUCE_BACKEND=trtllm \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai:hy3 tencent/Hy3 \
    --tensor-parallel-size 8 \
    --tool-call-parser hy_v3 \
    --reasoning-parser hy_v3 \
    --enable-auto-tool-choice \
    --served-model-name hy3

Installing from source

uv venv --python 3.12 --seed --managed-python
source .venv/bin/activate
git clone https://github.com/vllm-project/vllm.git
cd vllm
uv pip install --editable . --torch-backend=auto

Model Deployment

To serve Hy3 on 8 GPUs, use H20-3e(141GB), H200, AMD MI300X/MI325X (192 GB), AMD MI350X/MI355X (288 GB), or other GPUs with larger memory capacity. Smaller-memory 8-GPU configurations (8×H100 80GB, 8×A100 80GB) do not fit the BF16 weights plus KV cache — use multi-node TP for those.

Serving on 8×AMD MI300X / MI325X / MI350X / MI355X

Serve with the standard launcher plus the AITER environment variables (the recipe's hardware_overrides.amd.extra_env applies these automatically when the AMD profile is selected on the recipe site):

export VLLM_ROCM_USE_AITER=1
export VLLM_ROCM_USE_AITER_MOE=1
export VLLM_ROCM_USE_AITER_MHA=1
export VLLM_ROCM_USE_AITER_RMSNORM=1
export VLLM_ROCM_USE_AITER_LINEAR=1

vllm serve tencent/Hy3 \
  --tensor-parallel-size 8 \
  --tool-call-parser hy_v3 \
  --reasoning-parser hy_v3 \
  --enable-auto-tool-choice \
  --served-model-name hy3 \
  --gpu-memory-utilization 0.90

MTP (recommended on AMD for lower latency, same flags as the NVIDIA path):

vllm serve tencent/Hy3 \
  --tensor-parallel-size 8 \
  --speculative-config.method mtp \
  --speculative-config.num_speculative_tokens 2 \
  --tool-call-parser hy_v3 \
  --reasoning-parser hy_v3 \
  --enable-auto-tool-choice \
  --served-model-name hy3 \
  --gpu-memory-utilization 0.90

Serving on 8×H200 or 8×H20-3e(141GB)

Without Multi-Token Prediction (MTP):

# Switch to the trtllm all-reduce backend to work around an mnnvl workspace-size issue.
export VLLM_FLASHINFER_ALLREDUCE_BACKEND=trtllm
vllm serve tencent/Hy3 \
  --tensor-parallel-size 8 \
  --tool-call-parser hy_v3 \
  --reasoning-parser hy_v3 \
  --enable-auto-tool-choice \
  --served-model-name hy3

With MTP (recommended for lower latency):

# Switch to the trtllm all-reduce backend to work around an mnnvl workspace-size issue.
export VLLM_FLASHINFER_ALLREDUCE_BACKEND=trtllm
vllm serve tencent/Hy3 \
  --tensor-parallel-size 8 \
  --speculative-config.method mtp \
  --speculative-config.num_speculative_tokens 2 \
  --tool-call-parser hy_v3 \
  --reasoning-parser hy_v3 \
  --enable-auto-tool-choice \
  --served-model-name hy3

HPC-Ops backends (Hopper) — maximum throughput

Tencent's HPC-Ops attention and MoE kernels (#47433) reduce latency for mixed-length decode and small-batch MoE on Hopper GPUs. They ship as a separate wheel — build and install it first:

git clone https://github.com/Tencent/hpc-ops.git
cd hpc-ops
make wheel
python3 -m pip install dist/*.whl

Enable the attention backend with --attention-backend HPC_ATTN:

# Switch to the trtllm all-reduce backend to work around an mnnvl workspace-size issue.
export VLLM_FLASHINFER_ALLREDUCE_BACKEND=trtllm
vllm serve tencent/Hy3 \
  --tensor-parallel-size 8 \
  --attention-backend HPC_ATTN

Enable the fused MoE backend with --moe-backend hpc. On the FP8 checkpoint, add the FP8 KV-cache flags --kv-cache-dtype fp8_e4m3 --block-size 64:

# Switch to the trtllm all-reduce backend to work around an mnnvl workspace-size issue.
export VLLM_FLASHINFER_ALLREDUCE_BACKEND=trtllm
vllm serve tencent/Hy3-FP8 \
  --tensor-parallel-size 8 \
  --moe-backend hpc \
  --kv-cache-dtype fp8_e4m3 \
  --block-size 64

Both backends combine with the standard --tool-call-parser / --reasoning-parser and MTP flags shown above. See the vLLM × HPC-Ops post for benchmarks and details.

Sampling and Reasoning Modes

Tencent's recommended sampling parameters: temperature=0.9, top_p=1.0.

Reasoning is controlled via chat_template_kwargs.reasoning_effort:

Value	Behavior
no_think (default)	Direct response, no chain-of-thought
low	Light reasoning
high	Deep chain-of-thought for math/coding/complex reasoning

When tools are registered, set interleaved_thinking: true to allow the model to think between tool calls.

OpenAI Client Example

uv pip install -U openai

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "Hello."},
]

# Direct response (default).
resp = client.chat.completions.create(
    model="hy3",
    messages=messages,
    temperature=0.9,
    top_p=1.0,
    max_tokens=4096,
)
print(resp.choices[0].message.content)

# Deep reasoning: set reasoning_effort (and interleaved_thinking if using tools).
resp_think = client.chat.completions.create(
    model="hy3",
    messages=messages,
    temperature=0.9,
    top_p=1.0,
    max_tokens=4096,
    extra_body={
      "chat_template_kwargs": {
          "reasoning_effort": "high",
          "interleaved_thinking": True,
      },
    },
)
output_msg = resp_think.choices[0].message
print(output_msg.reasoning_content)  # chain-of-thought
print(output_msg.content)             # final answer

cURL Usage

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "hy3",
    "messages": [
      {"role": "system", "content": "You are a helpful assistant."},
      {"role": "user", "content": "Hello."}
    ],
    "temperature": 0.9,
    "top_p": 1.0,
    "max_tokens": 4096
  }'

Benchmarking

For benchmarking, disable prefix caching by adding --no-enable-prefix-caching to the server command.

The following benchmarks the FP8 checkpoint (8192→1024, concurrency 32, 160 prompts):

vllm bench serve \
  --model tencent/Hy3-FP8 \
  --dataset-name random \
  --random-input-len 8192 \
  --random-output-len 1024 \
  --max-concurrency 32 \
  --num-prompts 160 \
  --served-model-name hy3-fp8

Representative output — Hy3-FP8, TP4 on 4×GB300 (one NVL4 tray), trtllm all-reduce, MTP=2:

============ Serving Benchmark Result ============
Successful requests:                     160
Failed requests:                         0
Maximum request concurrency:             32
Benchmark duration (s):                  175.37
Request throughput (req/s):              0.91
Output token throughput (tok/s):         934.26
Peak output token throughput (tok/s):    1344.00
Total token throughput (tok/s):          8408.34
Mean TTFT (ms):                          2352.29
Median TTFT (ms):                         505.64
P99 TTFT (ms):                           14538.85
Mean TPOT (ms):                           30.11
Median TPOT (ms):                          30.86
P99 TPOT (ms):                             40.48
Mean ITL (ms):                            36.26
Median ITL (ms):                          24.09
==================================================

References

• Hugging Face model card
• FP8 checkpoint — tencent/Hy3-FP8
• GitHub: Tencent-Hunyuan/Hy3
• vLLM × HPC-Ops: high-performance attention & MoE backends
• Tencent Hy3