# inclusionAI/Ling-2.6-flash — vLLM serving recipe

> Ling-2.6-flash (BailingMoeV2_5) instruct model with 104B total / 7.4B active params, hybrid linear + MLA attention, 128K context, optimized for agent workloads

Configuration: hardware `h200` (8× H200) · strategy `single_node_tp` (Tensor Parallel) · variant `default` (BF16)

- Interactive page: https://recipes.vllm.ai/inclusionAI/Ling-2.6-flash
- This configuration as JSON: https://recipes.vllm.ai/inclusionAI/Ling-2.6-flash.json?hardware=h200

The commands below were rendered for exactly this configuration and take precedence over any example in the guide.

## Other configurations

Change the query string (the same parameters as the page URL) and fetch this URL again. Unknown or incompatible values fall back to the recipe default, so check the Configuration line.

- `hardware`: `h100`, `h200`, `b200`, `gb200`, `b300`, `gb300`, `mi300x`, `mi325x`, `mi350x`, `mi355x`
- `variant`: `default`
- `strategy`: `single_node_tp`, `multi_node_tp`, `multi_node_tp_pp`, `multi_node_tep`, `multi_node_dep` (multi-node strategies also need `nodes=<n>`)

## Install

Requires vLLM >= 0.20.2.

```bash
uv venv
source .venv/bin/activate
uv pip install -U vllm --torch-backend auto
```

Docker image: `vllm/vllm-openai:latest`

## Serve

```bash
vllm serve inclusionAI/Ling-2.6-flash \
  --trust-remote-code \
  --tensor-parallel-size 4
```

Docker equivalent:

```bash
docker run --gpus all \
  --privileged --ipc=host -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai:latest inclusionAI/Ling-2.6-flash \
  --trust-remote-code \
  --tensor-parallel-size 4
```

## Guide

> Background reference. Commands in it are examples and may target another configuration — use the Serve section above.

### Overview

[Ling-2.6-flash](https://huggingface.co/inclusionAI/Ling-2.6-flash) is a
BailingMoeV2_5 MoE instruct model with 104B total / 7.4B active parameters,
hybrid linear + MLA attention, and a 131K context window.

### Deployment Configurations

#### Docker (AMD MI300X / MI325X / MI355X, TP=2)

MI300X / MI325X / MI355X GPUs have larger per-GPU HBM, so TP=2 fits the full
131K context.

```bash
docker run --rm -it \
  --cap-add=SYS_PTRACE \
  --ipc=host \
  --privileged=true \
  --shm-size=128GB \
  --network=host \
  --device=/dev/kfd \
  --device=/dev/dri \
  --group-add video \
  -e VLLM_ROCM_USE_AITER=1 \
  vllm/vllm-openai-rocm:v0.28.0 \
    inclusionAI/Ling-2.6-flash \
    --tensor-parallel-size 2 \
    --trust-remote-code
```

### Client Usage

#### Text Generation
```python
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
response = client.chat.completions.create(
    model="inclusionAI/Ling-2.6-flash",
    messages=[{"role": "user", "content": "Write a poem about the ocean."}],
    max_tokens=512, temperature=0.7,
)
print(response.choices[0].message.content)
```

### References

- [Ling-2.6-flash on Hugging Face](https://huggingface.co/inclusionAI/Ling-2.6-flash)
- [vLLM ROCm GPU install docs](https://docs.vllm.ai/en/latest/getting_started/installation/gpu/?device=rocm)
- [ROCm vLLM optimization guide](https://rocm.docs.amd.com/en/latest/how-to/rocm-for-ai/inference-optimization/vllm-optimization.html)