{ "hf_id": "inclusionAI/Ling-2.6-flash", "meta": { "title": "Ling-2.6-flash", "slug": "ling-2.6-flash", "provider": "inclusionAI", "description": "Ling-2.6-flash (BailingMoeV2_5) instruct model with 104B total / 7.4B active params, hybrid linear + MLA attention, 128K context, optimized for agent workloads", "date_added": "2026-04-28", "date_updated": "2026-05-13", "tasks": [ "text" ], "related_recipes": [ "inclusionAI/Ring-1T-FP8" ], "hardware": { "h200": "verified", "mi300x": "verified" } }, "recommended_command": { "hardware": "h200", "strategy": "single_node_tp", "variant": "default", "node_count": 1, "deploy_type": "single_node", "env": {}, "docker_image": "vllm/vllm-openai:latest", "command": "vllm serve inclusionAI/Ling-2.6-flash \\\n --trust-remote-code \\\n --tensor-parallel-size 4", "argv": [ "vllm", "serve", "inclusionAI/Ling-2.6-flash", "--trust-remote-code", "--tensor-parallel-size", "4" ], "docker_command": "docker run --gpus all \\\n --privileged --ipc=host -p 8000:8000 \\\n -v ~/.cache/huggingface:/root/.cache/huggingface \\\n vllm/vllm-openai:latest inclusionAI/Ling-2.6-flash \\\n --trust-remote-code \\\n --tensor-parallel-size 4", "docker_argv": [ "docker", "run", "--gpus", "all", "--privileged", "--ipc=host", "-p", "8000:8000", "-v", "~/.cache/huggingface:/root/.cache/huggingface", "vllm/vllm-openai:latest", "inclusionAI/Ling-2.6-flash", "--trust-remote-code", "--tensor-parallel-size", "4" ], "strategy_spec": { "name": "single_node_tp", "deploy_type": "single_node", "display_name": "Tensor Parallel", "short_label": "TP", "orientation": "latency", "description": "Single-node tensor parallel. Splits the model across all local GPUs. TP size is set to the GPU count at deploy time. The simplest multi-GPU strategy — works for all model architectures.\n", "hardware_match": { "min_gpus": 1, "max_gpus": 8, "multi_node": false }, "vllm_args": [], "parallel_flag": "--tensor-parallel-size" }, "hardware_profile": { "brand": "NVIDIA", "generation": "hopper", "compute_arch": "sm90", "display_name": "H200", "description": "NVIDIA H200 SXM 141 GB HBM3e · 8-GPU HGX node", "gpu_count": 8, "vram_gb": 1128, "multi_node": false }, "alternatives": { "multi_node_tp": "/inclusionAI/Ling-2.6-flash/strategies/multi_node_tp.json", "multi_node_tp_pp": "/inclusionAI/Ling-2.6-flash/strategies/multi_node_tp_pp.json", "multi_node_tep": "/inclusionAI/Ling-2.6-flash/strategies/multi_node_tep.json", "multi_node_dep": "/inclusionAI/Ling-2.6-flash/strategies/multi_node_dep.json", "kv_store_centralized_mooncake": "/inclusionAI/Ling-2.6-flash/strategies/kv_store_centralized_mooncake.json", "kv_store_distributed_mooncake": "/inclusionAI/Ling-2.6-flash/strategies/kv_store_distributed_mooncake.json" }, "by_hardware": { "h200": "/inclusionAI/Ling-2.6-flash/hw/h200.json", "h100": "/inclusionAI/Ling-2.6-flash/hw/h100.json", "b200": "/inclusionAI/Ling-2.6-flash/hw/b200.json", "gb200": "/inclusionAI/Ling-2.6-flash/hw/gb200.json", "b300": "/inclusionAI/Ling-2.6-flash/hw/b300.json", "gb300": "/inclusionAI/Ling-2.6-flash/hw/gb300.json", "mi300x": "/inclusionAI/Ling-2.6-flash/hw/mi300x.json", "mi325x": "/inclusionAI/Ling-2.6-flash/hw/mi325x.json", "mi350x": "/inclusionAI/Ling-2.6-flash/hw/mi350x.json", "mi355x": "/inclusionAI/Ling-2.6-flash/hw/mi355x.json" } }, "model": { "model_id": "inclusionAI/Ling-2.6-flash", "min_vllm_version": "0.20.2", "architecture": "moe", "parameter_count": "104B", "active_parameters": "7.4B", "context_length": 131072, "base_args": [ "--trust-remote-code", "--tensor-parallel-size", "4" ], "base_env": {}, "install": { "pip": { "command": "uv venv\nsource .venv/bin/activate\nuv pip install -U vllm --torch-backend auto" }, "docker": { "command": "docker pull vllm/vllm-openai:latest" } } }, "features": {}, "opt_in_features": [], "variants": { "default": { "precision": "bf16", "vram_minimum_gb": 250, "description": "BF16 weights with base TP=4; the guide shows a tested 2-GPU AMD command" } }, "hardware_overrides": { "amd": { "extra_env": { "VLLM_ROCM_USE_AITER": "1" } } }, "guide": "## Overview\n\n[Ling-2.6-flash](https://huggingface.co/inclusionAI/Ling-2.6-flash) is a\nBailingMoeV2_5 MoE instruct model with 104B total / 7.4B active parameters,\nhybrid linear + MLA attention, and a 131K context window.\n\n## Deployment Configurations\n\n### Docker (AMD MI300X / MI325X / MI355X, TP=2)\n\nMI300X / MI325X / MI355X GPUs have larger per-GPU HBM, so TP=2 fits the full\n131K context.\n\n```bash\ndocker run --rm -it \\\n --cap-add=SYS_PTRACE \\\n --ipc=host \\\n --privileged=true \\\n --shm-size=128GB \\\n --network=host \\\n --device=/dev/kfd \\\n --device=/dev/dri \\\n --group-add video \\\n -e VLLM_ROCM_USE_AITER=1 \\\n vllm/vllm-openai-rocm:v0.28.0 \\\n inclusionAI/Ling-2.6-flash \\\n --tensor-parallel-size 2 \\\n --trust-remote-code\n```\n\n## Client Usage\n\n### Text Generation\n```python\nfrom openai import OpenAI\nclient = OpenAI(base_url=\"http://localhost:8000/v1\", api_key=\"EMPTY\")\nresponse = client.chat.completions.create(\n model=\"inclusionAI/Ling-2.6-flash\",\n messages=[{\"role\": \"user\", \"content\": \"Write a poem about the ocean.\"}],\n max_tokens=512, temperature=0.7,\n)\nprint(response.choices[0].message.content)\n```\n\n## References\n\n- [Ling-2.6-flash on Hugging Face](https://huggingface.co/inclusionAI/Ling-2.6-flash)\n- [vLLM ROCm GPU install docs](https://docs.vllm.ai/en/latest/getting_started/installation/gpu/?device=rocm)\n- [ROCm vLLM optimization guide](https://rocm.docs.amd.com/en/latest/how-to/rocm-for-ai/inference-optimization/vllm-optimization.html)\n", "source": { "source": "vllm", "repo": "vllm-project/vllm", "ref": "main", "sha": "e0d00e4b8eb0c89ec30921d0695cfdd409e9c211", "path": "recipes/models/inclusionAI/Ling-2.6-flash.yaml", "edit_url": "https://github.com/vllm-project/vllm/edit/main/recipes/models/inclusionAI/Ling-2.6-flash.yaml", "overlays": [] } }