{ "hf_id": "inclusionAI/Ling-3.0-tiny", "meta": { "title": "Ling-3.0-tiny", "slug": "ling-3-0-tiny", "provider": "inclusionAI", "description": "Ling-3.0-tiny lightweight hybrid-reasoning MoE with BF16, block-FP8, and compressed-tensors INT4 checkpoints", "date_added": "2026-08-31", "date_updated": "2026-09-09", "tasks": [ "text" ], "performance_headline": "7.9B total / 1.3B active parameters with a 3:1 KDA-MLA stack; all three official checkpoints run with TP1", "related_recipes": [ "inclusionAI/Ling-3.0-flash" ], "hardware": { "h200": "verified" } }, "recommended_command": { "hardware": "h200", "strategy": "single_node_tp", "variant": "default", "node_count": 1, "deploy_type": "single_node", "env": {}, "docker_image": "vllm/vllm-openai:latest", "command": "vllm serve inclusionAI/Ling-3.0-tiny \\\n --trust-remote-code \\\n --dtype bfloat16 \\\n --gpu-memory-utilization 0.85 \\\n --enable-prefix-caching \\\n --tensor-parallel-size 1 \\\n --enable-auto-tool-choice \\\n --tool-call-parser ling3 \\\n --reasoning-parser ling3", "argv": [ "vllm", "serve", "inclusionAI/Ling-3.0-tiny", "--trust-remote-code", "--dtype", "bfloat16", "--gpu-memory-utilization", "0.85", "--enable-prefix-caching", "--tensor-parallel-size", "1", "--enable-auto-tool-choice", "--tool-call-parser", "ling3", "--reasoning-parser", "ling3" ], "docker_command": "docker run --gpus all \\\n --privileged --ipc=host -p 8000:8000 \\\n -v ~/.cache/huggingface:/root/.cache/huggingface \\\n vllm/vllm-openai:latest inclusionAI/Ling-3.0-tiny \\\n --trust-remote-code \\\n --dtype bfloat16 \\\n --gpu-memory-utilization 0.85 \\\n --enable-prefix-caching \\\n --tensor-parallel-size 1 \\\n --enable-auto-tool-choice \\\n --tool-call-parser ling3 \\\n --reasoning-parser ling3", "docker_argv": [ "docker", "run", "--gpus", "all", "--privileged", "--ipc=host", "-p", "8000:8000", "-v", "~/.cache/huggingface:/root/.cache/huggingface", "vllm/vllm-openai:latest", "inclusionAI/Ling-3.0-tiny", "--trust-remote-code", "--dtype", "bfloat16", "--gpu-memory-utilization", "0.85", "--enable-prefix-caching", "--tensor-parallel-size", "1", "--enable-auto-tool-choice", "--tool-call-parser", "ling3", "--reasoning-parser", "ling3" ], "strategy_spec": { "name": "single_node_tp", "deploy_type": "single_node", "display_name": "Tensor Parallel", "short_label": "TP", "orientation": "latency", "description": "Single-node tensor parallel. Splits the model across all local GPUs. TP size is set to the GPU count at deploy time. The simplest multi-GPU strategy — works for all model architectures.\n", "hardware_match": { "min_gpus": 1, "max_gpus": 8, "multi_node": false }, "vllm_args": [], "parallel_flag": "--tensor-parallel-size" }, "hardware_profile": { "brand": "NVIDIA", "generation": "hopper", "compute_arch": "sm90", "display_name": "H200", "description": "NVIDIA H200 SXM 141 GB HBM3e · 8-GPU HGX node", "gpu_count": 8, "vram_gb": 1128, "multi_node": false }, "alternatives": { "kv_store_centralized_mooncake": "/inclusionAI/Ling-3.0-tiny/strategies/kv_store_centralized_mooncake.json", "kv_store_distributed_mooncake": "/inclusionAI/Ling-3.0-tiny/strategies/kv_store_distributed_mooncake.json" }, "by_hardware": { "h200": "/inclusionAI/Ling-3.0-tiny/hw/h200.json", "h100": "/inclusionAI/Ling-3.0-tiny/hw/h100.json", "b200": "/inclusionAI/Ling-3.0-tiny/hw/b200.json", "gb200": "/inclusionAI/Ling-3.0-tiny/hw/gb200.json", "b300": "/inclusionAI/Ling-3.0-tiny/hw/b300.json", "gb300": "/inclusionAI/Ling-3.0-tiny/hw/gb300.json", "mi300x": "/inclusionAI/Ling-3.0-tiny/hw/mi300x.json", "mi325x": "/inclusionAI/Ling-3.0-tiny/hw/mi325x.json", "mi350x": "/inclusionAI/Ling-3.0-tiny/hw/mi350x.json", "mi355x": "/inclusionAI/Ling-3.0-tiny/hw/mi355x.json" } }, "model": { "model_id": "inclusionAI/Ling-3.0-tiny", "min_vllm_version": "0.28.0", "architecture": "moe", "parameter_count": "7.9B", "active_parameters": "1.3B", "context_length": 131072, "base_args": [ "--trust-remote-code", "--dtype", "bfloat16", "--gpu-memory-utilization", "0.85", "--enable-prefix-caching" ], "install": { "pip": { "command": "uv venv\nsource .venv/bin/activate\nuv pip install -U vllm --torch-backend auto" }, "docker": { "command": "docker pull vllm/vllm-openai:latest" } } }, "features": { "tool_calling": { "description": "Enable Ling 3 automatic tool calling", "args": [ "--enable-auto-tool-choice", "--tool-call-parser", "ling3" ] }, "reasoning": { "description": "Split Ling 3 thinking traces into reasoning", "args": [ "--reasoning-parser", "ling3" ] } }, "opt_in_features": [], "variants": { "default": { "precision": "bf16", "vram_minimum_gb": 18, "tp": 1, "description": "Official BF16 checkpoint; approximately 15.8 GB of repository weights and served with TP1" }, "fp8": { "model_id": "inclusionAI/Ling-3.0-tiny-fp8", "precision": "fp8", "vram_minimum_gb": 11, "tp": 1, "description": "Official serialized block-FP8 checkpoint with dynamic activations and 128x128 weight blocks; served with TP1", "json": "/inclusionAI/Ling-3.0-tiny-fp8.json" }, "int4": { "model_id": "inclusionAI/Ling-3.0-tiny-int4", "precision": "int4", "vram_minimum_gb": 8, "tp": 1, "description": "Official compressed-tensors group-wise INT4 checkpoint; served with TP1", "json": "/inclusionAI/Ling-3.0-tiny-int4.json" } }, "hardware_overrides": {}, "guide": "## Overview\n\n`inclusionAI/Ling-3.0-tiny` is the lightweight member of the Ling 3.0\nfamily. It has 7.9B total parameters and activates 1.3B parameters per token.\nIts 24-layer hybrid stack repeats three Kimi Delta Attention (KDA) layers and\none Multi-Head Latent Attention (MLA) layer, while each sparse MoE block\nselects 8 of 128 routed experts plus one shared expert.\n\nThe official release provides three served checkpoints:\n\n- `inclusionAI/Ling-3.0-tiny`: BF16;\n- `inclusionAI/Ling-3.0-tiny-fp8`: serialized block FP8;\n- `inclusionAI/Ling-3.0-tiny-int4`: compressed-tensors INT4.\n\nAll three use the same `BailingMoeV3ForCausalLM` architecture and a native\n131,072-token context window. Select the corresponding Variant above; the\nserved model ID changes automatically.\n\n## Prerequisites\n\n- **vLLM:** 0.28.0 or newer, which includes Ling 3.0 / Bailing V3\n hybrid-attention support;\n- **Parallelism:** TP1 is recommended for all three checkpoints;\n- **Thinking mode:** enabled per request by the chat template.\n\n## Launching the Server\n\nBF16 on one GPU:\n\n```bash\nvllm serve inclusionAI/Ling-3.0-tiny \\\n --trust-remote-code \\\n --dtype bfloat16 \\\n --tensor-parallel-size 1 \\\n --gpu-memory-utilization 0.85 \\\n --enable-prefix-caching \\\n --enable-auto-tool-choice \\\n --tool-call-parser ling3 \\\n --reasoning-parser ling3\n```\n\nFor FP8 or INT4, keep the same arguments and replace the served checkpoint:\n\n```bash\n# Serialized block FP8\nvllm serve inclusionAI/Ling-3.0-tiny-fp8 \\\n --trust-remote-code --dtype bfloat16 --tensor-parallel-size 1 \\\n --gpu-memory-utilization 0.85 --enable-prefix-caching --enable-auto-tool-choice \\\n --tool-call-parser ling3 --reasoning-parser ling3\n\n# Compressed-tensors INT4\nvllm serve inclusionAI/Ling-3.0-tiny-int4 \\\n --trust-remote-code --dtype bfloat16 --tensor-parallel-size 1 \\\n --gpu-memory-utilization 0.85 --enable-prefix-caching --enable-auto-tool-choice \\\n --tool-call-parser ling3 --reasoning-parser ling3\n```\n\n## Thinking Mode\n\nThinking is controlled per request. The model card recommends\n`temperature=1.0`, `top_p=0.95`, and `top_k=20` when thinking is enabled:\n\n```python\nfrom openai import OpenAI\n\nclient = OpenAI(base_url=\"http://localhost:8000/v1\", api_key=\"EMPTY\")\nresponse = client.chat.completions.create(\n model=\"inclusionAI/Ling-3.0-tiny\",\n messages=[{\"role\": \"user\", \"content\": \"Solve the problem step by step.\"}],\n temperature=1.0,\n top_p=0.95,\n extra_body={\n \"top_k\": 20,\n \"chat_template_kwargs\": {\"enable_thinking\": True},\n },\n)\nmsg = response.choices[0].message\nprint(msg.reasoning)\nprint(msg.content)\n```\n\nSet `enable_thinking` to `false` for direct answers. When using a quantized\nvariant, set the client `model` to its FP8 or INT4 checkpoint ID.\n\n## References\n\n- [Ling 3.0 collection](https://huggingface.co/collections/inclusionAI/ling-30)\n- [Ling-3.0-tiny on Hugging Face](https://huggingface.co/inclusionAI/Ling-3.0-tiny)\n- [Ling-3.0-tiny-fp8 on Hugging Face](https://huggingface.co/inclusionAI/Ling-3.0-tiny-fp8)\n- [Ling-3.0-tiny-int4 on Hugging Face](https://huggingface.co/inclusionAI/Ling-3.0-tiny-int4)\n", "source": { "source": "vllm", "repo": "vllm-project/vllm", "ref": "main", "sha": "564ac0de1b228568cafcbfbe8ccaccc82d6023eb", "path": "recipes/models/inclusionAI/Ling-3.0-tiny.yaml", "edit_url": "https://github.com/vllm-project/vllm/edit/main/recipes/models/inclusionAI/Ling-3.0-tiny.yaml", "overlays": [] } }