# inclusionAI/Ling-3.0-tiny — vLLM serving recipe

> Ling-3.0-tiny lightweight hybrid-reasoning MoE with BF16, block-FP8, and compressed-tensors INT4 checkpoints

Configuration: hardware `h200` (8× H200) · strategy `single_node_tp` (Tensor Parallel) · variant `default` (BF16) · features `tool_calling`, `reasoning`

- Interactive page: https://recipes.vllm.ai/inclusionAI/Ling-3.0-tiny
- This configuration as JSON: https://recipes.vllm.ai/inclusionAI/Ling-3.0-tiny.json?hardware=h200

The commands below were rendered for exactly this configuration and take precedence over any example in the guide.

## Other configurations

Change the query string (the same parameters as the page URL) and fetch this URL again. Unknown or incompatible values fall back to the recipe default, so check the Configuration line.

- `hardware`: `h100`, `h200`, `b200`, `gb200`, `b300`, `gb300`, `mi300x`, `mi325x`, `mi350x`, `mi355x`
- `variant`: `default`, `fp8`, `int4`
- `strategy`: `single_node_tp` (multi-node strategies also need `nodes=<n>`)
- `features`: comma-separated list of `tool_calling`, `reasoning` (default: `tool_calling`, `reasoning`)

## Install

Requires vLLM >= 0.28.0.

```bash
uv venv
source .venv/bin/activate
uv pip install -U vllm --torch-backend auto
```

Docker image: `vllm/vllm-openai:latest`

## Serve

```bash
vllm serve inclusionAI/Ling-3.0-tiny \
  --trust-remote-code \
  --dtype bfloat16 \
  --gpu-memory-utilization 0.85 \
  --enable-prefix-caching \
  --tensor-parallel-size 1 \
  --enable-auto-tool-choice \
  --tool-call-parser ling3 \
  --reasoning-parser ling3
```

Docker equivalent:

```bash
docker run --gpus all \
  --privileged --ipc=host -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai:latest inclusionAI/Ling-3.0-tiny \
  --trust-remote-code \
  --dtype bfloat16 \
  --gpu-memory-utilization 0.85 \
  --enable-prefix-caching \
  --tensor-parallel-size 1 \
  --enable-auto-tool-choice \
  --tool-call-parser ling3 \
  --reasoning-parser ling3
```

## Guide

> Background reference. Commands in it are examples and may target another configuration — use the Serve section above.

### Overview

`inclusionAI/Ling-3.0-tiny` is the lightweight member of the Ling 3.0
family. It has 7.9B total parameters and activates 1.3B parameters per token.
Its 24-layer hybrid stack repeats three Kimi Delta Attention (KDA) layers and
one Multi-Head Latent Attention (MLA) layer, while each sparse MoE block
selects 8 of 128 routed experts plus one shared expert.

The official release provides three served checkpoints:

- `inclusionAI/Ling-3.0-tiny`: BF16;
- `inclusionAI/Ling-3.0-tiny-fp8`: serialized block FP8;
- `inclusionAI/Ling-3.0-tiny-int4`: compressed-tensors INT4.

All three use the same `BailingMoeV3ForCausalLM` architecture and a native
131,072-token context window. Select the corresponding Variant above; the
served model ID changes automatically.

### Prerequisites

- **vLLM:** 0.28.0 or newer, which includes Ling 3.0 / Bailing V3
  hybrid-attention support;
- **Parallelism:** TP1 is recommended for all three checkpoints;
- **Thinking mode:** enabled per request by the chat template.

### Launching the Server

BF16 on one GPU:

```bash
vllm serve inclusionAI/Ling-3.0-tiny \
  --trust-remote-code \
  --dtype bfloat16 \
  --tensor-parallel-size 1 \
  --gpu-memory-utilization 0.85 \
  --enable-prefix-caching \
  --enable-auto-tool-choice \
  --tool-call-parser ling3 \
  --reasoning-parser ling3
```

For FP8 or INT4, keep the same arguments and replace the served checkpoint:

```bash
# Serialized block FP8
vllm serve inclusionAI/Ling-3.0-tiny-fp8 \
  --trust-remote-code --dtype bfloat16 --tensor-parallel-size 1 \
  --gpu-memory-utilization 0.85 --enable-prefix-caching --enable-auto-tool-choice \
  --tool-call-parser ling3 --reasoning-parser ling3

# Compressed-tensors INT4
vllm serve inclusionAI/Ling-3.0-tiny-int4 \
  --trust-remote-code --dtype bfloat16 --tensor-parallel-size 1 \
  --gpu-memory-utilization 0.85 --enable-prefix-caching --enable-auto-tool-choice \
  --tool-call-parser ling3 --reasoning-parser ling3
```

### Thinking Mode

Thinking is controlled per request. The model card recommends
`temperature=1.0`, `top_p=0.95`, and `top_k=20` when thinking is enabled:

```python
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
response = client.chat.completions.create(
    model="inclusionAI/Ling-3.0-tiny",
    messages=[{"role": "user", "content": "Solve the problem step by step."}],
    temperature=1.0,
    top_p=0.95,
    extra_body={
        "top_k": 20,
        "chat_template_kwargs": {"enable_thinking": True},
    },
)
msg = response.choices[0].message
print(msg.reasoning)
print(msg.content)
```

Set `enable_thinking` to `false` for direct answers. When using a quantized
variant, set the client `model` to its FP8 or INT4 checkpoint ID.

### References

- [Ling 3.0 collection](https://huggingface.co/collections/inclusionAI/ling-30)
- [Ling-3.0-tiny on Hugging Face](https://huggingface.co/inclusionAI/Ling-3.0-tiny)
- [Ling-3.0-tiny-fp8 on Hugging Face](https://huggingface.co/inclusionAI/Ling-3.0-tiny-fp8)
- [Ling-3.0-tiny-int4 on Hugging Face](https://huggingface.co/inclusionAI/Ling-3.0-tiny-int4)