inclusionAI/Ling-3.0-tiny

Overview

inclusionAI/Ling-3.0-tiny is the lightweight member of the Ling 3.0 family. It has 7.9B total parameters and activates 1.3B parameters per token. Its 24-layer hybrid stack repeats three Kimi Delta Attention (KDA) layers and one Multi-Head Latent Attention (MLA) layer, while each sparse MoE block selects 8 of 128 routed experts plus one shared expert.

The official release provides three served checkpoints:

• inclusionAI/Ling-3.0-tiny: BF16;
• inclusionAI/Ling-3.0-tiny-fp8: serialized block FP8;
• inclusionAI/Ling-3.0-tiny-int4: compressed-tensors INT4.

All three use the same BailingMoeV3ForCausalLM architecture and a native 131,072-token context window. Select the corresponding Variant above; the served model ID changes automatically.

Prerequisites

• vLLM: use a nightly/source build containing Ling 3.0 / Bailing V3 hybrid-attention support;
• Parallelism: TP1 is recommended for all three checkpoints;
• Thinking mode: enabled per request by the chat template.

Launching the Server

BF16 on one GPU:

vllm serve inclusionAI/Ling-3.0-tiny \
  --trust-remote-code \
  --dtype bfloat16 \
  --tensor-parallel-size 1 \
  --gpu-memory-utilization 0.85 \
  --enable-prefix-caching \
  --enable-auto-tool-choice \
  --tool-call-parser ling3 \
  --reasoning-parser ling3

For FP8 or INT4, keep the same arguments and replace the served checkpoint:

# Serialized block FP8
vllm serve inclusionAI/Ling-3.0-tiny-fp8 \
  --trust-remote-code --dtype bfloat16 --tensor-parallel-size 1 \
  --gpu-memory-utilization 0.85 --enable-prefix-caching --enable-auto-tool-choice \
  --tool-call-parser ling3 --reasoning-parser ling3

# Compressed-tensors INT4
vllm serve inclusionAI/Ling-3.0-tiny-int4 \
  --trust-remote-code --dtype bfloat16 --tensor-parallel-size 1 \
  --gpu-memory-utilization 0.85 --enable-prefix-caching --enable-auto-tool-choice \
  --tool-call-parser ling3 --reasoning-parser ling3

Thinking Mode

Thinking is controlled per request. The model card recommends temperature=1.0, top_p=0.95, and top_k=20 when thinking is enabled:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
response = client.chat.completions.create(
    model="inclusionAI/Ling-3.0-tiny",
    messages=[{"role": "user", "content": "Solve the problem step by step."}],
    temperature=1.0,
    top_p=0.95,
    extra_body={
        "top_k": 20,
        "chat_template_kwargs": {"enable_thinking": True},
    },
)
print(response.choices[0].message.reasoning_content)
print(response.choices[0].message.content)

Set enable_thinking to false for direct answers. When using a quantized variant, set the client model to its FP8 or INT4 checkpoint ID.

References

• Ling 3.0 collection
• Ling-3.0-tiny on Hugging Face
• Ling-3.0-tiny-fp8 on Hugging Face
• Ling-3.0-tiny-int4 on Hugging Face