inclusionAI/Ling-3.0-flash

Overview

inclusionAI/Ling-3.0-flash uses the BailingMoeV3ForCausalLM architecture with a hybrid MLA/KDA attention stack, 512 routed experts (8 active per token), one shared expert, and a native multi-token prediction head. The 42-layer base model has 124.4B total and 5.5B active parameters. The checkpoint also contains a 3.1B MTP layer, bringing the complete checkpoint to 127.5B parameters.

Prerequisites

• vLLM: a build containing native Bailing V3 support
• Validated hardware: 4x NVIDIA H20-3e
• Precision: BF16
• Context length: 131,072 tokens

Launching the Server

NCCL_DEBUG=WARN vllm serve inclusionAI/Ling-3.0-flash \
  --trust-remote-code \
  --dtype bfloat16 \
  --tensor-parallel-size 4 \
  --gpu-memory-utilization 0.9 \
  --enable-chunked-prefill \
  --compilation-config '{"cudagraph_mode":"FULL_AND_PIECEWISE"}' \
  --enable-prefix-caching \
  --enable-auto-tool-choice \
  --tool-call-parser ling3 \
  --reasoning-parser ling3 \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}'

Thinking Mode

Thinking is selected per request through the chat template rather than by a server flag:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
response = client.chat.completions.create(
    model="inclusionAI/Ling-3.0-flash",
    messages=[{"role": "user", "content": "Solve the problem step by step."}],
    temperature=0.0,
    max_tokens=128000,
    extra_body={"chat_template_kwargs": {"enable_thinking": True}},
)
print(response.choices[0].message.reasoning_content)
print(response.choices[0].message.content)