inclusionAI/Ling-3.0-flash

Overview

inclusionAI/Ling-3.0-flash uses the BailingMoeV3ForCausalLM architecture with a hybrid MLA/KDA attention stack, 512 routed experts (8 active per token), one shared expert, and a native multi-token prediction head. The 42-layer base model has 124.4B total and 5.5B active parameters. The checkpoint also contains a 3.1B MTP layer, bringing the complete checkpoint to 127.5B parameters. A serialized block-FP8 checkpoint is available as inclusionAI/Ling-3.0-flash-fp8.

Prerequisites

• vLLM: a build containing native Bailing V3 support;
• Validated hardware: NVIDIA H20, H20-3e, and H200
• Precision: BF16 or serialized block FP8 weights with BF16 compute
• Context length: 262,144 tokens

Launching the Server

NCCL_DEBUG=WARN vllm serve inclusionAI/Ling-3.0-flash \
  --trust-remote-code \
  --dtype bfloat16 \
  --tensor-parallel-size 4 \
  --gpu-memory-utilization 0.9 \
  --enable-chunked-prefill \
  --compilation-config '{"cudagraph_mode":"FULL_AND_PIECEWISE"}' \
  --enable-prefix-caching \
  --enable-auto-tool-choice \
  --tool-call-parser ling3 \
  --reasoning-parser ling3 \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}'

For the serialized FP8 variant on 2x H200, use:

vllm serve inclusionAI/Ling-3.0-flash-fp8 \
  --trust-remote-code \
  --dtype bfloat16 \
  --tensor-parallel-size 2 \
  --gpu-memory-utilization 0.9 \
  --enable-chunked-prefill \
  --compilation-config '{"cudagraph_mode":"FULL_AND_PIECEWISE"}' \
  --enable-prefix-caching \
  --enable-auto-tool-choice \
  --tool-call-parser ling3 \
  --reasoning-parser ling3

For the validated TP4+EP4 alternative on 4x H200, use:

vllm serve inclusionAI/Ling-3.0-flash-fp8 \
  --trust-remote-code \
  --dtype bfloat16 \
  --tensor-parallel-size 4 \
  --enable-expert-parallel \
  --gpu-memory-utilization 0.9 \
  --enable-chunked-prefill \
  --compilation-config '{"cudagraph_mode":"FULL_AND_PIECEWISE"}' \
  --enable-prefix-caching \
  --enable-auto-tool-choice \
  --tool-call-parser ling3 \
  --reasoning-parser ling3

The native MTP head is available with either FP8 topology by adding:

--speculative-config '{"method":"mtp","num_speculative_tokens":3}'

Validation

Both the default FP8 TP2 path and the TP4+EP4 path were validated on NVIDIA H200. TP2 remains the recommended default because it uses fewer GPUs; TP4+EP4 is an alternative for deployments that prefer expert parallelism.

Thinking Mode

Thinking is selected per request through the chat template rather than by a server flag:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
response = client.chat.completions.create(
    model="inclusionAI/Ling-3.0-flash",
    messages=[{"role": "user", "content": "Solve the problem step by step."}],
    temperature=0.0,
    max_tokens=200000,
    extra_body={"chat_template_kwargs": {"enable_thinking": True}},
)
print(response.choices[0].message.reasoning_content)
print(response.choices[0].message.content)

When serving the FP8 variant, set model="inclusionAI/Ling-3.0-flash-fp8" in the client request.

References

• Ling-3.0-flash on Hugging Face
• Ling-3.0-flash-fp8 on Hugging Face