# inclusionAI/Ling-3.0-flash-VL — vLLM serving recipe

> Ling-3.0-flash vision-language MoE model with BF16, FP8, FP4, and INT4 checkpoints

Configuration: hardware `h200` (8× H200) · strategy `single_node_tp` (Tensor Parallel) · variant `default` (BF16) · features `tool_calling`, `reasoning`

- Interactive page: https://recipes.vllm.ai/inclusionAI/Ling-3.0-flash-VL
- This configuration as JSON: https://recipes.vllm.ai/inclusionAI/Ling-3.0-flash-VL.json?hardware=h200

The commands below were rendered for exactly this configuration and take precedence over any example in the guide.

## Other configurations

Change the query string (the same parameters as the page URL) and fetch this URL again. Unknown or incompatible values fall back to the recipe default, so check the Configuration line.

- `hardware`: `h100`, `h200`, `b200`, `gb200`, `b300`, `gb300`, `mi300x`, `mi325x`, `mi350x`, `mi355x`
- `variant`: `default`, `fp8`, `fp4`, `int4`
- `strategy`: `single_node_tp` (multi-node strategies also need `nodes=<n>`)
- `features`: comma-separated list of `tool_calling`, `reasoning` (default: `tool_calling`, `reasoning`)

## Install

Requires vLLM >= nightly (not released yet — nightly wheel).

vLLM nightly isn't released yet — nightly required. Nightly wheels are CUDA 13 (cu130) only; the host needs an r580+ NVIDIA driver.

```bash
uv venv
source .venv/bin/activate
uv pip install -U vllm --pre \
  --extra-index-url https://wheels.vllm.ai/nightly/cu130 \
  --extra-index-url https://download.pytorch.org/whl/cu130 \
  --index-strategy unsafe-best-match
```

Docker image: `vllm/vllm-openai:nightly`

## Serve

```bash
vllm serve inclusionAI/Ling-3.0-flash-VL \
  --trust-remote-code \
  --dtype bfloat16 \
  --gpu-memory-utilization 0.9 \
  --enable-chunked-prefill \
  --compilation-config '{"cudagraph_mode":"FULL_AND_PIECEWISE"}' \
  --enable-prefix-caching \
  --tensor-parallel-size 4 \
  --enable-auto-tool-choice \
  --tool-call-parser ling3 \
  --reasoning-parser ling3
```

Docker equivalent:

```bash
docker run --gpus all \
  --privileged --ipc=host -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai:nightly inclusionAI/Ling-3.0-flash-VL \
  --trust-remote-code \
  --dtype bfloat16 \
  --gpu-memory-utilization 0.9 \
  --enable-chunked-prefill \
  --compilation-config '{"cudagraph_mode":"FULL_AND_PIECEWISE"}' \
  --enable-prefix-caching \
  --tensor-parallel-size 4 \
  --enable-auto-tool-choice \
  --tool-call-parser ling3 \
  --reasoning-parser ling3
```

## Guide

> Background reference. Commands in it are examples and may target another configuration — use the Serve section above.

### Overview

`inclusionAI/Ling-3.0-flash-VL` uses the
`BailingMoeV3VLForConditionalGeneration` architecture. It combines the
Ling-3.0-flash hybrid MLA/KDA language backbone with a vision encoder and
projector for image understanding. The language backbone has 512 routed
experts (8 active per token) and one shared expert.

BF16, FP8, mixed block-FP8/MXFP4, and INT4 checkpoints are available. The
vision tower and projector remain BF16 in the mixed FP4 checkpoint.

### Prerequisites

- **vLLM:** an official nightly build containing Ling-3.0-flash-VL support
- **Precision:** BF16, FP8, FP4, or INT4 weights with BF16 compute
- **Context length:** 131,072 tokens

### Launching the Server

```bash
NCCL_DEBUG=WARN vllm serve inclusionAI/Ling-3.0-flash-VL \
  --trust-remote-code \
  --dtype bfloat16 \
  --tensor-parallel-size 4 \
  --gpu-memory-utilization 0.9 \
  --enable-chunked-prefill \
  --compilation-config '{"cudagraph_mode":"FULL_AND_PIECEWISE"}' \
  --enable-prefix-caching \
  --enable-auto-tool-choice \
  --tool-call-parser ling3 \
  --reasoning-parser ling3
```

For a quantized variant, use the same flags with the corresponding model
ID and tensor parallel size:

| Variant | Model ID | TP |
|---|---|---:|
| FP8 | `inclusionAI/Ling-3.0-flash-VL-fp8` | 2 |
| FP4 | `inclusionAI/Ling-3.0-flash-VL-fp4` | 1 |
| INT4 | `inclusionAI/Ling-3.0-flash-VL-int4` | 1 |

### Image Requests

```python
import base64
from pathlib import Path
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
image = base64.b64encode(Path("image.png").read_bytes()).decode("ascii")
response = client.chat.completions.create(
    model="inclusionAI/Ling-3.0-flash-VL",
    messages=[{
        "role": "user",
        "content": [
            {"type": "image_url", "image_url": {"url": f"data:image/png;base64,{image}"}},
            {"type": "text", "text": "Describe the image."},
        ],
    }],
    temperature=0.0,
    max_tokens=1024,
    extra_body={"chat_template_kwargs": {"enable_thinking": False}},
)
print(response.choices[0].message.content)
```

For multiple images, add more `image_url` entries to `content`. For
text-only requests, use ordinary string content. When serving a quantized
variant, use its checkpoint ID in the client request.

### Thinking Mode

As with Ling-3.0-flash, thinking is selected per request through
`chat_template_kwargs`. Set `enable_thinking` to `True` to enable it;
with `--reasoning-parser ling3`, the parsed trace is in `message.reasoning`
and the final answer is in `message.content`.

### References

- [Ling-3.0-flash-VL](https://huggingface.co/inclusionAI/Ling-3.0-flash-VL)
- [Ling-3.0-flash-VL-fp8](https://huggingface.co/inclusionAI/Ling-3.0-flash-VL-fp8)
- [Ling-3.0-flash-VL-fp4](https://huggingface.co/inclusionAI/Ling-3.0-flash-VL-fp4)
- [Ling-3.0-flash-VL-int4](https://huggingface.co/inclusionAI/Ling-3.0-flash-VL-int4)