# thinkingmachines/Inkling — vLLM serving recipe

> Natively multimodal 1T-parameter MoE from Thinking Machines Lab — text/image/audio in, text out, up to 1M context — with relative attention, short convolution, and shared expert sinks.

Configuration: hardware `h200` (8× H200) · strategy `single_node_tp` (Tensor Parallel) · variant `default` (NVFP4) · features `tool_calling`, `reasoning`

- Interactive page: https://recipes.vllm.ai/thinkingmachines/Inkling
- This configuration as JSON: https://recipes.vllm.ai/thinkingmachines/Inkling.json?hardware=h200

The commands below were rendered for exactly this configuration and take precedence over any example in the guide.

## Other configurations

Change the query string (the same parameters as the page URL) and fetch this URL again. Unknown or incompatible values fall back to the recipe default, so check the Configuration line.

- `hardware`: `h100`, `h200`, `b200`, `gb200`, `b300`, `gb300`
- `variant`: `default`, `mxfp4`, `bf16`, `fp8`, `redhat_fp8`, `redhat_nvfp4`
- `strategy`: `single_node_tp`, `single_node_tep`, `multi_node_tp`, `multi_node_tp_dp`, `multi_node_dep`, `multi_node_tep`, `pd_cluster` (multi-node strategies also need `nodes=<n>`)
- `features`: comma-separated list of `tool_calling`, `reasoning`, `spec_decoding` (default: `tool_calling`, `reasoning`)

## Install

Requires vLLM >= 0.26.0 (not released yet — nightly wheel).

vLLM 0.26.0 isn't released yet — nightly required. Nightly wheels are CUDA 13 (cu130) only; the host needs an r580+ NVIDIA driver.

```bash
uv venv
source .venv/bin/activate
uv pip install -U vllm --pre \
  --extra-index-url https://wheels.vllm.ai/nightly/cu130 \
  --extra-index-url https://download.pytorch.org/whl/cu130 \
  --index-strategy unsafe-best-match
```

Docker image: `vllm/vllm-openai:nightly`

Extra dependencies:

- Audio extras — only needed when serving the audio modality (optional)

```bash
uv pip install "vllm[audio]"
```

## Serve

```bash
export VLLM_USE_V2_MODEL_RUNNER=1
export FLASH_ATTENTION_CUTE_DSL_CACHE_ENABLED=1
vllm serve thinkingmachines/Inkling-NVFP4 \
  --trust-remote-code \
  --tokenizer-mode inkling \
  --disable-flashinfer-autotune \
  --tensor-parallel-size 8 \
  --enable-auto-tool-choice \
  --tool-call-parser inkling \
  --reasoning-parser inkling
```

Docker equivalent:

```bash
docker run --gpus all \
  --privileged --ipc=host -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -e VLLM_USE_V2_MODEL_RUNNER=1 \
  -e FLASH_ATTENTION_CUTE_DSL_CACHE_ENABLED=1 \
  vllm/vllm-openai:nightly thinkingmachines/Inkling-NVFP4 \
  --trust-remote-code \
  --tokenizer-mode inkling \
  --disable-flashinfer-autotune \
  --tensor-parallel-size 8 \
  --enable-auto-tool-choice \
  --tool-call-parser inkling \
  --reasoning-parser inkling
```

## Guide

> Background reference. Commands in it are examples and may target another configuration — use the Serve section above.

### Overview

TML Inkling is a natively multimodal 1T-parameter Mixture-of-Experts model from Thinking
Machines Lab. It accepts text, image, and audio inputs and generates text, with a context
length of up to 1M tokens. vLLM supports Inkling on Day 0 with full feature parity — LoRA,
speculative decoding (MTP), TP/DP/EP/PP parallelism, prefix caching, and disaggregated serving.

Architectural highlights:

- **Attention.** 66-layer decoder backbone — 11 full-attention layers and 55 sliding-window
  layers — all grouped-query attention with head size 128. Inkling replaces RoPE with
  **relative attention**: a learned relative-position term added to the pre-softmax logits.
- **Short convolution (sconv).** Each layer applies four window-4 sconv modules (on attention
  keys, values, output, and MoE output). vLLM manages the sconv cache as the KV cache of a
  virtual sliding-window layer, so prefix caching works seamlessly, and shards sconv across
  the channel dimension (reduce-scatter / all-gather) to avoid replicating cache and compute
  across TP ranks.
- **MoE with expert sinks.** 256 routed experts (top-6) plus 2 shared experts — 8 experts per
  token. The 2 shared experts act as *expert sinks*: they participate in routing-score
  computation but are excluded from the top-6 candidate pool.
- **MTP.** 8 chained MTP heads (single-layer full-attention transformers with a dense MLP,
  all BF16) enable up to 9 tokens per forward step for speculative decoding.

### Variants

- **NVFP4 (default):** Only the routed experts are quantized to NVFP4; shared experts and the
  qkvr linears remain BF16. This is a mixed-precision checkpoint. Blackwell uses its native
  FP4 tensor cores; on Hopper, the MoE kernel dequantizes the NVFP4 expert weights to BF16
  on the fly.
- **BF16:** Full BF16 weights (`thinkingmachines/Inkling`). Runs on both **Hopper**
  (H200&H20) and Blackwell, at the cost of substantially more VRAM.
- **MXFP4 (AMD):** `lightseekorg/Inkling-MXFP4` keeps the full 1M-token context
  at TP4 on MI355X and MI300X. MI355X runs native AITER MXFP4 MoE; MI300X uses
  Triton on-the-fly MXFP4 emulation.

### Prerequisites

- **Hardware:** NVIDIA Hopper (H200/H20) or Blackwell for NVFP4 and BF16;
  AMD MI300X, MI325X, or MI355X with TP4 for MXFP4.
- **vLLM:** Nightly wheels (Day-0 support just landed).
- **Parallelism:** 1T parameters requires multi-GPU. The featured configuration is 4x GB200
  (`--tensor-parallel-size 4`) for the NVFP4 variant. On Hopper, NVFP4 uses TP8 on H200;
  the BF16 variant needs multi-node.

### Client Usage

Launch the server (NVFP4 on 4x GB200):
```bash
export VLLM_USE_V2_MODEL_RUNNER=1
export FLASH_ATTENTION_CUTE_DSL_CACHE_ENABLED=1

vllm serve thinkingmachines/Inkling-NVFP4 \
  --tokenizer-mode inkling \
  --reasoning-parser inkling \
  --tool-call-parser inkling \
  --enable-auto-tool-choice \
  --tensor-parallel-size 4 \
  --disable-flashinfer-autotune \
  --trust-remote-code
```

For a multi-node BF16 deployment on Hopper (H200/H20), select the
**Multi-Node TP + Data Parallel** strategy with **Nodes = 2** in the command
builder above — it renders the per-node `vllm serve` commands (TP within
each node, DP across nodes, EP over all ranks).

On AMD, select the **MXFP4** variant. TP4 is validated with the native
1M context on both GPU families.

Equivalent MI355X launch:
```bash
export VLLM_USE_V2_MODEL_RUNNER=1
export VLLM_ROCM_USE_AITER=1
export VLLM_ROCM_USE_AITER_MOE=1
export INKLING_GFX950_GLUON=0

vllm serve lightseekorg/Inkling-MXFP4 \
  --tensor-parallel-size 4 \
  --tokenizer-mode inkling \
  --reasoning-parser inkling \
  --tool-call-parser inkling \
  --enable-auto-tool-choice \
  --moe-backend aiter \
  --block-size 128 \
  --trust-remote-code
```

Query the server with the OpenAI SDK, including an image input:
```python
from openai import OpenAI

client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1")

response = client.chat.completions.create(
    model="thinkingmachines/Inkling-NVFP4",
    messages=[{
        "role": "user",
        "content": [
            {"type": "text", "text": "Describe this image."},
            {"type": "image_url", "image_url": {"url": "https://example.com/photo.jpg"}},
        ],
    }],
)
print(response.choices[0].message.content)
```

### Performance

vLLM implements Inkling with a series of optimizations: sconv-aware TP sharding (each rank
stores only its channel shard of the sconv cache), low-latency fused reduce-scatter /
all-gather collectives built on FlashInfer's Lamport-protocol all-reduce, the FA4
sheared-bias attention kernel for relative attention, plus kernel fusion, PDL, and
multi-streaming. Enabling MTP speculative decoding adds further per-user throughput.

### References

- [TML Inkling (NVFP4)](https://huggingface.co/thinkingmachines/Inkling-NVFP4)
- [TML Inkling (BF16)](https://huggingface.co/thinkingmachines/Inkling)
- [Inkling MXFP4](https://huggingface.co/lightseekorg/Inkling-MXFP4)
- [vLLM integration PR](https://github.com/Inferact/mlj/pull/16)