# thinkingmachines/Inkling-Small — vLLM serving recipe

> Natively multimodal 276B-parameter MoE from Thinking Machines Lab — 12B active parameters, text/image/audio in, text out, and up to 1M context.

Configuration: hardware `h200` (8× H200) · strategy `single_node_tp` (Tensor Parallel) · variant `default` (NVFP4) · features `tool_calling`, `reasoning`

- Interactive page: https://recipes.vllm.ai/thinkingmachines/Inkling-Small
- This configuration as JSON: https://recipes.vllm.ai/thinkingmachines/Inkling-Small.json?hardware=h200

The commands below were rendered for exactly this configuration and take precedence over any example in the guide.

## Other configurations

Change the query string (the same parameters as the page URL) and fetch this URL again. Unknown or incompatible values fall back to the recipe default, so check the Configuration line.

- `hardware`: `h100`, `h200`, `b200`, `gb200`, `b300`, `gb300`
- `variant`: `default`, `mxfp4`, `bf16`, `fp8`
- `strategy`: `single_node_tp`, `single_node_tep`, `multi_node_tp`, `multi_node_tp_dp`, `multi_node_dep`, `multi_node_tep`, `pd_cluster` (multi-node strategies also need `nodes=<n>`)
- `features`: comma-separated list of `tool_calling`, `reasoning`, `spec_decoding` (default: `tool_calling`, `reasoning`)

## Install

Requires vLLM >= 0.26.0 (not released yet — nightly wheel).

vLLM 0.26.0 isn't released yet — nightly required. Nightly wheels are CUDA 13 (cu130) only; the host needs an r580+ NVIDIA driver.

```bash
uv venv
source .venv/bin/activate
uv pip install -U vllm --pre \
  --extra-index-url https://wheels.vllm.ai/nightly/cu130 \
  --extra-index-url https://download.pytorch.org/whl/cu130 \
  --index-strategy unsafe-best-match
```

Docker image: `vllm/vllm-openai:nightly`

Extra dependencies:

- Audio extras — only needed when serving the audio modality (optional)

```bash
uv pip install "vllm[audio]"
```

## Serve

```bash
export VLLM_USE_V2_MODEL_RUNNER=1
export FLASH_ATTENTION_CUTE_DSL_CACHE_ENABLED=1
vllm serve thinkingmachines/Inkling-Small-NVFP4 \
  --trust-remote-code \
  --tokenizer-mode inkling \
  --disable-flashinfer-autotune \
  --tensor-parallel-size 2 \
  --enable-auto-tool-choice \
  --tool-call-parser inkling \
  --reasoning-parser inkling
```

Docker equivalent:

```bash
docker run --gpus all \
  --privileged --ipc=host -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -e VLLM_USE_V2_MODEL_RUNNER=1 \
  -e FLASH_ATTENTION_CUTE_DSL_CACHE_ENABLED=1 \
  vllm/vllm-openai:nightly thinkingmachines/Inkling-Small-NVFP4 \
  --trust-remote-code \
  --tokenizer-mode inkling \
  --disable-flashinfer-autotune \
  --tensor-parallel-size 2 \
  --enable-auto-tool-choice \
  --tool-call-parser inkling \
  --reasoning-parser inkling
```

## Guide

> Background reference. Commands in it are examples and may target another configuration — use the Serve section above.

### Overview

TML Inkling-Small is a natively multimodal 276B-parameter Mixture-of-Experts model
from Thinking Machines Lab. It activates 12B parameters per token and uses the same
Inkling architecture and vLLM serving configuration as its larger 975B-parameter
sibling, while substantially reducing the deployment footprint.

Inkling-Small accepts text, image, and audio inputs and generates text. It supports
Inkling's relative-attention and short-convolution serving path, tool and reasoning
parsers, and MTP speculative decoding.

### Variants

- **NVFP4 (default):** Requires at least 180 GB aggregated VRAM. Run W4A4 with TP1
  on B300/GB300 or TP2 on B200/GB200. On Hopper, run W4A16 with TP4 on H100
  or TP2 on H200; the MoE kernel dequantizes the NVFP4 weights to BF16 on the fly.
- **BF16:** Requires at least 600 GB aggregated VRAM. Run TP4 on B300/GB200/GB300
  or AMD MI300X/MI325X/MI355X, and TP8 on B200/H200. MI355X TP4 is validated;
  direct MI300X/MI325X BF16 smoke is pending.
- **MXFP4 (AMD):** `EmbeddedLLM/Inkling-Small-MXFP4` keeps the full 1M-token
  context at TP1 on MI355X or TP2 on MI300X. MI355X runs native AITER MXFP4
  MoE; MI300X uses the supported Triton on-the-fly MXFP4 emulation path.

### Prerequisites

- **Hardware:** B300/GB300 TP1, B200/GB200/H200 TP2, or H100 TP4 for NVFP4;
  B300/GB200/GB300/MI300X/MI325X/MI355X TP4 or B200/H200 TP8 for BF16;
  MI355X TP1 or MI300X/MI325X TP2 for AMD MXFP4.
- **vLLM:** A nightly build with Inkling support.
- **Audio:** Install the optional `vllm[audio]` extra only when serving audio inputs.

### Client Usage

Launch the NVFP4 checkpoint in W4A4 mode on B300:
```bash
export VLLM_USE_V2_MODEL_RUNNER=1
export FLASH_ATTENTION_CUTE_DSL_CACHE_ENABLED=1

vllm serve thinkingmachines/Inkling-Small-NVFP4 \
  --tokenizer-mode inkling \
  --reasoning-parser inkling \
  --tool-call-parser inkling \
  --enable-auto-tool-choice \
  --tensor-parallel-size 1 \
  --disable-flashinfer-autotune \
  --trust-remote-code
```

Select **GB300** for W4A4 with TP1, or **B200**/**GB200** for W4A4 with TP2.
On H100, the builder emits TP4; on H200, it emits TP2. vLLM runs W4A16 with on-the-fly
dequantization. A reported 4×H100 deployment served a plain chat request, but
JSON-schema structured output returned HTTP 500; see
[vLLM #51693](https://github.com/vllm-project/vllm/issues/51693). H100 is not
marked verified for the full recipe.

On AMD, select the **MXFP4** variant. To fit 1M context, MI300X TP2 and MI355X TP1.
MI355X uses `--moe-backend aiter`; MI300X uses `--moe-backend auto`
because native AITER W4A4 is gated to gfx950, while gfx942 falls back to
`OCP_MXQuantizationEmulationTritonExperts`. Inkling relative attention remains
on its model-specific Triton path (`INKLING_GFX950_GLUON=0`), which was faster
than the optional gfx950 Gluon path in the serving benchmark.

Equivalent MI355X launch:
```bash
export VLLM_USE_V2_MODEL_RUNNER=1
export VLLM_ROCM_USE_AITER=1
export VLLM_ROCM_USE_AITER_MOE=1
export INKLING_GFX950_GLUON=0

vllm serve EmbeddedLLM/Inkling-Small-MXFP4 \
  --tensor-parallel-size 1 \
  --tokenizer-mode inkling \
  --reasoning-parser inkling \
  --tool-call-parser inkling \
  --enable-auto-tool-choice \
  --moe-backend aiter \
  --block-size 128 \
  --trust-remote-code
```

Query the server with the OpenAI SDK:
```python
from openai import OpenAI

client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1")

response = client.chat.completions.create(
    model="thinkingmachines/Inkling-Small-NVFP4",
    messages=[{
        "role": "user",
        "content": [
            {"type": "text", "text": "Describe this image."},
            {"type": "image_url", "image_url": {"url": "https://example.com/photo.jpg"}},
        ],
    }],
)
print(response.choices[0].message.content)
```

### References

- [TML Inkling announcement](https://thinkingmachines.ai/news/introducing-inkling/#inkling-small)
- [Inkling-Small MXFP4 for AMD](https://huggingface.co/EmbeddedLLM/Inkling-Small-MXFP4)
- [TML Inkling recipe](https://docs.vllm.ai/projects/recipes/en/latest/thinkingmachines/Inkling.html)