vLLM/Recipes
Thinking Machines Lab

thinkingmachines/Inkling-Small

Natively multimodal 276B-parameter MoE from Thinking Machines Lab — 12B active parameters, text/image/audio in, text out, and up to 1M context.

276B total / 12B active Inkling architecture for lower-cost, lower-latency deployment

View on HuggingFace
moe276B / 12B1,048,576 ctxvLLM 0.26.0+multimodal
Guide

Overview

TML Inkling-Small is a natively multimodal 276B-parameter Mixture-of-Experts model from Thinking Machines Lab. It activates 12B parameters per token and uses the same Inkling architecture and vLLM serving configuration as its larger 975B-parameter sibling, while substantially reducing the deployment footprint.

Inkling-Small accepts text, image, and audio inputs and generates text. It supports Inkling's relative-attention and short-convolution serving path, tool and reasoning parsers, and MTP speculative decoding.

Variants

  • NVFP4 (default): Requires at least 180 GB aggregated VRAM. Run W4A4 with TP1 on B300/GB300 or TP2 on B200/GB200. On H200, run W4A16 with TP2; the MoE kernel dequantizes the NVFP4 weights to BF16 on the fly.
  • BF16: Requires at least 600 GB aggregated VRAM. Run TP4 on B300/GB200/GB300 or TP8 on B200/H200.

Prerequisites

  • Hardware: B300/GB300 TP1 or B200/GB200/H200 TP2 for NVFP4; B300/GB200/GB300 TP4 or B200/H200 TP8 for BF16.
  • vLLM: A nightly build with Inkling support.
  • Audio: Install the optional vllm[audio] extra only when serving audio inputs.

Client Usage

Launch the NVFP4 checkpoint in W4A4 mode on B300:

export VLLM_USE_V2_MODEL_RUNNER=1
export FLASH_ATTENTION_CUTE_DSL_CACHE_ENABLED=1

vllm serve thinkingmachines/Inkling-Small-NVFP4 \
  --tokenizer-mode inkling \
  --reasoning-parser inkling \
  --tool-call-parser inkling \
  --enable-auto-tool-choice \
  --tensor-parallel-size 1 \
  --kernel-config.enable_flashinfer_autotune=False \
  --trust-remote-code

Select GB300 for W4A4 with TP1, or B200/GB200 for W4A4 with TP2. On H200, the builder emits TP2 and vLLM runs W4A16 with on-the-fly dequantization.

For BF16, select the BF16 variant. The command builder emits TP4 on B300/GB200/GB300 and TP8 on B200/H200.

Query the server with the OpenAI SDK:

from openai import OpenAI

client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1")

response = client.chat.completions.create(
    model="thinkingmachines/Inkling-Small-NVFP4",
    messages=[{
        "role": "user",
        "content": [
            {"type": "text", "text": "Describe this image."},
            {"type": "image_url", "image_url": {"url": "https://example.com/photo.jpg"}},
        ],
    }],
)
print(response.choices[0].message.content)

References