vLLM/Recipes
Search
Thinking Machines Lab

thinkingmachines/Inkling-2-Small

Multimodal (text and image in, text out) MoE from Thinking Machines Lab with 594B total and 17B active parameters, up to 1M context, attention residuals, relative-bias attention, short convolutions, and a bundled DSpark speculator.

~1,100 output tok/s/GPU at 100 tok/s/user (1K in / 1K out) with DSpark on 2× 4 GB200 (TP8)

View on HuggingFace
moe594B / 17B1,048,576 ctxvLLM nightly+multimodal
Default configuration: 8× B200 · Tensor Parallel · NVFP4
vllm serve thinkingmachines/Inkling-2-Small \
  --tensor-parallel-size 4 \
  --max-model-len 262144 \
  --max-num-seqs 256 \
  --reasoning-parser inkling \
  --speculative-config '{"method":"dspark","num_speculative_tokens":7,"draft_sample_method":"probabilistic"}'

The interactive builder loads with JavaScript. Append .md to this page's URL, query string included, for any configuration's command as Markdown — e.g. /thinkingmachines/Inkling-2-Small.md.

Overview

Inkling 2 Small is the next generation of Thinking Machines' Inkling and Inkling-Small: a multimodal Mixture-of-Experts model (text and image in, text out) with 594B total and 17B active parameters and up to 1M tokens of context. It ships with a DSpark speculator in the same checkpoint.

What changed from Inkling:

  • 48 layers (40 sliding-window + 8 full attention, the same 5:1 pattern) with relative-bias attention instead of RoPE and four short convolutions per layer, as in Inkling.
  • Block attention residuals replace the residual stream: before each sublayer, a learned query mixes the token embedding and the outputs of earlier 6-layer blocks.
  • Gated attention output with a per-token sink.
  • 384 routed experts, top-8, no shared experts. Experts are NVFP4 in layers 0-39 and MXFP8 in layers 40-47.
  • Image encoder: a 23-layer ViT. No audio input.
  • DSpark speculative decoding (7 tokens per step) instead of 8 chained MTP heads. Its recurrent Markov head is the RNN variant from the DSpark paper.

Inkling 2 Small runs its MoE expert-parallel across the tensor-parallel group, so TP4 means TP4 attention with EP4 experts, and TP8 means TP8 attention with EP8 experts.

Runtime

Use the vllm/vllm-openai:inkling-2-small image until Inkling 2 Small support is in a vLLM release.

Configurations

Verified on GB200 with DSpark and the reasoning parser enabled:

  • One tray, TP4: contexts up to 262,144 tokens.
  • Two trays, TP8: the full 1,048,576-token context.

LoRA adapters are supported on the attention projections and the routed experts. Expert adapters need BF16 expert weights; with the NVFP4/MXFP8 experts of this checkpoint, use attention-only adapters.

Tool calling uses the inkling tool-call parser (opt-in above). Its parser is unit-tested; no end-to-end tool-calling benchmark has been run yet.

Client Usage

from openai import OpenAI

client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1")

response = client.chat.completions.create(
    model="thinkingmachines/Inkling-2-Small",
    messages=[{
        "role": "user",
        "content": [
            {"type": "text", "text": "Describe this image."},
            {"type": "image_url", "image_url": {"url": "https://example.com/photo.jpg"}},
        ],
    }],
)
print(response.choices[0].message.content)

Benchmarking

Benchmark through the chat endpoint so the server applies the chat template and the reasoning parser. DSpark acceptance depends on the prompts and the sampling temperature: random-token prompts accept almost nothing, so benchmark on realistic text such as SPEED-Bench. The DSpark arguments above sample drafts probabilistically; with vLLM's default greedy drafting, acceptance at temperature 1.0 is about 15-25% lower.

vllm bench serve \
  --backend openai-chat --endpoint /v1/chat/completions \
  --model thinkingmachines/Inkling-2-Small \
  --dataset-name speed_bench --speed-bench-dataset-subset throughput_1k \
  --speed-bench-output-len 1024 --max-concurrency 16 --num-prompts 64

References