thinkingmachines/Inkling-2-Mini
Looped multimodal (text and image in, text out) MoE from Thinking Machines Lab with 26B total and 3.5B active parameters and up to 1M context: its 12 middle layers run three passes per token, each with its own KV cache and gated messages from the previous pass. Bundles a DSpark speculator.
~4,400 output tok/s/GPU at 100 tok/s/user (32K-context agentic sessions) with DSpark on 1 GB200 (TP1)
vllm serve thinkingmachines/Inkling-2-Mini \
--tensor-parallel-size 1 \
--max-model-len 262144 \
--reasoning-parser inkling \
--speculative-config '{"method":"dspark","num_speculative_tokens":7,"draft_sample_method":"probabilistic"}'The interactive builder loads with JavaScript. Append .md to this page's URL, query string included, for any configuration's command as Markdown — e.g. /thinkingmachines/Inkling-2-Mini.md.
Overview
Inkling 2 Mini is a looped Mixture-of-Experts model from Thinking Machines (text and image in, text out) with 26B total and 3.5B active parameters and up to 1M tokens of context. It ships with a DSpark speculator in the same checkpoint.
Its 24 layers split into a prelude (layers 0-5), a loop core (layers 6-17) and a coda (layers 18-23). The core runs three passes with shared weights, so every token makes 48 layer visits:
- Every pass starts from the prelude's output. From the second pass on, each core sublayer input (attention query, key and value inputs, and the MLP input) reads a message from the previous pass's two attention-residual blocks and merges it through a gated rank-64 controller.
- Each pass keeps its own memory: every core layer has a KV cache and a conv-state cache per pass (48 of each in total). vLLM manages them like any other cache layers, so prefix caching and CUDA graphs work as usual.
Each layer uses Inkling 2 Small's components at hidden size 2,304: block attention residuals, relative-bias attention with a gate and sink (20 sliding-window and 4 full-attention layers, 24 query and 8 KV heads), four short convolutions, and 128 routed experts (top-8). The image encoder is a 23-layer ViT.
Runtime
Use the vllm/vllm-openai:inkling-2-mini image until Inkling 2 Mini support is in a vLLM release.
Configurations
Inkling 2 Mini runs on NVIDIA Blackwell (B200, GB200, B300, GB300), Hopper (H100, H200) and DGX Spark, and is verified on GB200, B300, GB300, H100, H200 and DGX Spark. The configurations below were verified on GB200 with DSpark and the reasoning parser enabled:
- One GPU, TP1: contexts up to 262,144 tokens. This is the most efficient way to serve the model; TP4 gives lower output throughput per GPU.
- TP4 (
--tensor-parallel-size 4 --max-model-len 1048576): the full 1,048,576-token context.
On one GPU, the KV cache bounds the number of concurrent long sessions: with 32K, 64K and 128K-token sessions, about 45, 32 and 20 sessions run at 100 tokens/s per user.
Tool calling uses the inkling tool-call parser (opt-in above).
Client Usage
from openai import OpenAI
client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1")
response = client.chat.completions.create(
model="thinkingmachines/Inkling-2-Mini",
messages=[{
"role": "user",
"content": [
{"type": "text", "text": "Describe this image."},
{"type": "image_url", "image_url": {"url": "https://example.com/photo.jpg"}},
],
}],
)
print(response.choices[0].message.content)
Benchmarking
Benchmark through the chat endpoint so the server applies the chat template and the reasoning parser. DSpark acceptance depends on the prompts and the sampling temperature: random-token prompts accept almost nothing, so benchmark on realistic text such as SPEED-Bench.
vllm bench serve \
--backend openai-chat --endpoint /v1/chat/completions \
--model thinkingmachines/Inkling-2-Mini \
--dataset-name speed_bench --speed-bench-dataset-subset throughput_1k \
--speed-bench-output-len 1024 --max-concurrency 16 --num-prompts 64