# thinkingmachines/Inkling-2-Mini — vLLM serving recipe

> Looped multimodal (text and image in, text out) MoE from Thinking Machines Lab with 26B total and 3.5B active parameters and up to 1M context: its 12 middle layers run three passes per token, each with its own KV cache and gated messages from the previous pass. Bundles a DSpark speculator.

Configuration: hardware `gb200` (4× GB200 NVL4) · strategy `single_node_tp` (Tensor Parallel) · variant `default` (NVFP4) · features `reasoning`, `spec_decoding`

- Interactive page: https://recipes.vllm.ai/thinkingmachines/Inkling-2-Mini
- This configuration as JSON: https://recipes.vllm.ai/thinkingmachines/Inkling-2-Mini.json?hardware=gb200

The commands below were rendered for exactly this configuration and take precedence over any example in the guide.

## Other configurations

Change the query string (the same parameters as the page URL) and fetch this URL again. Unknown or incompatible values fall back to the recipe default, so check the Configuration line.

- `hardware`: `b200`, `gb200`, `b300`, `gb300`, `dgx_spark_gb10`
- `variant`: `default`
- `strategy`: `single_node_tp` (multi-node strategies also need `nodes=<n>`)
- `features`: comma-separated list of `reasoning`, `tool_calling`, `spec_decoding` (default: `reasoning`, `spec_decoding`). Use `features=none` to turn them all off.

## Install

Requires vLLM >= 0.35.0 (not released yet — nightly wheel).

Docker image: `vllm/vllm-openai:inkling-2-mini`

## Serve

```bash
vllm serve thinkingmachines/Inkling-2-Mini \
  --tensor-parallel-size 1 \
  --max-model-len 262144 \
  --reasoning-parser inkling \
  --speculative-config '{"method":"dspark","num_speculative_tokens":7,"draft_sample_method":"probabilistic"}'
```

Docker equivalent:

```bash
docker run --gpus all \
  --privileged --ipc=host -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai:inkling-2-mini thinkingmachines/Inkling-2-Mini \
  --tensor-parallel-size 1 \
  --max-model-len 262144 \
  --reasoning-parser inkling \
  --speculative-config '{"method":"dspark","num_speculative_tokens":7,"draft_sample_method":"probabilistic"}'
```

## Guide

> Background reference. Commands in it are examples and may target another configuration — use the Serve section above.

### Overview

Inkling 2 Mini is a looped Mixture-of-Experts model from Thinking Machines (text and image in,
text out) with 26B total and 3.5B active parameters and up to 1M tokens of context. It ships
with a DSpark speculator in the same checkpoint.

Its 24 layers split into a prelude (layers 0-5), a loop core (layers 6-17) and a coda
(layers 18-23). The core runs three passes with shared weights, so every token makes 48
layer visits:

- **Every pass starts from the prelude's output.** From the second pass on, each core
  sublayer input (attention query, key and value inputs, and the MLP input) reads a message
  from the previous pass's two attention-residual blocks and merges it through a gated
  rank-64 controller.
- **Each pass keeps its own memory:** every core layer has a KV cache and a conv-state cache
  per pass (48 of each in total). vLLM manages them like any other cache layers, so prefix
  caching and CUDA graphs work as usual.

Each layer uses Inkling 2 Small's components at hidden size 2,304: block attention residuals,
relative-bias attention with a gate and sink (20 sliding-window and 4 full-attention layers,
24 query and 8 KV heads), four short convolutions, and 128 routed experts (top-8). The image
encoder is a 23-layer ViT.

### Runtime

Use the `vllm/vllm-openai:inkling-2-mini` image until Inkling 2 Mini support is in a vLLM release.

### Configurations

Inkling 2 Mini runs on NVIDIA Blackwell (B200, GB200, B300, GB300), Hopper (H100, H200) and
DGX Spark, and is verified on GB200, B300, GB300, H100, H200 and DGX Spark. The configurations
below were verified on GB200 with DSpark and the reasoning parser enabled:

- **One GPU, TP1:** contexts up to 262,144 tokens. This is the most efficient way to serve
  the model; TP4 gives lower output throughput per GPU.
- **TP4 (`--tensor-parallel-size 4 --max-model-len 1048576`):** the full 1,048,576-token
  context.

On one GPU, the KV cache bounds the number of concurrent long sessions: with 32K, 64K and
128K-token sessions, about 45, 32 and 20 sessions run at 100 tokens/s per user.

Tool calling uses the `inkling` tool-call parser (opt-in above).

### Client Usage

```python
from openai import OpenAI

client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1")

response = client.chat.completions.create(
    model="thinkingmachines/Inkling-2-Mini",
    messages=[{
        "role": "user",
        "content": [
            {"type": "text", "text": "Describe this image."},
            {"type": "image_url", "image_url": {"url": "https://example.com/photo.jpg"}},
        ],
    }],
)
print(response.choices[0].message.content)
```

### Benchmarking

Benchmark through the chat endpoint so the server applies the chat template and the
reasoning parser. DSpark acceptance depends on the prompts and the sampling temperature:
random-token prompts accept almost nothing, so benchmark on realistic text such as SPEED-Bench.

```bash
vllm bench serve \
  --backend openai-chat --endpoint /v1/chat/completions \
  --model thinkingmachines/Inkling-2-Mini \
  --dataset-name speed_bench --speed-bench-dataset-subset throughput_1k \
  --speed-bench-output-len 1024 --max-concurrency 16 --num-prompts 64
```

### References

- [Inkling 2 Mini announcement](https://thinkingmachines.ai/news/inkling-2-mini/)
- [Inkling 2 Mini on Hugging Face](https://huggingface.co/thinkingmachines/Inkling-2-Mini)
- [vLLM blog: Inkling 2 Mini on vLLM](https://vllm.ai/blog/2026-11-20-inkling2-mini)
- [Inkling 2 Small recipe](https://recipes.vllm.ai/thinkingmachines/Inkling-2-Small)
- [DeepSeek DSpark (DeepSpec)](https://github.com/deepseek-ai/DeepSpec)
