# thinkingmachines/Inkling-2-Small — vLLM serving recipe

> Multimodal (text and image in, text out) MoE from Thinking Machines Lab with 594B total and 17B active parameters, up to 1M context, attention residuals, relative-bias attention, short convolutions, and a bundled DSpark speculator.

Configuration: hardware `b200` (8× B200) · strategy `single_node_tp` (Tensor Parallel) · variant `default` (NVFP4) · features `reasoning`, `spec_decoding`

- Interactive page: https://recipes.vllm.ai/thinkingmachines/Inkling-2-Small
- This configuration as JSON: https://recipes.vllm.ai/thinkingmachines/Inkling-2-Small.json?hardware=b200

The commands below were rendered for exactly this configuration and take precedence over any example in the guide.

## Other configurations

Change the query string (the same parameters as the page URL) and fetch this URL again. Unknown or incompatible values fall back to the recipe default, so check the Configuration line.

- `hardware`: `b200`, `gb200`, `b300`, `gb300`
- `variant`: `default`
- `strategy`: `single_node_tp`, `multi_node_tp` (multi-node strategies also need `nodes=<n>`)
- `features`: comma-separated list of `reasoning`, `tool_calling`, `spec_decoding` (default: `reasoning`, `spec_decoding`). Use `features=none` to turn them all off.

## Install

Requires vLLM >= nightly (not released yet — nightly wheel).

Docker image: `vllm/vllm-openai:inkling-2-small`

## Serve

```bash
vllm serve thinkingmachines/Inkling-2-Small \
  --tensor-parallel-size 4 \
  --max-model-len 262144 \
  --max-num-seqs 256 \
  --reasoning-parser inkling \
  --speculative-config '{"method":"dspark","num_speculative_tokens":7,"draft_sample_method":"probabilistic"}'
```

Docker equivalent:

```bash
docker run --gpus all \
  --privileged --ipc=host -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai:inkling-2-small thinkingmachines/Inkling-2-Small \
  --tensor-parallel-size 4 \
  --max-model-len 262144 \
  --max-num-seqs 256 \
  --reasoning-parser inkling \
  --speculative-config '{"method":"dspark","num_speculative_tokens":7,"draft_sample_method":"probabilistic"}'
```

## Known issues

### To serve at least one request with the model's max seq len (1048576), (64.47 GiB KV cache is needed, which is larger than the available KV cache memory

A 1M-token request does not fit on one 4-GPU tray. Keep `--max-model-len 262144` at TP4, or use the multi-node TP strategy with 2 nodes (TP8) for the full 1M context.

## Guide

> Background reference. Commands in it are examples and may target another configuration — use the Serve section above.

### Overview

Inkling 2 Small is the next generation of Thinking Machines' [Inkling](https://huggingface.co/thinkingmachines/Inkling)
and Inkling-Small: a multimodal Mixture-of-Experts model (text and image in, text out) with
594B total and 17B active parameters and up to 1M tokens of context. It ships with a DSpark
speculator in the same checkpoint.

What changed from Inkling:

- **48 layers** (40 sliding-window + 8 full attention, the same 5:1 pattern) with
  relative-bias attention instead of RoPE and four short convolutions per layer, as in Inkling.
- **Block attention residuals** replace the residual stream: before each sublayer, a learned
  query mixes the token embedding and the outputs of earlier 6-layer blocks.
- **Gated attention output** with a per-token sink.
- **384 routed experts, top-8, no shared experts.** Experts are NVFP4 in layers 0-39 and MXFP8
  in layers 40-47.
- **Image encoder:** a 23-layer ViT. No audio input.
- **DSpark** speculative decoding (7 tokens per step) instead of 8 chained MTP heads. Its
  recurrent Markov head is the RNN variant from the DSpark paper.

Inkling 2 Small runs its MoE expert-parallel across the tensor-parallel group, so TP4 means
TP4 attention with EP4 experts, and TP8 means TP8 attention with EP8 experts.

### Runtime

Use the `vllm/vllm-openai:inkling-2-small` image until Inkling 2 Small support is in a vLLM release.

### Configurations

Verified on GB200 with DSpark and the reasoning parser enabled:

- **One tray, TP4:** contexts up to 262,144 tokens.
- **Two trays, TP8:** the full 1,048,576-token context.

LoRA adapters are supported on the attention projections and the routed experts. Expert
adapters need BF16 expert weights; with the NVFP4/MXFP8 experts of this checkpoint, use
attention-only adapters.

Tool calling uses the `inkling` tool-call parser (opt-in above). Its parser is unit-tested;
no end-to-end tool-calling benchmark has been run yet.

### Client Usage

```python
from openai import OpenAI

client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1")

response = client.chat.completions.create(
    model="thinkingmachines/Inkling-2-Small",
    messages=[{
        "role": "user",
        "content": [
            {"type": "text", "text": "Describe this image."},
            {"type": "image_url", "image_url": {"url": "https://example.com/photo.jpg"}},
        ],
    }],
)
print(response.choices[0].message.content)
```

### Benchmarking

Benchmark through the chat endpoint so the server applies the chat template and the
reasoning parser. DSpark acceptance depends on the prompts and the sampling temperature:
random-token prompts accept almost nothing, so benchmark on realistic text such as SPEED-Bench.
The DSpark arguments above sample drafts probabilistically; with vLLM's default greedy
drafting, acceptance at temperature 1.0 is about 15-25% lower.

```bash
vllm bench serve \
  --backend openai-chat --endpoint /v1/chat/completions \
  --model thinkingmachines/Inkling-2-Small \
  --dataset-name speed_bench --speed-bench-dataset-subset throughput_1k \
  --speed-bench-output-len 1024 --max-concurrency 16 --num-prompts 64
```

### References

- [Inkling 2 Small announcement](https://thinkingmachines.ai/news/inkling-2-small/)
- [Inkling 2 Small on Hugging Face](https://huggingface.co/thinkingmachines/Inkling-2-Small)
- [vLLM blog: Inkling 2 Small on vLLM](https://vllm.ai/blog/2026-11-12-inkling2-small)
- [Inkling recipe](https://recipes.vllm.ai/thinkingmachines/Inkling)
- [DeepSeek DSpark (DeepSpec)](https://github.com/deepseek-ai/DeepSpec)