# thinkingmachines/Inkling-2-Small-MXFP8 — vLLM serving recipe

> MXFP8 checkpoint of Inkling 2 Small, the multimodal (text and image in, text out) MoE from Thinking Machines Lab with 594B total and 17B active parameters, up to 1M context, and a bundled DSpark speculator.

Configuration: hardware `h200` (8× H200) · strategy `single_node_tp` (Tensor Parallel) · variant `default` (MXFP8) · features `reasoning`, `spec_decoding`

- Interactive page: https://recipes.vllm.ai/thinkingmachines/Inkling-2-Small-MXFP8
- This configuration as JSON: https://recipes.vllm.ai/thinkingmachines/Inkling-2-Small-MXFP8.json?hardware=h200

The commands below were rendered for exactly this configuration and take precedence over any example in the guide.

## Other configurations

Change the query string (the same parameters as the page URL) and fetch this URL again. Unknown or incompatible values fall back to the recipe default, so check the Configuration line.

- `hardware`: `h200`, `b200`, `gb200`, `b300`, `gb300`
- `variant`: `default`
- `strategy`: `single_node_tp`, `multi_node_tp` (multi-node strategies also need `nodes=<n>`)
- `features`: comma-separated list of `reasoning`, `tool_calling`, `spec_decoding` (default: `reasoning`, `spec_decoding`). Use `features=none` to turn them all off.

## Install

Requires vLLM >= 0.35.0 (not released yet — nightly wheel).

vLLM 0.35.0 isn't released yet — nightly required. Nightly wheels are CUDA 13 (cu130) only; the host needs an r580+ NVIDIA driver.

```bash
uv venv
source .venv/bin/activate
uv pip install -U vllm --pre \
  --extra-index-url https://wheels.vllm.ai/nightly/cu130 \
  --extra-index-url https://download.pytorch.org/whl/cu130 \
  --index-strategy unsafe-best-match
```

Docker image: `vllm/vllm-openai:nightly`

## Serve

```bash
vllm serve thinkingmachines/Inkling-2-Small-MXFP8 \
  --tensor-parallel-size 8 \
  --max-model-len 524288 \
  --max-num-seqs 256 \
  --reasoning-parser inkling \
  --speculative-config '{"method":"dspark","num_speculative_tokens":7,"draft_sample_method":"probabilistic"}'
```

Docker equivalent:

```bash
docker run --gpus all \
  --privileged --ipc=host -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai:nightly thinkingmachines/Inkling-2-Small-MXFP8 \
  --tensor-parallel-size 8 \
  --max-model-len 524288 \
  --max-num-seqs 256 \
  --reasoning-parser inkling \
  --speculative-config '{"method":"dspark","num_speculative_tokens":7,"draft_sample_method":"probabilistic"}'
```

## Guide

> Background reference. Commands in it are examples and may target another configuration — use the Serve section above.

### Overview

This is the MXFP8 checkpoint of [Inkling 2 Small](https://recipes.vllm.ai/thinkingmachines/Inkling-2-Small):
the same model, with every routed expert in MXFP8 instead of NVFP4 (layers 0-39) and MXFP8
(layers 40-47). Inkling 2 Small was trained in NVFP4, and Thinking Machines recommends the
NVFP4 checkpoint for all use cases. MXFP8 scores 1-2 points higher on evals, at about 1.6x the
memory. See the NVFP4 recipe for the architecture, LoRA support, client usage and benchmarking.

### Runtime

Inkling 2 Small is supported in vLLM main. Use the `vllm/vllm-openai:nightly` image or a
nightly wheel until it is in a vLLM release.

### Configurations

The checkpoint needs about 750 GB, so it runs at TP8 (TP8 attention with EP8 experts): one
8-GPU H200, B200 or B300 node, or two GB200 or GB300 trays.

- **H200, TP8:** contexts up to 524,288 tokens. With the command above (DSpark and the
  reasoning parser), GSM8K is 0.972-0.974 (0-shot, chat) with a DSpark acceptance length of 4.5.
- **Blackwell, TP8:** the full 1,048,576-token context.

On Hopper, the experts run as W8A16 GEMMs with BF16 activations.

### References

- [Inkling 2 Small recipe (NVFP4)](https://recipes.vllm.ai/thinkingmachines/Inkling-2-Small)
- [Inkling 2 Small MXFP8 on Hugging Face](https://huggingface.co/thinkingmachines/Inkling-2-Small-MXFP8)
- [Inkling 2 Small announcement](https://thinkingmachines.ai/news/inkling-2-small/)
