vLLM/Recipes
Search
Thinking Machines Lab

thinkingmachines/Inkling-2-Small-MXFP8

MXFP8 checkpoint of Inkling 2 Small, the multimodal (text and image in, text out) MoE from Thinking Machines Lab with 594B total and 17B active parameters, up to 1M context, and a bundled DSpark speculator.

moe594B / 17B1,048,576 ctxvLLM 0.35.0+nightlyMultimodal
Default configuration: 8× H200 · Tensor Parallel · MXFP8
vllm serve thinkingmachines/Inkling-2-Small-MXFP8 \
  --tensor-parallel-size 8 \
  --max-model-len 524288 \
  --max-num-seqs 256 \
  --reasoning-parser inkling \
  --speculative-config '{"method":"dspark","num_speculative_tokens":7,"draft_sample_method":"probabilistic"}'

The interactive builder loads with JavaScript. Append .md to this page's URL, query string included, for any configuration's command as Markdown — e.g. /thinkingmachines/Inkling-2-Small-MXFP8.md.

Overview

This is the MXFP8 checkpoint of Inkling 2 Small: the same model, with every routed expert in MXFP8 instead of NVFP4 (layers 0-39) and MXFP8 (layers 40-47). Inkling 2 Small was trained in NVFP4, and Thinking Machines recommends the NVFP4 checkpoint for all use cases. MXFP8 scores 1-2 points higher on evals, at about 1.6x the memory. See the NVFP4 recipe for the architecture, LoRA support, client usage and benchmarking.

Runtime

Inkling 2 Small is supported in vLLM main. Use the vllm/vllm-openai:nightly image or a nightly wheel until it is in a vLLM release.

Configurations

The checkpoint needs about 750 GB, so it runs at TP8 (TP8 attention with EP8 experts): one 8-GPU H200, B200 or B300 node, or two GB200 or GB300 trays.

  • H200, TP8: contexts up to 524,288 tokens. With the command above (DSpark and the reasoning parser), GSM8K is 0.972-0.974 (0-shot, chat) with a DSpark acceptance length of 4.5.
  • Blackwell, TP8: the full 1,048,576-token context.

On Hopper, the experts run as W8A16 GEMMs with BF16 activations.

References