thinkingmachines/Inkling-2-Small-MXFP8
MXFP8 checkpoint of Inkling 2 Small, the multimodal (text and image in, text out) MoE from Thinking Machines Lab with 594B total and 17B active parameters, up to 1M context, and a bundled DSpark speculator.
vllm serve thinkingmachines/Inkling-2-Small-MXFP8 \
--tensor-parallel-size 8 \
--max-model-len 524288 \
--max-num-seqs 256 \
--reasoning-parser inkling \
--speculative-config '{"method":"dspark","num_speculative_tokens":7,"draft_sample_method":"probabilistic"}'The interactive builder loads with JavaScript. Append .md to this page's URL, query string included, for any configuration's command as Markdown — e.g. /thinkingmachines/Inkling-2-Small-MXFP8.md.
Overview
This is the MXFP8 checkpoint of Inkling 2 Small: the same model, with every routed expert in MXFP8 instead of NVFP4 (layers 0-39) and MXFP8 (layers 40-47). Inkling 2 Small was trained in NVFP4, and Thinking Machines recommends the NVFP4 checkpoint for all use cases. MXFP8 scores 1-2 points higher on evals, at about 1.6x the memory. See the NVFP4 recipe for the architecture, LoRA support, client usage and benchmarking.
Runtime
Inkling 2 Small is supported in vLLM main. Use the vllm/vllm-openai:nightly image or a
nightly wheel until it is in a vLLM release.
Configurations
The checkpoint needs about 750 GB, so it runs at TP8 (TP8 attention with EP8 experts): one 8-GPU H200, B200 or B300 node, or two GB200 or GB300 trays.
- H200, TP8: contexts up to 524,288 tokens. With the command above (DSpark and the reasoning parser), GSM8K is 0.972-0.974 (0-shot, chat) with a DSpark acceptance length of 4.5.
- Blackwell, TP8: the full 1,048,576-token context.
On Hopper, the experts run as W8A16 GEMMs with BF16 activations.
References
AssertionError: Executable was changed during NUMA binding contextMulti-Node Tensor Parallel
On the headless node of a multi-node launch, vLLM's default NUMA binding swaps the global multiprocessing executable while workers start from concurrent threads. Launch every node with --no-numa-bind.