{
  "hf_id": "thinkingmachines/Inkling-2-Mini",
  "meta": {
    "title": "Inkling 2 Mini",
    "slug": "inkling-2-mini",
    "provider": "Thinking Machines Lab",
    "description": "Looped multimodal (text and image in, text out) MoE from Thinking Machines Lab with 26B total and 3.5B active parameters and up to 1M context: its 12 middle layers run three passes per token, each with its own KV cache and gated messages from the previous pass. Bundles a DSpark speculator.",
    "date_added": "2026-11-20",
    "date_updated": "2026-11-20",
    "difficulty": "intermediate",
    "tasks": [
      "multimodal"
    ],
    "performance_headline": "~4,400 output tok/s/GPU at 100 tok/s/user (32K-context agentic sessions) with DSpark on 1 GB200 (TP1)",
    "related_recipes": [
      "thinkingmachines/Inkling-2-Small"
    ],
    "default_hardware": "gb200",
    "hardware": {
      "gb200": "verified",
      "b300": "verified",
      "gb300": "verified",
      "h100": "verified",
      "h200": "verified",
      "dgx_spark_gb10": "verified"
    }
  },
  "recommended_command": {
    "hardware": "gb200",
    "strategy": "single_node_tp",
    "variant": "default",
    "node_count": 1,
    "deploy_type": "single_node",
    "env": {},
    "docker_image": "vllm/vllm-openai:inkling-2-mini",
    "command": "vllm serve thinkingmachines/Inkling-2-Mini \\\n  --tensor-parallel-size 1 \\\n  --max-model-len 262144 \\\n  --reasoning-parser inkling \\\n  --speculative-config '{\"method\":\"dspark\",\"num_speculative_tokens\":7,\"draft_sample_method\":\"probabilistic\"}'",
    "argv": [
      "vllm",
      "serve",
      "thinkingmachines/Inkling-2-Mini",
      "--tensor-parallel-size",
      "1",
      "--max-model-len",
      "262144",
      "--reasoning-parser",
      "inkling",
      "--speculative-config",
      "{\"method\":\"dspark\",\"num_speculative_tokens\":7,\"draft_sample_method\":\"probabilistic\"}"
    ],
    "docker_command": "docker run --gpus all \\\n  --privileged --ipc=host -p 8000:8000 \\\n  -v ~/.cache/huggingface:/root/.cache/huggingface \\\n  vllm/vllm-openai:inkling-2-mini thinkingmachines/Inkling-2-Mini \\\n  --tensor-parallel-size 1 \\\n  --max-model-len 262144 \\\n  --reasoning-parser inkling \\\n  --speculative-config '{\"method\":\"dspark\",\"num_speculative_tokens\":7,\"draft_sample_method\":\"probabilistic\"}'",
    "docker_argv": [
      "docker",
      "run",
      "--gpus",
      "all",
      "--privileged",
      "--ipc=host",
      "-p",
      "8000:8000",
      "-v",
      "~/.cache/huggingface:/root/.cache/huggingface",
      "vllm/vllm-openai:inkling-2-mini",
      "thinkingmachines/Inkling-2-Mini",
      "--tensor-parallel-size",
      "1",
      "--max-model-len",
      "262144",
      "--reasoning-parser",
      "inkling",
      "--speculative-config",
      "{\"method\":\"dspark\",\"num_speculative_tokens\":7,\"draft_sample_method\":\"probabilistic\"}"
    ],
    "strategy_spec": {
      "name": "single_node_tp",
      "deploy_type": "single_node",
      "display_name": "Tensor Parallel",
      "short_label": "TP",
      "orientation": "latency",
      "description": "Single-node tensor parallel. Splits the model across all local GPUs. TP size is set to the GPU count at deploy time. The simplest multi-GPU strategy — works for all model architectures.\n",
      "hardware_match": {
        "min_gpus": 1,
        "max_gpus": 8,
        "multi_node": false
      },
      "vllm_args": [],
      "parallel_flag": "--tensor-parallel-size"
    },
    "hardware_profile": {
      "brand": "NVIDIA",
      "generation": "blackwell",
      "compute_arch": "sm100",
      "display_name": "GB200 NVL4",
      "description": "NVIDIA GB200 Grace-Blackwell compute tray · 192 GB HBM3e/GPU · 4 Blackwell GPUs (NVL72 tray unit)",
      "gpu_count": 4,
      "vram_gb": 768,
      "multi_node": false
    },
    "alternatives": {
      "kv_store_centralized_mooncake": "/thinkingmachines/Inkling-2-Mini/strategies/kv_store_centralized_mooncake.json",
      "kv_store_distributed_mooncake": "/thinkingmachines/Inkling-2-Mini/strategies/kv_store_distributed_mooncake.json"
    },
    "by_hardware": {
      "gb200": "/thinkingmachines/Inkling-2-Mini/hw/gb200.json",
      "b200": "/thinkingmachines/Inkling-2-Mini/hw/b200.json",
      "b300": "/thinkingmachines/Inkling-2-Mini/hw/b300.json",
      "gb300": "/thinkingmachines/Inkling-2-Mini/hw/gb300.json",
      "dgx_spark_gb10": "/thinkingmachines/Inkling-2-Mini/hw/dgx_spark_gb10.json"
    }
  },
  "model": {
    "model_id": "thinkingmachines/Inkling-2-Mini",
    "min_vllm_version": "0.35.0",
    "nightly_required": true,
    "docker_image": "vllm/vllm-openai:inkling-2-mini",
    "architecture": "moe",
    "parameter_count": "26B",
    "active_parameters": "3.5B",
    "context_length": 1048576,
    "base_args": [],
    "base_env": {},
    "install": {
      "docker": {
        "command": "docker pull vllm/vllm-openai:inkling-2-mini",
        "note": "Serve from the `vllm/vllm-openai:inkling-2-mini` image; no pip wheel is available yet."
      }
    }
  },
  "features": {
    "reasoning": {
      "description": "Parse reasoning output with the Inkling reasoning parser.",
      "args": [
        "--reasoning-parser",
        "inkling"
      ]
    },
    "tool_calling": {
      "description": "Enable auto tool choice with the Inkling tool-call parser.",
      "args": [
        "--enable-auto-tool-choice",
        "--tool-call-parser",
        "inkling"
      ]
    },
    "spec_decoding": {
      "description": "DSpark speculative decoding with the draft bundled in the checkpoint: a 3-layer parallel block drafter plus a recurrent head, 7 draft tokens per step, reading the target's hidden states at four layer visits (two inside the loop). Drafts are sampled from the draft distribution, as in the reference.",
      "args": [
        "--speculative-config",
        "{\"method\":\"dspark\",\"num_speculative_tokens\":7,\"draft_sample_method\":\"probabilistic\"}"
      ]
    }
  },
  "opt_in_features": [
    "tool_calling"
  ],
  "variants": {
    "default": {
      "precision": "nvfp4",
      "vram_minimum_gb": 24,
      "description": "Routed experts in NVFP4 (layers 0-19) and MXFP8 (layers 20-23) with activations quantized to the same format; attention, routers, attention-residual queries, loop controllers and the LM head in BF16/FP32. ~20 GB on disk including the 0.6 GB DSpark draft."
    }
  },
  "hardware_overrides": {},
  "guide": "## Overview\n\nInkling 2 Mini is a looped Mixture-of-Experts model from Thinking Machines (text and image in,\ntext out) with 26B total and 3.5B active parameters and up to 1M tokens of context. It ships\nwith a DSpark speculator in the same checkpoint.\n\nIts 24 layers split into a prelude (layers 0-5), a loop core (layers 6-17) and a coda\n(layers 18-23). The core runs three passes with shared weights, so every token makes 48\nlayer visits:\n\n- **Every pass starts from the prelude's output.** From the second pass on, each core\n  sublayer input (attention query, key and value inputs, and the MLP input) reads a message\n  from the previous pass's two attention-residual blocks and merges it through a gated\n  rank-64 controller.\n- **Each pass keeps its own memory:** every core layer has a KV cache and a conv-state cache\n  per pass (48 of each in total). vLLM manages them like any other cache layers, so prefix\n  caching and CUDA graphs work as usual.\n\nEach layer uses Inkling 2 Small's components at hidden size 2,304: block attention residuals,\nrelative-bias attention with a gate and sink (20 sliding-window and 4 full-attention layers,\n24 query and 8 KV heads), four short convolutions, and 128 routed experts (top-8). The image\nencoder is a 23-layer ViT.\n\n## Runtime\n\nUse the `vllm/vllm-openai:inkling-2-mini` image until Inkling 2 Mini support is in a vLLM release.\n\n## Configurations\n\nInkling 2 Mini runs on NVIDIA Blackwell (B200, GB200, B300, GB300), Hopper (H100, H200) and\nDGX Spark, and is verified on GB200, B300, GB300, H100, H200 and DGX Spark. The configurations\nbelow were verified on GB200 with DSpark and the reasoning parser enabled:\n\n- **One GPU, TP1:** contexts up to 262,144 tokens. This is the most efficient way to serve\n  the model; TP4 gives lower output throughput per GPU.\n- **TP4 (`--tensor-parallel-size 4 --max-model-len 1048576`):** the full 1,048,576-token\n  context.\n\nOn one GPU, the KV cache bounds the number of concurrent long sessions: with 32K, 64K and\n128K-token sessions, about 45, 32 and 20 sessions run at 100 tokens/s per user.\n\nTool calling uses the `inkling` tool-call parser (opt-in above).\n\n## Client Usage\n\n```python\nfrom openai import OpenAI\n\nclient = OpenAI(api_key=\"EMPTY\", base_url=\"http://localhost:8000/v1\")\n\nresponse = client.chat.completions.create(\n    model=\"thinkingmachines/Inkling-2-Mini\",\n    messages=[{\n        \"role\": \"user\",\n        \"content\": [\n            {\"type\": \"text\", \"text\": \"Describe this image.\"},\n            {\"type\": \"image_url\", \"image_url\": {\"url\": \"https://example.com/photo.jpg\"}},\n        ],\n    }],\n)\nprint(response.choices[0].message.content)\n```\n\n## Benchmarking\n\nBenchmark through the chat endpoint so the server applies the chat template and the\nreasoning parser. DSpark acceptance depends on the prompts and the sampling temperature:\nrandom-token prompts accept almost nothing, so benchmark on realistic text such as SPEED-Bench.\n\n```bash\nvllm bench serve \\\n  --backend openai-chat --endpoint /v1/chat/completions \\\n  --model thinkingmachines/Inkling-2-Mini \\\n  --dataset-name speed_bench --speed-bench-dataset-subset throughput_1k \\\n  --speed-bench-output-len 1024 --max-concurrency 16 --num-prompts 64\n```\n\n## References\n\n- [Inkling 2 Mini announcement](https://thinkingmachines.ai/news/inkling-2-mini/)\n- [Inkling 2 Mini on Hugging Face](https://huggingface.co/thinkingmachines/Inkling-2-Mini)\n- [vLLM blog: Inkling 2 Mini on vLLM](https://vllm.ai/blog/2026-11-20-inkling2-mini)\n- [Inkling 2 Small recipe](https://recipes.vllm.ai/thinkingmachines/Inkling-2-Small)\n- [DeepSeek DSpark (DeepSpec)](https://github.com/deepseek-ai/DeepSpec)\n",
  "source": {
    "source": "vllm",
    "repo": "vllm-project/vllm",
    "ref": "main",
    "sha": "d53154d9f354ebd4a4deb4fd81935db0e266c5fc",
    "path": "recipes/models/thinkingmachines/Inkling-2-Mini.yaml",
    "edit_url": "https://github.com/vllm-project/vllm/edit/main/recipes/models/thinkingmachines/Inkling-2-Mini.yaml",
    "overlays": []
  }
}