{ "hf_id": "thinkingmachines/Inkling-Small", "meta": { "title": "TML Inkling-Small", "slug": "tml-inkling-small", "provider": "Thinking Machines Lab", "description": "Natively multimodal 276B-parameter MoE from Thinking Machines Lab — 12B active parameters, text/image/audio in, text out, and up to 1M context.", "date_added": "2026-07-30", "date_updated": "2026-09-29", "tasks": [ "multimodal" ], "performance_headline": "276B total / 12B active Inkling architecture for lower-cost, lower-latency deployment", "related_recipes": [ "thinkingmachines/Inkling" ], "hardware": { "mi300x": "verified", "mi355x": "verified" } }, "recommended_command": { "hardware": "h200", "strategy": "single_node_tp", "variant": "default", "node_count": 1, "deploy_type": "single_node", "env": { "VLLM_USE_V2_MODEL_RUNNER": "1", "FLASH_ATTENTION_CUTE_DSL_CACHE_ENABLED": "1" }, "docker_image": "vllm/vllm-openai:nightly", "command": "vllm serve thinkingmachines/Inkling-Small-NVFP4 \\\n --trust-remote-code \\\n --tokenizer-mode inkling \\\n --disable-flashinfer-autotune \\\n --tensor-parallel-size 2 \\\n --enable-auto-tool-choice \\\n --tool-call-parser inkling \\\n --reasoning-parser inkling", "argv": [ "vllm", "serve", "thinkingmachines/Inkling-Small-NVFP4", "--trust-remote-code", "--tokenizer-mode", "inkling", "--disable-flashinfer-autotune", "--tensor-parallel-size", "2", "--enable-auto-tool-choice", "--tool-call-parser", "inkling", "--reasoning-parser", "inkling" ], "docker_command": "docker run --gpus all \\\n --privileged --ipc=host -p 8000:8000 \\\n -v ~/.cache/huggingface:/root/.cache/huggingface \\\n -e VLLM_USE_V2_MODEL_RUNNER=1 \\\n -e FLASH_ATTENTION_CUTE_DSL_CACHE_ENABLED=1 \\\n vllm/vllm-openai:nightly thinkingmachines/Inkling-Small-NVFP4 \\\n --trust-remote-code \\\n --tokenizer-mode inkling \\\n --disable-flashinfer-autotune \\\n --tensor-parallel-size 2 \\\n --enable-auto-tool-choice \\\n --tool-call-parser inkling \\\n --reasoning-parser inkling", "docker_argv": [ "docker", "run", "--gpus", "all", "--privileged", "--ipc=host", "-p", "8000:8000", "-v", "~/.cache/huggingface:/root/.cache/huggingface", "-e", "VLLM_USE_V2_MODEL_RUNNER=1", "-e", "FLASH_ATTENTION_CUTE_DSL_CACHE_ENABLED=1", "vllm/vllm-openai:nightly", "thinkingmachines/Inkling-Small-NVFP4", "--trust-remote-code", "--tokenizer-mode", "inkling", "--disable-flashinfer-autotune", "--tensor-parallel-size", "2", "--enable-auto-tool-choice", "--tool-call-parser", "inkling", "--reasoning-parser", "inkling" ], "strategy_spec": { "name": "single_node_tp", "deploy_type": "single_node", "display_name": "Tensor Parallel", "short_label": "TP", "orientation": "latency", "description": "Single-node tensor parallel. Splits the model across all local GPUs. TP size is set to the GPU count at deploy time. The simplest multi-GPU strategy — works for all model architectures.\n", "hardware_match": { "min_gpus": 1, "max_gpus": 8, "multi_node": false }, "vllm_args": [], "parallel_flag": "--tensor-parallel-size" }, "hardware_profile": { "brand": "NVIDIA", "generation": "hopper", "compute_arch": "sm90", "display_name": "H200", "description": "NVIDIA H200 SXM 141 GB HBM3e · 8-GPU HGX node", "gpu_count": 8, "vram_gb": 1128, "multi_node": false }, "alternatives": { "single_node_tep": "/thinkingmachines/Inkling-Small/strategies/single_node_tep.json", "multi_node_tp_dp": "/thinkingmachines/Inkling-Small/strategies/multi_node_tp_dp.json", "multi_node_dep": "/thinkingmachines/Inkling-Small/strategies/multi_node_dep.json", "pd_cluster": "/thinkingmachines/Inkling-Small/strategies/pd_cluster.json", "kv_store_centralized_mooncake": "/thinkingmachines/Inkling-Small/strategies/kv_store_centralized_mooncake.json", "kv_store_distributed_mooncake": "/thinkingmachines/Inkling-Small/strategies/kv_store_distributed_mooncake.json" }, "by_hardware": { "h200": "/thinkingmachines/Inkling-Small/hw/h200.json", "h100": "/thinkingmachines/Inkling-Small/hw/h100.json", "b200": "/thinkingmachines/Inkling-Small/hw/b200.json", "gb200": "/thinkingmachines/Inkling-Small/hw/gb200.json", "b300": "/thinkingmachines/Inkling-Small/hw/b300.json", "gb300": "/thinkingmachines/Inkling-Small/hw/gb300.json" } }, "model": { "model_id": "thinkingmachines/Inkling-Small-NVFP4", "min_vllm_version": "0.26.0", "nightly_required": true, "architecture": "moe", "parameter_count": "276B", "active_parameters": "12B", "context_length": 1048576, "base_args": [ "--trust-remote-code", "--tokenizer-mode", "inkling" ], "base_env": { "VLLM_USE_V2_MODEL_RUNNER": "1" }, "install": { "pip": { "command": "uv venv\nsource .venv/bin/activate\nuv pip install -U vllm --pre \\\n --extra-index-url https://wheels.vllm.ai/nightly/cu130 \\\n --extra-index-url https://download.pytorch.org/whl/cu130 \\\n --index-strategy unsafe-best-match", "note": "vLLM 0.26.0 isn't released yet — nightly required. Nightly wheels are CUDA 13 (cu130) only; the host needs an r580+ NVIDIA driver." }, "docker": { "command": "docker pull vllm/vllm-openai:latest" }, "extras": [ { "note": "Audio extras — only needed when serving the audio modality", "command": "uv pip install \"vllm[audio]\"", "optional": true } ] } }, "features": { "tool_calling": { "description": "Enable auto tool choice with the Inkling tool-call parser.", "args": [ "--enable-auto-tool-choice", "--tool-call-parser", "inkling" ] }, "reasoning": { "description": "Enable reasoning output parsing with the Inkling reasoning parser.", "args": [ "--reasoning-parser", "inkling" ] }, "spec_decoding": { "description": "Multi-Token Prediction (MTP) speculative decoding. Inkling ships 8 chained MTP heads, but the stable version of vLLM only supports 1 speculative token for now. Will support 8 in the next release.", "args": [ "--speculative-config", "{\"method\":\"mtp\",\"num_speculative_tokens\":1}" ] } }, "opt_in_features": [ "spec_decoding" ], "variants": { "default": { "precision": "nvfp4", "precision_hardware": { "brand": "NVIDIA" }, "vram_minimum_gb": 180, "supported_hardware": [ "arch:sm90", "arch:sm100", "arch:sm103" ], "tp": { "hopper": 4, "b300": 1, "b200": 2, "gb200": 2, "gb300": 1, "h200": 2 }, "description": "NVFP4 checkpoint requiring at least 180 GB aggregated VRAM: W4A4 with TP1 on B300/GB300 or TP2 on B200/GB200, and W4A16 with TP4 on H100 or TP2 on H200.", "extra_args": [ "--disable-flashinfer-autotune" ], "extra_env": { "FLASH_ATTENTION_CUTE_DSL_CACHE_ENABLED": "1" } }, "mxfp4": { "model_id": "EmbeddedLLM/Inkling-Small-MXFP4", "precision": "mxfp4", "precision_hardware": { "brand": "AMD" }, "vram_minimum_gb": 160, "supported_hardware": [ "mi300x", "mi325x", "mi355x" ], "tp": { "mi300x": 2, "mi325x": 2, "mi355x": 1 }, "description": "Quark OCP MXFP4 checkpoint for AMD Instinct. Its measured MI355X TP1 load footprint was 149.03 GiB consumed (~160.0 GB decimal); add KV-cache capacity for serving. Full 1M context is validated at TP1 on MI355X and TP2 on MI300X. MI325X uses the same gfx942 TP2 path but is not yet smoke-tested.", "hardware_overrides": { "mi300x": { "docker_image": "vllm/vllm-openai-rocm:nightly", "extra_args": [ "--moe-backend", "auto", "--block-size", "128", "--gpu-memory-utilization", "0.85" ], "extra_env": { "VLLM_ROCM_USE_AITER": "1", "VLLM_ROCM_USE_AITER_MOE": "1", "INKLING_GFX950_GLUON": "0" } }, "mi325x": { "docker_image": "vllm/vllm-openai-rocm:nightly", "extra_args": [ "--moe-backend", "auto", "--block-size", "128", "--gpu-memory-utilization", "0.85" ], "extra_env": { "VLLM_ROCM_USE_AITER": "1", "VLLM_ROCM_USE_AITER_MOE": "1", "INKLING_GFX950_GLUON": "0" } }, "mi355x": { "docker_image": "vllm/vllm-openai-rocm:nightly", "extra_args": [ "--moe-backend", "aiter", "--block-size", "128", "--gpu-memory-utilization", "0.95" ], "extra_env": { "VLLM_ROCM_USE_AITER": "1", "VLLM_ROCM_USE_AITER_MOE": "1", "INKLING_GFX950_GLUON": "0" } } }, "json": "/EmbeddedLLM/Inkling-Small-MXFP4.json" }, "bf16": { "model_id": "thinkingmachines/Inkling-Small", "precision": "bf16", "vram_minimum_gb": 600, "supported_hardware": [ "b300", "b200", "gb200", "gb300", "h200", "mi300x", "mi325x", "mi355x" ], "tp": { "b300": 4, "b200": 8, "gb200": 4, "gb300": 4, "h200": 8, "mi300x": 4, "mi325x": 4, "mi355x": 4 }, "description": "Full BF16 checkpoint requiring at least 600 GB aggregated VRAM: TP4 on B300/GB200/GB300 and AMD Instinct, or TP8 on B200/H200. MI355X TP4 is smoke-tested; MI300X/MI325X use the gfx942 automatic MoE path pending direct BF16 smoke.", "hardware_overrides": { "nvidia": { "extra_args": [ "--disable-flashinfer-autotune" ], "extra_env": { "FLASH_ATTENTION_CUTE_DSL_CACHE_ENABLED": "1" } }, "mi300x": { "docker_image": "vllm/vllm-openai-rocm:nightly", "extra_args": [ "--moe-backend", "auto", "--block-size", "128", "--gpu-memory-utilization", "0.90", "--enforce-eager" ], "extra_env": { "VLLM_ROCM_USE_AITER": "1", "VLLM_ROCM_USE_AITER_MOE": "1" } }, "mi325x": { "docker_image": "vllm/vllm-openai-rocm:nightly", "extra_args": [ "--moe-backend", "auto", "--block-size", "128", "--gpu-memory-utilization", "0.90", "--enforce-eager" ], "extra_env": { "VLLM_ROCM_USE_AITER": "1", "VLLM_ROCM_USE_AITER_MOE": "1" } }, "mi355x": { "docker_image": "vllm/vllm-openai-rocm:nightly", "extra_args": [ "--moe-backend", "aiter", "--block-size", "128", "--gpu-memory-utilization", "0.90", "--enforce-eager" ], "extra_env": { "VLLM_ROCM_USE_AITER": "1", "VLLM_ROCM_USE_AITER_MOE": "1" } } } }, "fp8": { "model_id": "RedHatAI/Inkling-Small-FP8-dynamic", "precision": "fp8", "vram_minimum_gb": 332, "description": "RedHatAI FP8 (E4M3) with dynamic per-token activation quantization", "extra_args": [ "--kv-cache-dtype", "fp8" ], "json": "/RedHatAI/Inkling-Small-FP8-dynamic.json" } }, "strategy_hardware": { "multi_node_tp": { "hopper": "unsupported" }, "multi_node_tep": { "hopper": "unsupported" } }, "kv_offload_support": { "offloading_cpu": "verified", "offloading_fs": "verified" }, "hardware_overrides": {}, "guide": "## Overview\n\nTML Inkling-Small is a natively multimodal 276B-parameter Mixture-of-Experts model\nfrom Thinking Machines Lab. It activates 12B parameters per token and uses the same\nInkling architecture and vLLM serving configuration as its larger 975B-parameter\nsibling, while substantially reducing the deployment footprint.\n\nInkling-Small accepts text, image, and audio inputs and generates text. It supports\nInkling's relative-attention and short-convolution serving path, tool and reasoning\nparsers, and MTP speculative decoding.\n\n## Variants\n\n- **NVFP4 (default):** Requires at least 180 GB aggregated VRAM. Run W4A4 with TP1\n on B300/GB300 or TP2 on B200/GB200. On Hopper, run W4A16 with TP4 on H100\n or TP2 on H200; the MoE kernel dequantizes the NVFP4 weights to BF16 on the fly.\n- **BF16:** Requires at least 600 GB aggregated VRAM. Run TP4 on B300/GB200/GB300\n or AMD MI300X/MI325X/MI355X, and TP8 on B200/H200. MI355X TP4 is validated;\n direct MI300X/MI325X BF16 smoke is pending.\n- **MXFP4 (AMD):** `EmbeddedLLM/Inkling-Small-MXFP4` keeps the full 1M-token\n context at TP1 on MI355X or TP2 on MI300X. MI355X runs native AITER MXFP4\n MoE; MI300X uses the supported Triton on-the-fly MXFP4 emulation path.\n\n## Prerequisites\n\n- **Hardware:** B300/GB300 TP1, B200/GB200/H200 TP2, or H100 TP4 for NVFP4;\n B300/GB200/GB300/MI300X/MI325X/MI355X TP4 or B200/H200 TP8 for BF16;\n MI355X TP1 or MI300X/MI325X TP2 for AMD MXFP4.\n- **vLLM:** A nightly build with Inkling support.\n- **Audio:** Install the optional `vllm[audio]` extra only when serving audio inputs.\n\n## Client Usage\n\nLaunch the NVFP4 checkpoint in W4A4 mode on B300:\n```bash\nexport VLLM_USE_V2_MODEL_RUNNER=1\nexport FLASH_ATTENTION_CUTE_DSL_CACHE_ENABLED=1\n\nvllm serve thinkingmachines/Inkling-Small-NVFP4 \\\n --tokenizer-mode inkling \\\n --reasoning-parser inkling \\\n --tool-call-parser inkling \\\n --enable-auto-tool-choice \\\n --tensor-parallel-size 1 \\\n --disable-flashinfer-autotune \\\n --trust-remote-code\n```\n\nSelect **GB300** for W4A4 with TP1, or **B200**/**GB200** for W4A4 with TP2.\nOn H100, the builder emits TP4; on H200, it emits TP2. vLLM runs W4A16 with on-the-fly\ndequantization. A reported 4×H100 deployment served a plain chat request, but\nJSON-schema structured output returned HTTP 500; see\n[vLLM #51693](https://github.com/vllm-project/vllm/issues/51693). H100 is not\nmarked verified for the full recipe.\n\nOn AMD, select the **MXFP4** variant. To fit 1M context, MI300X TP2 and MI355X TP1.\nMI355X uses `--moe-backend aiter`; MI300X uses `--moe-backend auto`\nbecause native AITER W4A4 is gated to gfx950, while gfx942 falls back to\n`OCP_MXQuantizationEmulationTritonExperts`. Inkling relative attention remains\non its model-specific Triton path (`INKLING_GFX950_GLUON=0`), which was faster\nthan the optional gfx950 Gluon path in the serving benchmark.\n\nEquivalent MI355X launch:\n```bash\nexport VLLM_USE_V2_MODEL_RUNNER=1\nexport VLLM_ROCM_USE_AITER=1\nexport VLLM_ROCM_USE_AITER_MOE=1\nexport INKLING_GFX950_GLUON=0\n\nvllm serve EmbeddedLLM/Inkling-Small-MXFP4 \\\n --tensor-parallel-size 1 \\\n --tokenizer-mode inkling \\\n --reasoning-parser inkling \\\n --tool-call-parser inkling \\\n --enable-auto-tool-choice \\\n --moe-backend aiter \\\n --block-size 128 \\\n --trust-remote-code\n```\n\nQuery the server with the OpenAI SDK:\n```python\nfrom openai import OpenAI\n\nclient = OpenAI(api_key=\"EMPTY\", base_url=\"http://localhost:8000/v1\")\n\nresponse = client.chat.completions.create(\n model=\"thinkingmachines/Inkling-Small-NVFP4\",\n messages=[{\n \"role\": \"user\",\n \"content\": [\n {\"type\": \"text\", \"text\": \"Describe this image.\"},\n {\"type\": \"image_url\", \"image_url\": {\"url\": \"https://example.com/photo.jpg\"}},\n ],\n }],\n)\nprint(response.choices[0].message.content)\n```\n\n## References\n\n- [TML Inkling announcement](https://thinkingmachines.ai/news/introducing-inkling/#inkling-small)\n- [Inkling-Small MXFP4 for AMD](https://huggingface.co/EmbeddedLLM/Inkling-Small-MXFP4)\n- [TML Inkling recipe](https://docs.vllm.ai/projects/recipes/en/latest/thinkingmachines/Inkling.html)\n", "source": { "source": "vllm", "repo": "vllm-project/vllm", "ref": "main", "sha": "564ac0de1b228568cafcbfbe8ccaccc82d6023eb", "path": "recipes/models/thinkingmachines/Inkling-Small.yaml", "edit_url": "https://github.com/vllm-project/vllm/edit/main/recipes/models/thinkingmachines/Inkling-Small.yaml", "overlays": [] } }