{ "hf_id": "IFM/K2-Horizon-375B-A23B", "meta": { "title": "K2-Horizon-375B-A23B", "slug": "k2-horizon-375b-a23b", "provider": "IFM", "description": "Frontier-scale sparse MoE model for long-context research and high-capacity serving", "date_added": "2026-08-31", "date_updated": "2026-09-08", "tasks": [ "text" ], "performance_headline": "379.17B stored parameters including embeddings / 26.67B active parameters per token", "related_recipes": [ "IFM/K2-Horizon-MoVA-36B-A4B" ], "hardware": { "h200": "verified", "mi355x": "verified" } }, "recommended_command": { "hardware": "h200", "strategy": "single_node_tep", "variant": "default", "node_count": 1, "deploy_type": "single_node", "env": {}, "docker_image": "vllm/vllm-openai:latest", "command": "vllm serve IFM/K2-Horizon-375B-A23B \\\n --trust-remote-code \\\n --chat-template-content-format string \\\n --dtype bfloat16 \\\n --enable-expert-parallel \\\n --tensor-parallel-size 8 \\\n --reasoning-parser k2_horizon \\\n --enable-auto-tool-choice \\\n --tool-call-parser k2_horizon", "argv": [ "vllm", "serve", "IFM/K2-Horizon-375B-A23B", "--trust-remote-code", "--chat-template-content-format", "string", "--dtype", "bfloat16", "--enable-expert-parallel", "--tensor-parallel-size", "8", "--reasoning-parser", "k2_horizon", "--enable-auto-tool-choice", "--tool-call-parser", "k2_horizon" ], "docker_command": "docker run --gpus all \\\n --privileged --ipc=host -p 8000:8000 \\\n -v ~/.cache/huggingface:/root/.cache/huggingface \\\n vllm/vllm-openai:latest IFM/K2-Horizon-375B-A23B \\\n --trust-remote-code \\\n --chat-template-content-format string \\\n --dtype bfloat16 \\\n --enable-expert-parallel \\\n --tensor-parallel-size 8 \\\n --reasoning-parser k2_horizon \\\n --enable-auto-tool-choice \\\n --tool-call-parser k2_horizon", "docker_argv": [ "docker", "run", "--gpus", "all", "--privileged", "--ipc=host", "-p", "8000:8000", "-v", "~/.cache/huggingface:/root/.cache/huggingface", "vllm/vllm-openai:latest", "IFM/K2-Horizon-375B-A23B", "--trust-remote-code", "--chat-template-content-format", "string", "--dtype", "bfloat16", "--enable-expert-parallel", "--tensor-parallel-size", "8", "--reasoning-parser", "k2_horizon", "--enable-auto-tool-choice", "--tool-call-parser", "k2_horizon" ], "strategy_spec": { "name": "single_node_tep", "deploy_type": "single_node", "display_name": "Tensor + Expert Parallel", "short_label": "TEP", "orientation": "balanced", "description": "Single-node TEP. TP splits dense layers and EP splits expert layers across local GPUs. TP must be set to GPU count to avoid OOM from replicated dense layers. For MoE models only.\n", "hardware_match": { "min_gpus": 2, "max_gpus": 8, "multi_node": false, "architecture": "moe" }, "vllm_args": [ "--enable-expert-parallel" ], "parallel_flag": "--tensor-parallel-size" }, "hardware_profile": { "brand": "NVIDIA", "generation": "hopper", "compute_arch": "sm90", "display_name": "H200", "description": "NVIDIA H200 SXM 141 GB HBM3e ยท 8-GPU HGX node", "gpu_count": 8, "vram_gb": 1128, "multi_node": false }, "alternatives": { "single_node_tp": "/IFM/K2-Horizon-375B-A23B/strategies/single_node_tp.json", "multi_node_tep": "/IFM/K2-Horizon-375B-A23B/strategies/multi_node_tep.json", "multi_node_tp_pp": "/IFM/K2-Horizon-375B-A23B/strategies/multi_node_tp_pp.json", "kv_store_centralized_mooncake": "/IFM/K2-Horizon-375B-A23B/strategies/kv_store_centralized_mooncake.json", "kv_store_distributed_mooncake": "/IFM/K2-Horizon-375B-A23B/strategies/kv_store_distributed_mooncake.json" }, "by_hardware": { "h200": "/IFM/K2-Horizon-375B-A23B/hw/h200.json", "h100": "/IFM/K2-Horizon-375B-A23B/hw/h100.json", "b200": "/IFM/K2-Horizon-375B-A23B/hw/b200.json", "gb200": "/IFM/K2-Horizon-375B-A23B/hw/gb200.json", "b300": "/IFM/K2-Horizon-375B-A23B/hw/b300.json", "gb300": "/IFM/K2-Horizon-375B-A23B/hw/gb300.json", "mi300x": "/IFM/K2-Horizon-375B-A23B/hw/mi300x.json", "mi325x": "/IFM/K2-Horizon-375B-A23B/hw/mi325x.json", "mi350x": "/IFM/K2-Horizon-375B-A23B/hw/mi350x.json", "mi355x": "/IFM/K2-Horizon-375B-A23B/hw/mi355x.json" } }, "model": { "model_id": "IFM/K2-Horizon-375B-A23B", "architecture": "moe", "parameter_count": "379.17B", "active_parameters": "26.67B", "context_length": 524288, "base_args": [ "--trust-remote-code", "--chat-template-content-format", "string", "--dtype", "bfloat16" ], "base_env": {}, "install": { "pip": { "command": "uv venv\nsource .venv/bin/activate\nuv pip install -U vllm --torch-backend auto" }, "docker": { "command": "docker pull vllm/vllm-openai:latest" } } }, "features": { "reasoning": { "description": "Extract K2-Horizon reasoning into message.reasoning", "args": [ "--reasoning-parser", "k2_horizon" ] }, "tool_calling": { "description": "Enable automatic tool choice with the K2-Horizon parser", "args": [ "--enable-auto-tool-choice", "--tool-call-parser", "k2_horizon" ] } }, "variants": { "default": { "precision": "bf16", "vram_minimum_gb": 911, "description": "BF16 serving runtime with an eight-GPU tensor-parallel baseline" } }, "strategy_hardware": { "single_node_tep": { "amd": "unsupported" }, "multi_node_tep": { "amd": "unsupported" } }, "hardware_overrides": { "amd": { "extra_args": [ "--attention-backend", "ROCM_AITER_UNIFIED_ATTN", "--gpu-memory-utilization", "0.9" ], "extra_env": { "VLLM_ROCM_USE_AITER": "1", "NCCL_MIN_NCHANNELS": "112" } } }, "guide": "## Overview\n\nK2-Horizon-375B-A23B is a frontier-scale mixture-of-expert (MoE) model in the IFM\nK2-Horizon family, with 379.17B stored parameters including embeddings,\n26.67B active parameters per token, and support for up to 524,288 tokens of\ncontext.\n\nK2-Horizon is an open-weight IFM model family built for transparent\nfoundation-model research, staged checkpoint analysis, and practical\ndeployment. The series spans compact dense models for local experimentation,\ndense mid-size and large models for high-quality research workloads, and\nmixture-of-expert (MoE) models such as K2-Horizon-375B-A23B for frontier-scale\nresearch and higher-capacity serving.\n\n## Launch Commands\n\n```bash\nvllm serve IFM/K2-Horizon-375B-A23B \\\n --tensor-parallel-size 8 \\\n --trust-remote-code \\\n --chat-template-content-format string \\\n --dtype bfloat16 \\\n --enable-expert-parallel \\\n --reasoning-parser k2_horizon \\\n --enable-auto-tool-choice \\\n --tool-call-parser k2_horizon\n```\n\n### AMD MI355X\n\nVerified on an 8-GPU MI355X (gfx950) node. AITER supplies the fused MoE and\nthe unified attention backend:\n\n```bash\nexport VLLM_ROCM_USE_AITER=1\nexport NCCL_MIN_NCHANNELS=112\n\nvllm serve IFM/K2-Horizon-375B-A23B \\\n --tensor-parallel-size 8 \\\n --trust-remote-code \\\n --chat-template-content-format string \\\n --dtype bfloat16 \\\n --max-model-len 131072 \\\n --gpu-memory-utilization 0.9 \\\n --attention-backend ROCM_AITER_UNIFIED_ATTN \\\n --generation-config auto \\\n --reasoning-parser k2_horizon \\\n --enable-auto-tool-choice \\\n --tool-call-parser k2_horizon\n```\n\n## Client usage\n\n```python\nfrom openai import OpenAI\n\nclient = OpenAI(\n api_key=\"EMPTY\", base_url=\"http://localhost:8000/v1\", timeout=3600\n)\n\nresp = client.chat.completions.create(\n model=\"IFM/K2-Horizon-375B-A23B\",\n messages=[{\"role\": \"user\", \"content\": \"Give me three primes above 100.\"}],\n temperature=1.0, top_p=0.95, max_tokens=2048,\n)\nprint(resp.choices[0].message.content)\n```\n\n### Thinking modes\n\nThe model supports selectable thinking through `chat_template_kwargs`, per\nrequest or server-wide via `--default-chat-template-kwargs`:\n\n- `{\"reasoning_effort\": \"high\"}` โ€” full thinking (default).\n- `{\"reasoning_effort\": \"medium\"}` โ€” faster thinking.\n- `{\"reasoning_effort\": \"low\"}` โ€” fastest thinking.\n\n## References\n\n- [IFM/K2-Horizon-375B-A23B on Hugging Face](https://huggingface.co/IFM/K2-Horizon-375B-A23B)\n- [K2 Horizon Collection](https://huggingface.co/collections/IFM/k2-horizon)\n", "source": { "source": "vllm", "repo": "vllm-project/vllm", "ref": "main", "sha": "564ac0de1b228568cafcbfbe8ccaccc82d6023eb", "path": "recipes/models/IFM/K2-Horizon-375B-A23B.yaml", "edit_url": "https://github.com/vllm-project/vllm/edit/main/recipes/models/IFM/K2-Horizon-375B-A23B.yaml", "overlays": [] } }