{ "hf_id": "arcee-ai/Trinity-Large-Thinking", "meta": { "title": "Trinity-Large-Thinking", "slug": "trinity-large-thinking", "provider": "Arcee AI", "description": "Arcee AI's reasoning-focused sparse MoE (AfmoeForCausalLM) with structured traces and agentic tool use", "date_added": "2026-04-10", "date_updated": "2026-04-17", "tasks": [ "text" ], "related_recipes": [] }, "recommended_command": { "hardware": "h200", "strategy": "single_node_tp", "variant": "default", "node_count": 1, "deploy_type": "single_node", "env": {}, "docker_image": "vllm/vllm-openai:latest", "command": "vllm serve arcee-ai/Trinity-Large-Thinking \\\n --dtype bfloat16 \\\n --tensor-parallel-size 8 \\\n --enable-auto-tool-choice \\\n --tool-call-parser qwen3_coder \\\n --reasoning-parser deepseek_r1", "argv": [ "vllm", "serve", "arcee-ai/Trinity-Large-Thinking", "--dtype", "bfloat16", "--tensor-parallel-size", "8", "--enable-auto-tool-choice", "--tool-call-parser", "qwen3_coder", "--reasoning-parser", "deepseek_r1" ], "docker_command": "docker run --gpus all \\\n --privileged --ipc=host -p 8000:8000 \\\n -v ~/.cache/huggingface:/root/.cache/huggingface \\\n vllm/vllm-openai:latest arcee-ai/Trinity-Large-Thinking \\\n --dtype bfloat16 \\\n --tensor-parallel-size 8 \\\n --enable-auto-tool-choice \\\n --tool-call-parser qwen3_coder \\\n --reasoning-parser deepseek_r1", "docker_argv": [ "docker", "run", "--gpus", "all", "--privileged", "--ipc=host", "-p", "8000:8000", "-v", "~/.cache/huggingface:/root/.cache/huggingface", "vllm/vllm-openai:latest", "arcee-ai/Trinity-Large-Thinking", "--dtype", "bfloat16", "--tensor-parallel-size", "8", "--enable-auto-tool-choice", "--tool-call-parser", "qwen3_coder", "--reasoning-parser", "deepseek_r1" ], "strategy_spec": { "name": "single_node_tp", "deploy_type": "single_node", "display_name": "Tensor Parallel", "short_label": "TP", "orientation": "latency", "description": "Single-node tensor parallel. Splits the model across all local GPUs. TP size is set to the GPU count at deploy time. The simplest multi-GPU strategy โ€” works for all model architectures.\n", "hardware_match": { "min_gpus": 1, "max_gpus": 8, "multi_node": false }, "vllm_args": [], "parallel_flag": "--tensor-parallel-size" }, "hardware_profile": { "brand": "NVIDIA", "generation": "hopper", "compute_arch": "sm90", "display_name": "H200", "description": "NVIDIA H200 SXM 141 GB HBM3e ยท 8-GPU HGX node", "gpu_count": 8, "vram_gb": 1128, "multi_node": false }, "alternatives": { "multi_node_tp": "/arcee-ai/Trinity-Large-Thinking/strategies/multi_node_tp.json", "multi_node_tp_pp": "/arcee-ai/Trinity-Large-Thinking/strategies/multi_node_tp_pp.json", "multi_node_tep": "/arcee-ai/Trinity-Large-Thinking/strategies/multi_node_tep.json", "multi_node_dep": "/arcee-ai/Trinity-Large-Thinking/strategies/multi_node_dep.json", "kv_store_centralized_mooncake": "/arcee-ai/Trinity-Large-Thinking/strategies/kv_store_centralized_mooncake.json", "kv_store_distributed_mooncake": "/arcee-ai/Trinity-Large-Thinking/strategies/kv_store_distributed_mooncake.json" }, "by_hardware": { "h200": "/arcee-ai/Trinity-Large-Thinking/hw/h200.json", "h100": "/arcee-ai/Trinity-Large-Thinking/hw/h100.json", "b200": "/arcee-ai/Trinity-Large-Thinking/hw/b200.json", "gb200": "/arcee-ai/Trinity-Large-Thinking/hw/gb200.json", "b300": "/arcee-ai/Trinity-Large-Thinking/hw/b300.json", "gb300": "/arcee-ai/Trinity-Large-Thinking/hw/gb300.json", "mi300x": "/arcee-ai/Trinity-Large-Thinking/hw/mi300x.json", "mi325x": "/arcee-ai/Trinity-Large-Thinking/hw/mi325x.json", "mi350x": "/arcee-ai/Trinity-Large-Thinking/hw/mi350x.json", "mi355x": "/arcee-ai/Trinity-Large-Thinking/hw/mi355x.json" } }, "model": { "model_id": "arcee-ai/Trinity-Large-Thinking", "min_vllm_version": "0.11.1", "architecture": "moe", "parameter_count": "398B", "active_parameters": "13B", "context_length": 262144, "base_args": [ "--dtype", "bfloat16" ], "base_env": {}, "install": { "pip": { "command": "uv venv\nsource .venv/bin/activate\nuv pip install -U vllm --torch-backend auto" }, "docker": { "command": "docker pull vllm/vllm-openai:latest" } } }, "features": { "tool_calling": { "description": "Qwen3 Coder tool-call parser with automatic tool choice", "args": [ "--enable-auto-tool-choice", "--tool-call-parser", "qwen3_coder" ] }, "reasoning": { "description": "DeepSeek-R1 reasoning parser extracts ... into message.reasoning", "args": [ "--reasoning-parser", "deepseek_r1" ] } }, "opt_in_features": [], "variants": { "default": { "precision": "bf16", "vram_minimum_gb": 955, "description": "Full precision BF16 on multi-GPU (sparse MoE โ€” multi-GPU recommended)" }, "nvfp4": { "model_id": "arcee-ai/Trinity-Large-Thinking-NVFP4", "precision": "nvfp4", "vram_minimum_gb": 239, "description": "NVIDIA NVFP4 quantized weights for Blackwell GPUs", "extra_args": [ "--kv-cache-dtype", "fp8" ], "json": "/arcee-ai/Trinity-Large-Thinking-NVFP4.json" }, "redhat_nvfp4": { "model_id": "RedHatAI/Trinity-Large-Thinking-NVFP4", "precision": "nvfp4", "label": "NVFP4 (RedHat)", "vram_minimum_gb": 254, "description": "RedHatAI NVFP4 quantization", "json": "/RedHatAI/Trinity-Large-Thinking-NVFP4.json" } }, "hardware_overrides": {}, "guide": "## Overview\n\n[Trinity-Large-Thinking](https://huggingface.co/arcee-ai/Trinity-Large-Thinking) is\nArcee AI's reasoning-focused Trinity Large checkpoint โ€” a sparse MoE model designed for\nlong-horizon planning, tool use, and multi-step agent workflows. It uses the\n`AfmoeForCausalLM` architecture and emits explicit reasoning traces inside `...`\nblocks.\n\nFor multi-turn chat and agentic loops, reasoning tokens should be preserved across\nturns as part of the working state.\n\n## Prerequisites\n\n- vLLM >= 0.11.1\n- Hardware: multi-GPU recommended for production deployments\n\n### Install vLLM\n\n```bash\nuv venv\nsource .venv/bin/activate\nuv pip install -U vllm openai --torch-backend auto\n```\n\n## Launch command\n\n```bash\nvllm serve arcee-ai/Trinity-Large-Thinking \\\n --dtype bfloat16 \\\n --reasoning-parser deepseek_r1 \\\n --enable-auto-tool-choice \\\n --tool-call-parser qwen3_coder\n```\n\nWhy these flags:\n- `--reasoning-parser deepseek_r1` extracts `...` into `message.reasoning`.\n- `--enable-auto-tool-choice` lets the model decide when to call tools.\n- `--tool-call-parser qwen3_coder` converts tool calls into OpenAI-style `tool_calls`.\n- `--dtype bfloat16` matches the recommended serving dtype.\n\nAdd parallelism flags (`--tensor-parallel-size`, `--data-parallel-size`, or\n`--enable-expert-parallel`) for your hardware. Lower `--max-model-len` if you don't\nneed the full long-context config.\n\n## Validation Request\n\n```python\nfrom openai import OpenAI\n\nclient = OpenAI(api_key=\"EMPTY\", base_url=\"http://localhost:8000/v1\")\nmodel = client.models.list().data[0].id\n\ntools = [{\n \"type\": \"function\",\n \"function\": {\n \"name\": \"get_weather\",\n \"description\": \"Get the current weather for a location.\",\n \"parameters\": {\n \"type\": \"object\",\n \"properties\": {\"location\": {\"type\": \"string\"}},\n \"required\": [\"location\"],\n },\n },\n}]\n\nresponse = client.chat.completions.create(\n model=model,\n messages=[{\"role\": \"user\", \"content\": \"What is the weather in Paris right now?\"}],\n tools=tools, tool_choice=\"auto\",\n)\n\nmsg = response.choices[0].message\nreasoning = msg.reasoning\nprint(\"reasoning:\", reasoning)\nprint(\"content:\", msg.content)\nprint(\"tool_calls:\", msg.tool_calls)\n```\n\n## Preserving Reasoning Across Turns\n\nPass reasoning back as `reasoning` on assistant messages:\n\n```python\nassistant_msg = {\"role\": \"assistant\", \"content\": msg.content or \"\"}\nif reasoning:\n assistant_msg[\"reasoning\"] = reasoning\nif msg.tool_calls:\n assistant_msg[\"tool_calls\"] = [\n {\"id\": tc.id, \"type\": \"function\",\n \"function\": {\"name\": tc.function.name, \"arguments\": tc.function.arguments}}\n for tc in msg.tool_calls\n ]\nmessages.append(assistant_msg)\n```\n\nRules:\n- Pass reasoning back as `reasoning`.\n- Keep `content` as an empty string (not `null`) on tool-only turns.\n- Append the assistant message before tool-result messages.\n- Use `/v1/chat/completions` for structured reasoning output.\n\n## Troubleshooting\n\n- **No reasoning:** start server with `--reasoning-parser deepseek_r1`; use `/v1/chat/completions`.\n- **Tool calls as plain text:** enable `--enable-auto-tool-choice` and `--tool-call-parser qwen3_coder`.\n- **Loses coherence after tool turns:** preserve `reasoning` on each assistant turn; don't set content to `null`.\n- **OOM:** lower `--max-model-len`; scale parallelism; use a local checkpoint path.\n\n## References\n\n- [Trinity-Large-Thinking on Hugging Face](https://huggingface.co/arcee-ai/Trinity-Large-Thinking)\n", "source": { "source": "vllm", "repo": "vllm-project/vllm", "ref": "main", "sha": "564ac0de1b228568cafcbfbe8ccaccc82d6023eb", "path": "recipes/models/arcee-ai/Trinity-Large-Thinking.yaml", "edit_url": "https://github.com/vllm-project/vllm/edit/main/recipes/models/arcee-ai/Trinity-Large-Thinking.yaml", "overlays": [] } }