bharatgenai/Param2-17B-A2.4B-Thinking

BharatGen's hybrid-MoE reasoning model for English, Hindi and 21 Indian languages, with thinking traces and Hermes-style tool calling

17B total / 2.4B active parameters; the full BF16 checkpoint serves at TP1 on a single H100 or a single 48 GB L40S

View on HuggingFace

moe17B / 2.4B32,768 ctxvLLM 0.20.0+text

Guide

Overview

bharatgenai/Param2-17B-A2.4B-Thinking is BharatGen's early post-training checkpoint of Param-2-17B, a hybrid Mixture-of-Experts model with 17B total parameters and 2.4B activated per token. Each sparse block routes to 6 of 64 experts and additionally keeps 2 shared experts always active; the first layer is dense (first_k_dense_replace: 1) before the MoE stack begins. The shared-expert design is aimed at stable cross-lingual representations for Indian languages.

Beyond English and Hindi it covers 21 scheduled Indian languages — Assamese, Bengali, Bodo, Dogri, Gujarati, Kannada, Konkani, Kashmiri, Maithili, Malayalam, Manipuri, Marathi, Nepali, Oriya, Punjabi, Sanskrit, Santali, Sindhi, Tamil, Telugu and Urdu — and was pretrained on roughly 22.6T tokens.

The checkpoint is a thinking model: its chat template always opens an assistant turn with a <think> block, and tool calls are emitted as Hermes-style <tool_call>{"name": ..., "arguments": ...}</tool_call> JSON. That is why the two Features above map onto the qwen3 reasoning parser and the hermes tool-call parser.

Prerequisites

vLLM: 0.20.0 or newer. Param2MoEForCausalLM was registered upstream in PR #38000 and the tensor-parallel attention-head fix landed in PR #39707; both ship in v0.20.0. Nothing extra needs to be installed.
--trust-remote-code is required. vLLM implements the model natively (vllm/model_executor/models/param2moe.py), but model_type: param2moe is not registered with Transformers, so the config still loads from the repo's configuration_param2moe.py.
Hardware: one GPU is enough. The BF16 weights are ~34.3 GB, so an H100 80GB leaves ample room for the 32,768-token KV cache. A single 48 GB L40S also runs the same configuration: at the default --gpu-memory-utilization 0.9 roughly 8–9 GB remains for KV, and this model's KV cache is small (21 layers × 8 KV heads × 64 head dim ≈ 42 KB per token), so about 6 concurrent sequences fit at the full 32,768-token window — more if you cap --max-model-len.
License: the checkpoint is released under the BharatGen Non-Commercial license — review it before deploying.

Launching the Server

This is the exact configuration verified on both H100 and L40S:

vllm serve bharatgenai/Param2-17B-A2.4B-Thinking \
  --trust-remote-code \
  --async-scheduling \
  --tensor-parallel-size 1 \
  --enable-auto-tool-choice \
  --tool-call-parser hermes \
  --reasoning-parser qwen3

To pin the server to one specific GPU on a multi-GPU node, prefix the command with CUDA_VISIBLE_DEVICES=<id>; add --port <port> if 8000 is taken.

--async-scheduling overlaps scheduling with the model forward pass and cuts host overhead between decode steps. It is not required — drop it if you hit an incompatibility with another feature you enable.

TP1 is the recommended layout: the model fits on one GPU and a 2.4B active-parameter forward pass gains little from sharding. To raise aggregate throughput, run several independent TP1 replicas behind a load balancer rather than increasing TP — especially on L40S, which has no NVLink, so cross-GPU TP would fall back to PCIe.

Client Usage

Reasoning traces arrive in message.reasoning once --reasoning-parser qwen3 is active:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
response = client.chat.completions.create(
    model="bharatgenai/Param2-17B-A2.4B-Thinking",
    messages=[
        {"role": "user", "content": "एक कक्षा में 30 छात्र हैं। पूरी कक्षा का औसत 70 है। शीर्ष 10 का औसत 85 और निचले 10 का औसत 50 है। मध्य 10 छात्रों का औसत क्या है?"}
    ],
    temperature=0,
)
msg = response.choices[0].message
print(msg.reasoning)
print(msg.content)

The model card recommends deterministic decoding (temperature=0) for evaluations, structured tasks and production use; sampling defaults of temperature=0.7, top_p=0.9, top_k=50 are suggested for open-ended generation. The card also notes the model is strong at structured output (Markdown, tables, JSON), and that clear task/format/constraint instructions in the prompt matter noticeably.

Skipping the thinking block

The chat template accepts enable_thinking. Setting it to false prefills an empty <think></think> block so the model answers directly:

response = client.chat.completions.create(
    model="bharatgenai/Param2-17B-A2.4B-Thinking",
    messages=[{"role": "user", "content": "Name the capital of Maharashtra."}],
    extra_body={"chat_template_kwargs": {"enable_thinking": False}},
)

Tool calling

With the Tool Calling feature on, pass tools as usual and read message.tool_calls:

response = client.chat.completions.create(
    model="bharatgenai/Param2-17B-A2.4B-Thinking",
    messages=[{"role": "user", "content": "What is the weather in Pune?"}],
    tools=[{
        "type": "function",
        "function": {
            "name": "get_weather",
            "description": "Get the current weather for a city",
            "parameters": {
                "type": "object",
                "properties": {"city": {"type": "string"}},
                "required": ["city"],
            },
        },
    }],
    tool_choice="auto",
)
print(response.choices[0].message.tool_calls)

Benchmarking

vllm bench serve \
  --model bharatgenai/Param2-17B-A2.4B-Thinking \
  --dataset-name random \
  --random-input-len 2048 \
  --random-output-len 1024 \
  --num-prompts 200

Because thinking traces count toward the output budget, keep --random-output-len generous when benchmarking with reasoning enabled, or benchmark with enable_thinking: false to measure the non-reasoning path.

Notes and Limitations

Context: the native window is 32,768 tokens. The model card states a 128k-context variant is planned but not yet released, so do not raise --max-model-len past 32,768 on this checkpoint.
No MTP head: num_nextn_predict_layers: 0, so there is no speculative-decoding draft shipped with this checkpoint.
Early post-training checkpoint: BharatGen describes the release as an early post-training snapshot intended for downstream use and further fine-tuning.
Verification scope: the verified H100 and L40S badges cover the BF16 checkpoint at TP1 with the launch command above. Other GPUs are untested rather than known-bad.
L40S headroom: at 48 GB the weights alone take ~34.3 GB, so KV cache is the binding constraint rather than weight capacity. Lower --max-model-len if you need more concurrency than the full context window allows.