mindlab-research/Macaron-V1-Coding-Venti

Overview

Macaron-V1-Coding-Venti is MindLab Research's coding-specialist checkpoint in the Macaron-V1 family. It is produced by merging the Macaron-V1-Venti L2 Coding LoRA specialist into the GLM-5.2 BF16 base, yielding a single merged checkpoint — no PEFT adapter or Mixture-of-LoRA (MoL) router is needed at runtime. It targets code understanding, repository-level software engineering, terminal use, and coding-agent workflows, and reports strong coding/terminal slices (SWE Verified 85.6, TerminalBench 2.1 87.6, DeepSWE 58.4) on the Macaron evaluation suite.

Because it is a weight-level merge of a LoRA onto GLM-5.2, the runtime architecture is identical to GLM-5.2 — same GlmMoeDsaForCausalLM MoE (~743B total / 39B active), same 1M native context, same MTP indexer. Launch it with the same serving stack as GLM-5.2 recipe.

This model ships BF16 only (no native FP8 / NVFP4 checkpoint). At full BF16 precision (≈1786 GB) it needs multi-node; the FP8 (on-the-fly) variant loads the same weights as FP8 (--quantization fp8, ≈893 GB) to fit a single 8×H200 / 8×B200 node — or to free VRAM on larger trays.

Prerequisites

• vLLM 0.25.0 (stable) — the GlmMoeDsaForCausalLM handler is the same one GLM-5.2 uses.
• GPU: at full BF16 precision (≈1786 GB) a single 8×H200 / 8×B200 node does not fit — run multi-node (e.g. 2× 8-GPU nodes, tensor-parallel 16), or pick the FP8 (on-the-fly) variant (≈893 GB, --quantization fp8) to run single-node. The native FP8/NVFP4 GLM-5.2 checkpoints are a calibrated-quant alternative.

Installation

Docker

The BF16 variant needs two nodes (the FP8 (on-the-fly) variant fits one — see Launching below). Run one container per node (each --gpus all, 8 local GPUs); both pass --tensor-parallel-size 16 --nnodes 2, varying --node-rank. This is the head node — node 1 repeats with --node-rank 1 --headless.

docker run --gpus all \
  -p 8000:8000 \
  --ipc=host \
  --network host \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai:v0.25.0 mindlab-research/Macaron-V1-Coding-Venti \
    --tensor-parallel-size 16 \
    --nnodes 2 \
    --node-rank 0 \
    --master-addr $HEAD_IP \
    --kv-cache-dtype fp8 \
    --tool-call-parser glm47 \
    --reasoning-parser glm45 \
    --enable-auto-tool-choice \
    --served-model-name macaron-v1-coding-venti

From source

uv venv
source .venv/bin/activate
uv pip install "vllm==0.25.0" --torch-backend=auto
uv pip install "transformers>=5.9.0"

Launching the server

Pick the Variant that matches your hardware: BF16 for full precision on multi-node, or FP8 (on-the-fly) to fit a single node (or free VRAM on a larger tray). The builder renders the exact flags per your Variant / Strategy / Nodes picks.

BF16 — 2× 8xH200 (multi-node TP)

Full-precision BF16 (≈1786 GB) needs multi-node. The command below is a 2-node tensor-parallel example (16 GPUs). Run the head command on node 0 and the worker on node 1, pointing --master-addr at the head's IP. Pick the Multi-node TP strategy in the builder to render both tabs.

# Node 0 (head)
vllm serve mindlab-research/Macaron-V1-Coding-Venti \
  --kv-cache-dtype fp8 \
  --tensor-parallel-size 16 \
  --tool-call-parser glm47 \
  --reasoning-parser glm45 \
  --enable-auto-tool-choice \
  --served-model-name macaron-v1-coding-venti

# Node 1
vllm serve mindlab-research/Macaron-V1-Coding-Venti \
  --kv-cache-dtype fp8 \
  --tensor-parallel-size 16 \
  --tool-call-parser glm47 \
  --reasoning-parser glm45 \
  --enable-auto-tool-choice \
  --served-model-name macaron-v1-coding-venti

The builder emits the exact multi-node multiprocessing args (--nnodes, --node-rank, --master-addr for TP) when you select Multi-node TP.

FP8 (on-the-fly) on a single node

Pick the FP8 (on-the-fly) variant to load the same BF16 checkpoint as FP8 via --quantization fp8 (runtime quantization — not a native FP8 weight set). At ≈893 GB it fits a single 8×H200 / 8×B200 node, and on larger trays (e.g. B300) it frees VRAM for longer context or more concurrent sequences. Startup is slower while the quantization pass runs at load; the accuracy / KV-cache trade-off is small (see the B300 numbers below).

vllm serve mindlab-research/Macaron-V1-Coding-Venti \
  --quantization fp8 \
  --tensor-parallel-size 8 \
  --kv-cache-dtype fp8 \
  --tool-call-parser glm47 \
  --reasoning-parser glm45 \
  --enable-auto-tool-choice \
  --served-model-name macaron-v1-coding-venti

• --max-model-len — raise toward the 1M window when KV-cache headroom allows; lower on OOM.
• --max-num-seqs 32 — the main knob for fitting long context under concurrency.
• --gpu-memory-utilization 0.80 — leaves ROCm runtime headroom for MTP graph capture.

On a B300 node, on-the-fly FP8 trades a negligible accuracy drop for a large KV-cache headroom gain:

Variant	GSM8K 5-shot (exact-match)	B300 GPU KV-cache size
BF16	0.9431 (±0.0064)	≈1.12M
FP8 (on-the-fly)	0.9424 (±0.0064)	≈3.07M

The GSM8K scores sit within stderr of each other, while FP8 leaves ~2.7× the KV-cache space — so on B300 (where BF16 already fits) FP8 buys far more concurrency / context at almost no accuracy cost.

Speculative decoding (MTP)

The merged weights inherit GLM-5.2's built-in Multi-Token Prediction head, so MTP works exactly as on GLM-5.2 — on both the BF16 and FP8 (on-the-fly) variants. Toggle spec_decoding → MTP in the builder:

vllm serve mindlab-research/Macaron-V1-Coding-Venti \
  --kv-cache-dtype fp8 \
  --tensor-parallel-size 16 \
  --speculative-config '{"method":"mtp","num_speculative_tokens":5}' \
  --tool-call-parser glm47 \
  --reasoning-parser glm45 \
  --enable-auto-tool-choice \
  --served-model-name macaron-v1-coding-venti

AMD MI300X / MI355X

Use the AITER backends under ROCm, same as GLM-5.2. Both the BF16 (multi-node) and FP8 (on-the-fly) variants run; whether the full 1M window fits is a KV-cache VRAM question, so tune --max-model-len and --max-num-seqs to your nodes' HBM.

VLLM_ROCM_USE_AITER=1 \
VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS=1 \
vllm serve mindlab-research/Macaron-V1-Coding-Venti \
  --kv-cache-dtype fp8_e4m3 \
  --tensor-parallel-size 16 \
  --gpu-memory-utilization 0.80 \
  --max-model-len 524288 \
  --max-num-seqs 32 \
  --linear-backend aiter \
  --moe-backend aiter \
  --tool-call-parser glm47 \
  --enable-auto-tool-choice \
  --reasoning-parser glm45

• --max-model-len — raise toward the 1M window when KV-cache headroom allows; lower on OOM.
• --max-num-seqs 32 — the main knob for fitting long context under concurrency.
• --gpu-memory-utilization 0.80 — leaves ROCm runtime headroom for MTP graph capture.

Reasoning modes

Thinking is on by default and reuses the GLM/DeepSeek-V4 reasoning_effort mechanism, identical to GLM-5.2. The chat template resolves effort to max unless reasoning_effort is explicitly "high".

Mode	How to request	Behavior
Think Max (default)	omit reasoning_effort, or set "max"	Deepest reasoning — hard math, multi-step planning, agentic tasks.
Think High	"reasoning_effort": "high"	Balanced depth and latency.
Non-think	chat_template_kwargs.enable_thinking: false	Fast, no chain-of-thought.

Client usage

from openai import OpenAI

client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1")
msgs = [{"role": "user", "content": "Refactor this function to handle empty input."}]

# Think Max (default) — just omit reasoning_effort
client.chat.completions.create(model="macaron-v1-coding-venti", messages=msgs, max_tokens=4096)

# Think High
client.chat.completions.create(
    model="macaron-v1-coding-venti",
    messages=msgs,
    max_tokens=4096,
    extra_body={"chat_template_kwargs": {"reasoning_effort": "high"}},
)

# Non-think
client.chat.completions.create(
    model="macaron-v1-coding-venti",
    messages=msgs,
    max_tokens=4096,
    extra_body={"chat_template_kwargs": {"enable_thinking": False}},
)

Benchmarking

Add --no-enable-prefix-caching to the server command for a clean measurement.

vllm bench serve \
  --model mindlab-research/Macaron-V1-Coding-Venti \
  --dataset-name random \
  --random-input 8000 \
  --random-output 1024 \
  --request-rate 10 \
  --num-prompts 32 \
  --ignore-eos

Note: pure throughput benchmarks tend to under-report real speed, because MTP's acceptance rate is usually low in synthetic runs.

References

• Model card
• Base model: GLM-5.2 recipe
• Introducing Macaron-V1 (blog)
• Macaron Artifacts (GitHub)