# meta-models/Muse-Glimmer-30B — vLLM serving recipe > Dense 29.6B vision-language model with a ViT-G/14 perception encoder and 128K context, distilled from Muse Spark for local agentic use. Emits channel-scoped reasoning and XML-style ATEM tool calls rather than JSON, so it needs the dedicated `muse_glimmer` tool-call and reasoning parsers. Configuration: hardware `dgx_spark_gb10` (1× DGX Spark (GB10)) · strategy `single_node_tp` (Tensor Parallel) · variant `default` (BF16) · features `tool_calling`, `reasoning` - Interactive page: https://recipes.vllm.ai/meta-models/Muse-Glimmer-30B - This configuration as JSON: https://recipes.vllm.ai/meta-models/Muse-Glimmer-30B.json?hardware=dgx_spark_gb10 The commands below were rendered for exactly this configuration and take precedence over any example in the guide. ## Other configurations Change the query string (the same parameters as the page URL) and fetch this URL again. Unknown or incompatible values fall back to the recipe default, so check the Configuration line. - `hardware`: `h100`, `h200`, `b200`, `gb200`, `b300`, `gb300`, `dgx_spark_gb10`, `rtx_5090`, `rtx_5090_2x`, `mi300x`, `mi325x`, `mi350x`, `mi355x`, `arc_pro_b60`, `arc_pro_b70` - `variant`: `default`, `fp8_block`, `nvfp4` - `strategy`: `single_node_tp` (multi-node strategies also need `nodes=`) - `features`: comma-separated list of `tool_calling`, `reasoning`, `spec_decoding` (default: `tool_calling`, `reasoning`) ## Install Requires vLLM >= 0.28.0. ```bash uv venv source .venv/bin/activate uv pip install -U vllm --torch-backend auto ``` Docker image: `vllm/vllm-openai:v0.28.0` ## Serve ```bash vllm serve meta-models/Muse-Glimmer-30B \ --generation-config auto \ --tensor-parallel-size 1 \ --enable-auto-tool-choice \ --tool-call-parser muse_glimmer \ --reasoning-parser muse_glimmer ``` Docker equivalent: ```bash docker run --gpus all \ --privileged --ipc=host -p 8000:8000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ vllm/vllm-openai:v0.28.0 meta-models/Muse-Glimmer-30B \ --generation-config auto \ --tensor-parallel-size 1 \ --enable-auto-tool-choice \ --tool-call-parser muse_glimmer \ --reasoning-parser muse_glimmer ``` ## Guide > Background reference. Commands in it are examples and may target another configuration — use the Serve section above. ### Overview Muse Glimmer 30B is a dense vision-language model built for agentic work on consumer hardware: a 52-layer text decoder (hidden 6656) plus a ~1.8B ViT-G/14 perception encoder, 128K trained context, BF16. Apache 2.0, knowledge cutoff January 4 2026, trained on 100+ languages. What makes it unlike most recipes here is its output format. It does not emit JSON tool calls and it does not wrap reasoning in `` tags. Every turn is written as a sequence of channel-scoped messages: ``` to=self<|message|>...chain of thought...<|eom|> <|start|>assistant to=<|message|> value <|eom|> <|start|>assistant to=user<|message|>...final answer...<|eot|> ``` Both `--tool-call-parser muse_glimmer` and `--reasoning-parser muse_glimmer` key off that framing, so run them together. The reasoning parser also forces `skip_special_tokens=False` — without it the markers are stripped before parsing and both channels collapse into content. ### Prerequisites - **Hardware**: DGX Spark. FP4 weights might also run on 5090. - **Image**: `vllm/vllm-openai:v0.28.0`. ### Checkpoints | variant | repo | size | notes | |---|---|---|---| | BF16 | [`meta-models/Muse-Glimmer-30B`](https://huggingface.co/meta-models/Muse-Glimmer-30B) | 59.58 GB | reference precision | | FP8 | [`RedHatAI/Muse-Glimmer-30B-FP8-block`](https://huggingface.co/RedHatAI/Muse-Glimmer-30B-FP8-block) | 32.78 GB | FP8 block-scaled weights, dynamic activations (vision tower + embeddings BF16) | | NVFP4 | [`Inferact/Muse-Glimmer-30B-NVFP4-W4A4`](https://huggingface.co/Inferact/Muse-Glimmer-30B-NVFP4-W4A4) | 25.42 GB | ModelOpt NVFP4, **W4A4**, group size 16 | | draft | [`meta-models/Muse-Glimmer-30B-assistant`](https://huggingface.co/meta-models/Muse-Glimmer-30B-assistant) | 5.11 GB | DFlash draft head for speculative decoding | The NVFP4 build quantizes **both weights and activations** to 4 bits across 364 language-model projections (52 layers x 7); embeddings, `lm_head` and the vision tower stay unquantized, which is why it is 25 GB rather than the ~15 GB a uniform 4-bit quant would give. vLLM detects it from `config.json` (`quant_method: modelopt`, `quant_algo: NVFP4`) and selects `modelopt_fp4` with fused activation quantization — no extra flags, the launch command below is unchanged apart from the model path. NVFP4 kernels are Blackwell-only. ### Launching the Server ```bash vllm serve /model \ --served-model-name muse-glimmer \ --tensor-parallel-size 4 \ --gpu-memory-utilization 0.92 \ --max-model-len 131072 \ --max-num-seqs 64 \ --enable-auto-tool-choice --tool-call-parser muse_glimmer \ --reasoning-parser muse_glimmer \ --generation-config auto ``` At TP=4 on GB300 this reports a 26.8M-token KV pool and ~204x max concurrency at the full 128K context. ### Running on DGX Spark (GB10) Muse Glimmer was built for local deployment and runs on a DGX Spark as an ordinary single-card deployment — **no special configuration**. GB10 is aarch64 like GB300 so the same image applies, and the launch command above is unchanged apart from having one GPU to run on: ```bash vllm serve /model \ --served-model-name muse-glimmer \ --tensor-parallel-size 1 \ --enable-auto-tool-choice --tool-call-parser muse_glimmer \ --reasoning-parser muse_glimmer \ --generation-config auto ``` There is no tensor parallelism to configure: GB10 is a single GPU, so TP is 1. BF16 weights are ~55 GiB against 128 GB of unified memory, leaving the rest for the KV cache and the perception encoder. The **NVFP4** checkpoint is 25.42 GB, which leaves considerably more of that shared pool for KV and the host — the better fit of the two on this hardware. ### Running on MI300X/MI325X/MI355X Use the official `vllm/vllm-openai-rocm:nightly` image. ```bash export VLLM_ROCM_USE_AITER=1 vllm serve /model \ --served-model-name muse-glimmer \ --tensor-parallel-size 4 \ --max-model-len 131072 \ --enable-auto-tool-choice \ --tool-call-parser muse_glimmer \ --reasoning-parser muse_glimmer \ --generation-config auto \ --attention-backend ROCM_AITER_FA ``` At TP=1 on MI300X this reports an 6.66M-token KV pool and ~50.82x max concurrency at the full 128K context. At TP=4 on MI300X this reports an 17.79M-token KV pool and ~135.73x max concurrency at the full 128K context. At TP=1 on MI355X this reports an 11.64M-token KV pool and ~88.86x max concurrency at the full 128K context. At TP=4 on MI355X this reports an 27.68M-token KV pool and ~211.22x max concurrency at the full 128K context. ### Running on Intel Arc Pro (XPU) and Xeon (CPU) validated on Intel Arc Pro B70, Arc Pro B60, and Xeon 6. **Intel GPU (XPU):** ```bash docker run -it --rm --network=host --ipc=host --privileged \ --device /dev/dri:/dev/dri \ -v /dev/dri/by-path:/dev/dri/by-path \ --entrypoint bash \ vllm/vllm-openai-xpu:latest ``` Serve inside the container — `--enforce-eager` is required on the XPU backend: ```bash vllm serve meta-models/Muse-Glimmer-30B \ --tensor-parallel-size 4 \ --reasoning-parser muse_glimmer \ --enforce-eager ``` ### Sampling Use the published settings: ``` temperature = 1.0 top_p = 0.95 top_k = 64 ``` Do not run it greedy — it is a reasoning model, and greedy is not reproducible here either: identical requests at `temperature: 0` with a fixed `seed` returned 70, 80, and 86 completion tokens across runs. **Reasoning strength.** Effort is set with a `Reasoning strength: ` line in the system prompt, one of `low` / `medium` / `high` / `xhigh`. Use `high` or `xhigh` for coding and agentic tasks. ### RTX 5090 (32 GB, consumer Blackwell) The `nvfp4` variant runs on a **single** RTX 5090 with no overrides: 131,072 context, a 179,647-token KV cache (1.37x concurrency at full length), 28.8 GB of 32.6 GB used, and 68.8 tok/s on a 600-token greedy generation. Tool calling and vision both work. `vram_minimum_gb: 31` is accurate — the margin is real but thin. **Speculative decoding needs a second card.** The draft head is 5.11 GB and the nvfp4 weights are 25.42 GB, so on one 32 GB card vLLM OOMs during init with roughly 400 MiB free. On 2x 5090 with `--tensor-parallel-size 2` it is well worth the second card — decode roughly triples. ### 2x RTX 5090 Verified with the drafter enabled, and worth the second card: decode goes from 68.8 tok/s to ~240 tok/s. `--gpu-memory-utilization` only sizes the KV cache; it does not gate initialisation. Measured across the range on the base recipe (131,072 ctx, bf16 KV) plus DFlash: | `--gpu-memory-utilization` | KV cache | concurrency @131K | |---|---|---| | 0.92 | 1,202,076 | 9.17x | | 0.88 | 1,054,407 | 8.04x | | 0.84 | 906,720 | 6.92x | | 0.80 | 759,051 | 5.79x | All seven steps from 0.92 down to 0.80 initialised cleanly, including the default 0.92. Decode was 212-261 tok/s with no trend attributable to utilisation — the spread is run-to-run noise (another model shared the host during the early steps) — so treat ~240 tok/s as the figure and pick utilisation purely for the cache size you want. If you *extend* beyond the recipe — a longer `--max-model-len` via rope scaling, or `--kv-cache-dtype fp8` — the arithmetic changes and 0.92 can OOM: at 262,144 with fp8 KV, vLLM sized a 2,594,229-token cache and then failed to allocate 396 MiB for the speculator. Lowering to 0.80 fixed it. That is a consequence of the extension, not of the recipe as written. Not tested at 2x: vision, and concurrent load at `--max-num-seqs 64`. ### fp8_block on sm120 needs DeepGEMM off The `fp8_block` variant does not load on RTX 5090 with DeepGEMM enabled. It fails in `process_weights_after_loading`, before the engine starts: ``` RuntimeError: Assertion error (deepgemm/csrc/apis/layout.hpp:60): Unknown SF transformation ``` vLLM treats capability family 120 as a supported DeepGEMM target, so the path is selected and then DeepGEMM rejects the scale-factor layout. Filed as vllm-project/vllm#51884. Set both to work around it: ```bash VLLM_USE_DEEP_GEMM=0 VLLM_MOE_USE_DEEP_GEMM=0 ``` With that, fp8_block serves fine on 2x 5090 and is the better choice than nvfp4 if you have the second card — 8-bit weights instead of W4A4, and faster: 177.6 tok/s with the drafter at 262,144 context (rope-extended, so outside the recipe as written). nvfp4 is unaffected and loads with DeepGEMM at its default. A note on `max_tokens`: the channel-scoped output means a tight budget can truncate before the final channel closes, returning empty `content` with `finish_reason: stop`. Give it room. ### Speculative decoding Muse Glimmer ships a DFlash block-diffusion draft head. Rather than proposing one token at a time, it predicts a whole block in a single forward — the slots attend to each other bidirectionally — and the target verifies them in parallel. DFlash uses 16 query positions per active request during warmup. To keep the expanded batch within vLLM's token budget, replace the earlier sequence limit with: ```bash --max-num-seqs 32 \ --speculative-config '{"method": "dflash", "model": "meta-models/Muse-Glimmer-30B-assistant", "num_speculative_tokens": 15}' ``` **`num_speculative_tokens: 15` is fixed, not tuned.** The head has `block_size: 16` and slot 0 re-presents the last accepted token, so 15 slots remain to predict. The head is 5 layers / 5.11 GB, reads the target's residual stream at layers `[1, 13, 25, 37, 49]`, and ties its embedding and output projection to the target — so it stores neither and is useless on its own. It must be paired with the specific target it was distilled against, not merely a compatible one.