meta-models/Muse-Spark-Open-816B-A42B

Overview

Muse Spark Open is a sparse-MoE reasoning model — 816B total parameters, 42B active — with a 1M-token context window, aimed at long-horizon agentic coding. It is the model Muse Glimmer 30B was distilled from.

Architecture

62 layers, hidden size 8192, and every layer is a mixture-of-experts layer — there are no dense layers. Each holds 256 experts and uses 8 per token. The experts are 799B of the 816B; everything else is small.

The experts work narrow. A learned matrix squeezes each token from 8192 to 4096, all 256 experts run at that width, and a second matrix widens the result back. Both stay in full precision even when the experts are 4-bit. The router reads the token before the squeeze, and in fp32 — picking the wrong expert costs more than running a smaller one. There is no shared expert.

Attention is 128 query heads sharing 16 key-value heads, head dim 64, so the KV cache is eight times smaller than with one key-value head per query head.

Layers alternate between a narrow and a wide view, on a cycle of four:

sliding_window: 2048
layer_types: [sliding_attention, full_attention, sliding_attention, sliding_attention, ...]

Three layers in each cycle look back 2048 tokens; the other (layers 1, 5, …, 61) sees the whole context and drops rotary position (layer_rope_theta: 0), because rotary becomes unreliable over very long distances. Across 62 layers that is 46 narrow layers and 16 wide ones, and it is what makes a 1M-token context affordable — only the 16 wide layers hold KV for the whole sequence.

Normalization appears in more places than usual, each doing a job: a norm before and after each block (the ones after carry no learnable weight), on queries and keys before rotary position, on each attention head before its sigmoid gate, and on every expert's output before the weighted blend. The residual stream is carried in fp32.

Logits are scaled and then squashed through tanh (final_logit_softcapping: 20.0) before the softmax.

Prerequisites

• vLLM: Use Docker image for now until the code is in vLLM main repo.
• Hardware: the checkpoints hold ~1.64 TB of weights in BF16 and ~509 GB in Quark MXFP4, before KV cache. On AMD, all three variants are validated on gfx950 (MI350X/MI355X). NVFP4 is the primary NVIDIA Blackwell path.

AMD ROCm variants

Quark MXFP4 is the default AMD variant.

Variant	Checkpoint	AMD hardware	Experts / attention
MXFP4	amd/Muse-Spark-Open-816B-A42B-Quark-MXFP4	MI350X, MI355X	AITER MXFP4 MoE / AITER FA
BF16 → MXFP4	meta-models/Muse-Spark-Open-816B-A42B	MI350X, MI355X	experts of layers 1-60 rounded to MXFP4 at load, AITER MXFP4 MoE / AITER FA
BF16	meta-models/Muse-Spark-Open-816B-A42B	MI350X, MI355X	Triton BF16 MoE / AITER FA

In both MXFP4 variants the experts of the first and last layers stay BF16 and run on Triton BF16 MoE.

On AMD, the recipe's commands are text-only (--language-model-only), use the V2 model runner, disable prefix caching, enable chunked prefill, and request breakable FULL_AND_PIECEWISE graphs. The ROCm MoE path supports the Tensor Parallel and Tensor + Expert Parallel strategies; data-parallel strategies are not available on AMD.

Choosing TP or TP + expert parallel (MXFP4 variants)

Use plain TP for latency-bound serving (a handful of concurrent requests) and the Tensor + Expert Parallel strategy (TP8 + --enable-expert-parallel, 32 of 256 experts per GPU) for throughput at 32 or more concurrent requests, where it also lowers TTFT. Accuracy is the same under both. Keep TP for BF16: its Triton experts are slower with expert parallelism. With DSpark on, expert parallelism pays off from a few concurrent requests; see the DSpark section.

The AMD commands use --max-model-len 32768. For the full 1M-token context, the MXFP4 variant is validated on TP with --max-model-len 1048576 and --max-num-seqs 8, with either the default or --kv-cache-dtype fp8 KV cache.

Launching the Server

vllm serve meta-models/Muse-Spark-Open-816B-A42B-NVFP4 \
  --tensor-parallel-size 8 \
  --enable-expert-parallel \
  --max-model-len 1048576 \
  --reasoning-parser muse \
  --tool-call-parser muse \
  --enable-auto-tool-choice

Frontends

Commands use the Rust frontend by default on NVIDIA and AMD ROCm.

To use the Python frontend, select Python in the command builder or replace VLLM_USE_RUST_FRONTEND=1 with VLLM_USE_RUST_FRONTEND=0 in the generated command.

Reasoning effort

Set the strength per request with the top-level reasoning_effort field:

reasoning_effort	Strength hint
low	8
medium	32
high	256
xhigh	512 (default)

curl http://localhost:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "meta-models/Muse-Spark-Open-816B-A42B-NVFP4",
    "messages": [{"role": "user", "content": "Refactor this module."}],
    "reasoning_effort": "xhigh"
  }'

Multimodal

Muse Spark is a native multimodal model that understands image, audio and video. Video is processed through a series of sampled video frames at load time. On AMD, the recipe serves text only.

curl http://localhost:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "meta-models/Muse-Spark-Open-816B-A42B-NVFP4",
    "messages": [{"role": "user", "content": [
      {"type": "video_url", "video_url": {"url": "https://example.com/demo.mp4"}},
      {"type": "text", "text": "What happens, and when?"}
    ]}]
  }'

Video sampling rate can be changed when starting the server.

--media-io-kwargs '{"video": {"fps": 1}}'

Speculative decoding (DSpark)

Inferact trained a DSpark speculator for this model. The draft is a small sliding-window model that reads the target's residual stream at selected layers (target_layer_ids) and proposes a block per forward.

On AMD, selecting DSpark emits the fixed seven-token profile with AITER FA attention on every variant. Its causal multi-token decode is handled by AITER unified attention, with the gfx950 short-query 3D schedule enabled. DSpark gains the most when only a few requests are in flight, and still stays ahead of plain TP at 32 concurrent requests. It also runs under the Tensor + Expert Parallel strategy on the MXFP4 variants (TP8 + --enable-expert-parallel), which is the faster combination from a few concurrent requests up to about 64 at the same accuracy; plain TP + DSpark remains the lowest-latency choice for a single request. With expert parallelism, DSpark is validated up to --max-num-seqs 128; with plain TP, keep --max-num-seqs 32. For throughput beyond about 64 concurrent requests, use expert parallelism without DSpark.

Adaptive DSpark (manual opt-in, Triton attention)

Adaptive verification captures variable-length decode CUDA graphs, which the AITER FA attention backend does not support; with the AMD MXFP4 command above it fails at startup. To use it on MI350X/MI355X, switch the target to Triton attention and use the adaptive config:

export VLLM_ADAPTIVE_VERIFICATION_PROFILE_CONTEXT_LEN=8192

--attention-backend TRITON_ATTN \
--speculative-config '{"method":"dspark","model":"Inferact/Muse-Spark-Open-DSpark","num_speculative_tokens":7,"draft_sample_method":"probabilistic","rejection_sample_method":"standard","enable_adaptive_verification":true,"attention_backend":"TRITON_ATTN"}'

Seven tokens remain the maximum draft depth. At startup, adaptive verification profiles the DSpark and target costs, then uses draft confidence to keep only the slots whose expected accepted-token benefit justifies their verification cost. There is no hand-tuned acceptance threshold. Keep the profile context representative of production traffic.

References

• Muse Glimmer 30B recipe — the distilled 30B sibling