Google/gemma-4-26B-A4B-it

Google's Gemma 4 MoE multimodal model (26B total / 4B active) with 128 fine-grained experts, top-8 routing, thinking mode, and tool-use protocol.

MoE multimodal model — 26B total / 4B active, 128 experts with top-8 routing

View on HuggingFace

moe26B / 4B131,072 ctxvLLM 0.25.0+multimodaltext

Guide

Overview

Gemma 4 26B-A4B is the Mixture-of-Experts member of Google's Gemma 4 family — 26B total parameters with only 4B active per token via 128 fine-grained experts and top-8 routing. It supports text + images natively, structured thinking, function calling, and dynamic vision resolution.

Intel Xeon 6 CPU support remains verified. Benchmark-specific TP/DP, CPU binding, sequence limits, and repro-only timeouts remain deployment choices.

Key Features

MoE: 128 fine-grained experts with top-8 routing and custom GELU-activated FFN.
Multimodal: Text + images natively (video via custom frame-extraction pipeline). Audio is only supported on the smaller E2B/E4B variants.
Dual Attention: Alternating sliding-window (local) and global attention with different head dimensions.
Thinking Mode: Structured reasoning via <|channel>thought\n...<channel|> delimiters.
Function Calling: Custom tool-call protocol with dedicated special tokens.
Dynamic Vision Resolution: Per-request configurable vision token budget (70, 140, 280, 560, 1120 tokens).

TPU support is provided through vLLM TPU with recipes for Trillium and Ironwood.

Prerequisites

Hardware: Intel Xeon 6/Xeon 5 CPUs
vLLM >= 0.25.0

pip (NVIDIA CUDA)

uv venv
source .venv/bin/activate
uv pip install -U vllm --pre \
  --extra-index-url https://wheels.vllm.ai/nightly/cu129 \
  --extra-index-url https://download.pytorch.org/whl/cu129 \
  --index-strategy unsafe-best-match

pip (AMD ROCm: MI300X, MI325X, MI350X, MI355X)

Requires Python 3.12, ROCm 7.2.1, glibc >= 2.35 (Ubuntu 22.04+).

uv venv --python 3.12
source .venv/bin/activate
uv pip install vllm --pre \
  --extra-index-url https://wheels.vllm.ai/rocm/nightly/rocm721 --upgrade

pip (Intel Xeon 6 CPUs)

For Intel and AMD x86 CPUs, follow the CPU pre-built wheels installation instructions.

Docker

docker pull vllm/vllm-openai:gemma4-0505-cu129  # NVIDIA Hopper (H100/H200, CUDA 12.9)
docker pull vllm/vllm-openai:gemma4-0505-cu130  # NVIDIA Blackwell (B200/B300, CUDA 13.0)
docker pull vllm/vllm-openai-rocm:latest   # AMD
docker pull vllm/vllm-openai-cpu:latest-x86_64 # For Intel Xeon 6

TPU images are published separately by vllm-project/tpu-inference; see the Trillium / Ironwood tpu-recipes below for the pinned tag.

Deployment Configurations

26B MoE on 1x A100/H100 (BF16)

vllm serve google/gemma-4-26B-A4B-it \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.90

Full-Featured Server Launch

Enables text, image, thinking, and tool calling:

vllm serve google/gemma-4-26B-A4B-it \
  --max-model-len 16384 \
  --gpu-memory-utilization 0.90 \
  --enable-auto-tool-choice \
  --reasoning-parser gemma4 \
  --tool-call-parser gemma4 \
  --chat-template examples/tool_chat_template_gemma4.jinja \
  --limit-mm-per-prompt.image 4 \
  --async-scheduling \
  --host 0.0.0.0 \
  --port 8000

Docker (NVIDIA)

docker run -itd --name gemma4-moe \
  --ipc=host --network host --shm-size 16G --gpus all \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai:gemma4-0505-cu129 \
    --model google/gemma-4-26B-A4B-it \
    --max-model-len 32768 \
    --gpu-memory-utilization 0.90 \
    --host 0.0.0.0 --port 8000

Swap vllm/vllm-openai:gemma4-0505-cu129 for vllm/vllm-openai:gemma4-0505-cu130 on Blackwell (B200/B300).

Docker (AMD MI300X/MI325X/MI350X/MI355X)

docker run -itd --name gemma4-rocm \
  --ipc=host --network=host --privileged \
  --cap-add=CAP_SYS_ADMIN --device=/dev/kfd --device=/dev/dri \
  --group-add=video --cap-add=SYS_PTRACE \
  --security-opt=seccomp=unconfined --shm-size 16G \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai-rocm:latest \
    --model google/gemma-4-26B-A4B-it \
    --host 0.0.0.0 --port 8000

Docker (Cloud TPU — Trillium / Ironwood)

TPU uses the separate vllm/vllm-tpu image (no pip wheel). Pull the tag specified by the upstream Trillium or Ironwood recipe, then run:

docker run -itd --name gemma4-tpu \
  --privileged --network host --shm-size 16G \
  -v /dev/shm:/dev/shm -e HF_TOKEN=$HF_TOKEN \
  vllm/vllm-tpu:latest \
    --model google/gemma-4-26B-A4B-it \
    --tensor-parallel-size 8 \
    --max-model-len 16384 \
    --disable_chunked_mm_input \
    --host 0.0.0.0 --port 8000

Trillium requires a 4-chip slice minimum; Ironwood runs on a single chip.

Intel Xeon 6 Deployment via Docker

Launch the x86 CPU vLLM Docker container for google/gemma-4-26B-A4B-it:

docker run -itd --name gemma4-cpu \
  --network host \
  --shm-size 16g \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai-cpu:latest-x86_64 \
    --model google/gemma-4-26B-A4B-it \
    --host 0.0.0.0 \
    --port 8000

For additional Intel Xeon 6 deployment details, see the Intel Software Catalog entries for Gemma 4 26B-A4B IT.

The following settings are intentionally not prescribed as portable CPU model defaults because they depend on the workload, hardware, or runtime environment:

--max-num-batched-tokens: scheduler/throughput tuning.
--max-num-seqs: concurrency and scheduler-capacity tuning.
--gpu-memory-utilization: platform memory-budget tuning.
--no-enable-prefix-caching: workload/benchmark cache-policy tuning.
VLLM_ENGINE_ITERATION_TIMEOUT_S: operational runtime timeout.

These settings may still appear in validated hardware-specific overrides. Tune them at deployment time based on platform resources, workload shape, and latency/throughput goals.

Client Usage

Text Generation

from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
response = client.chat.completions.create(
    model="google/gemma-4-26B-A4B-it",
    messages=[{"role": "user", "content": "Write a poem about the ocean."}],
    max_tokens=512, temperature=0.7,
)
print(response.choices[0].message.content)

Image Understanding

response = client.chat.completions.create(
    model="google/gemma-4-26B-A4B-it",
    messages=[{"role": "user", "content": [
        {"type": "image_url", "image_url": {"url": "https://upload.wikimedia.org/wikipedia/commons/thumb/3/3a/Cat03.jpg/1200px-Cat03.jpg"}},
        {"type": "text", "text": "Describe this image in detail."},
    ]}],
    max_tokens=1024,
)

Thinking Mode

vllm serve google/gemma-4-26B-A4B-it \
  --max-model-len 16384 \
  --reasoning-parser gemma4 \
  --tool-call-parser gemma4 \
  --enable-auto-tool-choice \
  --chat-template examples/tool_chat_template_gemma4.jinja

Enable per-request via extra_body={"chat_template_kwargs": {"enable_thinking": True}}.

Dynamic Vision Resolution

Supported values: 70, 140, 280 (default), 560, 1120 tokens/image.

vllm serve google/gemma-4-26B-A4B-it \
  --mm-processor-kwargs '{"max_soft_tokens": 560}'

Configuration Tips

Set --max-model-len to match your workload.
--gpu-memory-utilization 0.90-0.95 maximizes KV cache.
Text-only workloads: --limit-mm-per-prompt '{"image": 0, "audio": 0}'.
--async-scheduling improves throughput.
FP8 KV cache (--kv-cache-dtype fp8) saves ~50% KV memory.
For MoE, TEP (tensor-expert parallelism) and DEP (data-expert parallelism) strategies scale better than pure TP at large node counts.

Quantized Variants

Three pre-quantized checkpoints are available:

RedHatAI/gemma-4-26B-A4B-it-FP8-dynamic — FP8 (E4M3) weights with dynamic per-token activation quantization; runs on Hopper and Blackwell.
RedHatAI/gemma-4-26B-A4B-it-NVFP4 — NVFP4 (4-bit) weights; requires Blackwell (B200/B300).
nvidia/Gemma-4-26B-A4B-NVFP4 — NVIDIA ModelOpt NVFP4 (4-bit) weights; this is the checkpoint validated on the single-GPU Blackwell desktops below.

Pick them from the Variant dropdown above, or pass the repo id directly to vllm serve.

NVFP4 on the Blackwell desktops

nvidia/Gemma-4-26B-A4B-NVFP4 has been run end to end on DGX Spark (GB10), DGX Station (GB300) and RTX PRO 6000, in each case on one GPU at TP1 with MTP drafting against google/gemma-4-26B-A4B-it-assistant.

Select the NVIDIA NVFP4 variant, enable Spec Decoding, and switch the hardware pill — memory utilization, batch size, prefill scheduling and the drafter's MoE backend all differ per box and are encoded per hardware, so the generated command is the tested one. Highlights:

DGX Spark (GB10) — 128 GB shared with the CPU, so --gpu-memory-utilization 0.8 and an 8-way batch. Serve from eugr/spark-vllm:nightly-20260704: vLLM 0.24 and the generic nightly both fail on this checkpoint, so the pip tab is hidden on this hardware.
DGX Station (GB300) — 252 GB on one GPU, so --gpu-memory-utilization 0.92 with chunked prefill and a 64-way batch.
RTX PRO 6000 — 96 GB, --gpu-memory-utilization 0.92 and an 8-way batch.

All three run the full 262,144-token context, above the 131,072 in config.json, and pass "moe_backend":"triton" in --speculative-config because the drafter's default fused MoE path is not built in these images.

The GB300 and RTX PRO 6000 runs used an internal NVIDIA nightly vLLM build. As noted under Speculative Decoding below, Gemma 4 MTP has not landed in a stable release, so the default vllm/vllm-openai:latest tag will not serve these commands with Spec Decoding enabled — use a nightly wheel or a nightly image on those two boxes.

Throughput vs Latency

Goal	TP	`--max-num-seqs`	Notes
Max throughput	1-2	256-512	Best tok/s per GPU
Min latency	4-8	8-16	Best TTFT/TPOT
Balanced	2	128	Mixed workloads

Speculative Decoding (MTP)

Enable the Spec Decoding feature toggle (above) or add --speculative-config manually to use MTP drafting with the assistant model. Recommended num_speculative_tokens: 4 for this model. See the Gemma 4 usage guide for details and benchmarks.

Note: MTP speculative decoding for Gemma 4 is only available on the vLLM nightly build — it has not yet landed in a stable release. Install via the nightly wheel (uv pip install -U vllm --pre --extra-index-url https://wheels.vllm.ai/nightly/cu129 …) or use the vllm/vllm-openai:gemma4-0505-cu129 / vllm/vllm-openai:gemma4-0505-cu130 images above; the standard :latest stable tag does not include this feature.