# meta-llama/Llama-4-Scout-17B-16E-Instruct — vLLM serving recipe

> Llama 4 Scout 17B-16E MoE model with NVIDIA FP8/FP4 variants, fits on a single GPU with quantization

Configuration: hardware `b200` (8× B200) · strategy `single_node_tp` (Tensor Parallel) · variant `default` (BF16)

- Interactive page: https://recipes.vllm.ai/meta-llama/Llama-4-Scout-17B-16E-Instruct
- This configuration as JSON: https://recipes.vllm.ai/meta-llama/Llama-4-Scout-17B-16E-Instruct.json?hardware=b200

The commands below were rendered for exactly this configuration and take precedence over any example in the guide.

## Other configurations

Change the query string (the same parameters as the page URL) and fetch this URL again. Unknown or incompatible values fall back to the recipe default, so check the Configuration line.

- `hardware`: `h100`, `b200`, `gb200`, `mi300x`, `mi325x`, `mi355x`, `xeon6`
- `variant`: `default`, `fp8`, `nvfp4`, `redhat_fp8`, `redhat_nvfp4`, `redhat_fp8_block`
- `strategy`: `single_node_tp`, `single_node_tep`, `single_node_dep`, `multi_node_tp`, `multi_node_tp_pp`, `multi_node_tep`, `multi_node_dep` (multi-node strategies also need `nodes=<n>`)

## Install

Requires vLLM >= 0.12.0.

```bash
uv venv
source .venv/bin/activate
uv pip install -U vllm --torch-backend auto
```

Docker image: `vllm/vllm-openai:latest`

## Serve

```bash
vllm serve meta-llama/Llama-4-Scout-17B-16E-Instruct \
  --tensor-parallel-size 8 \
  --async-scheduling \
  --no-enable-prefix-caching \
  --max-num-batched-tokens 8192 \
  --compilation-config '{"pass_config":{"fuse_allreduce_rms":true,"eliminate_noops":true}}'
```

Docker equivalent:

```bash
docker run --gpus all \
  --privileged --ipc=host -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai:latest meta-llama/Llama-4-Scout-17B-16E-Instruct \
  --tensor-parallel-size 8 \
  --async-scheduling \
  --no-enable-prefix-caching \
  --max-num-batched-tokens 8192 \
  --compilation-config '{"pass_config":{"fuse_allreduce_rms":true,"eliminate_noops":true}}'
```

## Guide

> Background reference. Commands in it are examples and may target another configuration — use the Serve section above.

### Overview

Llama 4 Scout is Meta's MoE model with 17B active parameters across 16 experts
(109B total). NVIDIA provides FP8 and FP4 quantized variants. With FP4 quantization,
the model fits on a single B200 GPU — making it one of the most accessible MoE models.

### Prerequisites

- Hardware: 1x B200 (FP4), 1x H100 (FP8), 4x GPUs (BF16), 8x MI300X/MI325X/MI355X, or 4x Xeon6/Xeon5 NUMA nodes
- vLLM >= 0.12.0
- CUDA Driver >= 575 for GPUs
- Docker with NVIDIA Container Toolkit (recommended) for GPUs
- License: Must agree to Meta's Llama 4 Scout Community License for GPUs

#### AMD ROCm

The ROCm wheel requires Python 3.12, ROCm 7.0+, and glibc >= 2.35. Use the
ROCm Docker image when your host environment does not meet those requirements.

```bash
uv venv --python 3.12
source .venv/bin/activate
uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/
```

Launch on 8x MI300X / MI325X / MI355X with AITER enabled:

```bash
export VLLM_ROCM_USE_AITER=1
vllm serve meta-llama/Llama-4-Scout-17B-16E-Instruct \
  --tensor-parallel-size 8 \
  --max-num-seqs 64 \
  --no-enable-prefix-caching \
  --max-num-batched-tokens 16384 \
  --max-model-len 32000
```

The first AITER launch may take several minutes while ROCm kernels are compiled and cached.

#### pip (Intel Xeon 6 CPUs)
For Intel and AMD x86 CPUs, follow the [CPU pre-built wheels](https://docs.vllm.ai/en/latest/getting_started/installation/cpu/#pre-built-wheels) installation instructions.

#### Docker (Intel Xeon 6 CPUs)

```bash
docker pull vllm/vllm-openai-cpu:latest-x86_64 # For Intel Xeon 6
```

#### Intel Xeon 6 Deployment via Docker

Launch the x86 CPU vLLM Docker container for `meta-llama/Llama-4-Scout-17B-16E-Instruct`:

```bash
docker run \
  --privileged --ipc=host -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai-cpu:latest-x86_64 meta-llama/Llama-4-Scout-17B-16E-Instruct \
  --tensor-parallel-size 4 \
  --max-model-len 8192
```

`--max-model-len 8192` is retained because it is the model-specific row argument
in the validation workbook. TP/DP and host-specific CPU placement are intentionally
not prescribed.

The following validation settings are not model requirements and are intentionally
not portable CPU recipe defaults:

- `--max-num-batched-tokens` and `--max-num-seqs`: scheduler batch/concurrency
  tuning for workload shape, latency targets, and platform capacity.
- `--gpu-memory-utilization`: platform memory-budget tuning.
- `--no-enable-prefix-caching`: workload/benchmark cache-policy tuning.
- `VLLM_ENGINE_ITERATION_TIMEOUT_S`: runtime operational timeout tuning.

Hardware-specific overrides may still use these settings when they are part of a
validated platform configuration.

### Client Usage

```python
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="unused")
response = client.chat.completions.create(
    model="nvidia/Llama-4-Scout-17B-16E-Instruct-FP8",
    messages=[{"role": "user", "content": "Explain MoE models briefly."}],
)
print(response.choices[0].message.content)
```

### Troubleshooting

**FP4 only works on Blackwell:**
FP4 quantization requires compute capability 10.0 (B200/GB200). Use FP8 on Hopper.

**TP=1 recommended for best throughput:**
For maximum throughput per GPU, keep TP=1. Increase TP to 2/4/8 for lower latency.

### References

- [Llama-4-Scout-17B-16E-Instruct](https://huggingface.co/meta-llama/Llama-4-Scout-17B-16E-Instruct)
- [NVIDIA FP8 variant](https://huggingface.co/nvidia/Llama-4-Scout-17B-16E-Instruct-FP8)
- [NVIDIA FP4 variant](https://huggingface.co/nvidia/Llama-4-Scout-17B-16E-Instruct-FP4)