meta-llama/Llama-3.2-1B-Instruct

Overview

This recipe groups the Llama 3.2 1B Instruct family under the canonical Meta parent. The Xeon 6 validation workbook has PASS rows for the Red Hat AI W8A8 and AMead10 AWQ variants.

The W8A8 checkpoint is detected from compressed-tensors metadata. The AWQ variant uses the recipe's --quantization awq selector.

Prerequisites

• Hardware: 1x Xeon6/Xeon5 NUMA node
• vLLM >= 0.6.2

pip (Intel Xeon 6 CPUs)

For Intel and AMD x86 CPUs, follow the CPU pre-built wheels installation instructions.

Docker (Intel Xeon 6 CPUs)

docker pull vllm/vllm-openai-cpu:latest-x86_64

Intel Xeon 6

TP/DP and host-specific CPU binding are intentionally not hard-coded.

Runtime and Platform Tuning

The following validation settings are not model requirements and are intentionally not portable CPU recipe defaults:

• --max-num-batched-tokens and --max-num-seqs: scheduler batch/concurrency tuning for workload shape, latency targets, and platform capacity.
• --gpu-memory-utilization: platform memory-budget tuning.
• --no-enable-prefix-caching: workload/benchmark cache-policy tuning.
• VLLM_ENGINE_ITERATION_TIMEOUT_S: runtime operational timeout tuning.

Hardware-specific overrides may still use these settings when they are part of a validated platform configuration.

References

• Meta parent model
• Red Hat AI W8A8 checkpoint
• AMead10 AWQ checkpoint