meta-llama/Llama-3.1-8B

Overview

Llama 3.1 8B is Meta's dense pretrained base model with 128K context. This recipe is separate from Llama-3.1-8B-Instruct because the Red Hat AI Meta-Llama-3.1-8B-quantized.w8a8 checkpoint is quantized from the base model, not the instruction-tuned checkpoint.

The recipe retains portable model-level CPU settings while leaving TP/DP and host-specific CPU placement as deployment choices.

Prerequisites

• Hardware: 2x Xeon 6 NUMA nodes
• vLLM >= 0.6.0

pip (Intel Xeon 6 CPUs)

For Intel and AMD x86 CPUs, follow the CPU pre-built wheels installation instructions.

Docker (Intel Xeon 6 CPUs)

docker pull vllm/vllm-openai-cpu:latest-x86_64

Intel Xeon 6

vllm serve RedHatAI/Meta-Llama-3.1-8B-quantized.w8a8

Docker:

docker run -itd --name llama31-8b-w8a8-cpu \
  --network host \
  --shm-size 16g \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai-cpu:latest-x86_64 \
    --model RedHatAI/Meta-Llama-3.1-8B-quantized.w8a8 \
    --host 0.0.0.0 \
    --port 8000

Runtime and Platform Tuning

The following validation settings are not model requirements and are intentionally not portable CPU recipe defaults:

• --max-num-batched-tokens and --max-num-seqs: scheduler batch/concurrency tuning for workload shape, latency targets, and platform capacity.
• --gpu-memory-utilization: platform memory-budget tuning.
• --no-enable-prefix-caching: workload/benchmark cache-policy tuning.
• VLLM_ENGINE_ITERATION_TIMEOUT_S: runtime operational timeout tuning.

Hardware-specific overrides may still use these settings when they are part of a validated platform configuration.

Configuration Notes

• The validation TP/DP layout is benchmark topology and is not hard-coded.
• Llama has native vLLM architecture support, so --trust-remote-code is not required by this recipe.

References

• Parent model
• Red Hat AI W8A8 variant
• Llama-3.1-8B-Instruct recipe
• vLLM CPU installation