meta-llama/Llama-3.2-1B

Overview

Llama 3.2 1B is Meta's compact pretrained dense language model with 128K context. This recipe includes an Intel Xeon 6 CPU configuration validated with BF16 serving.

Prerequisites

• Hardware: 1x Xeon 6 NUMA node
• vLLM >= 0.6.2

pip (Intel Xeon 6 CPUs)

For Intel and AMD x86 CPUs, follow the CPU pre-built wheels installation instructions.

Docker (Intel Xeon 6 CPUs)

docker pull vllm/vllm-openai-cpu:latest-x86_64

Runtime and Platform Tuning

The following validation settings are not model requirements and are intentionally not portable CPU recipe defaults:

• --max-num-batched-tokens and --max-num-seqs: scheduler batch/concurrency tuning for workload shape, latency targets, and platform capacity.
• --gpu-memory-utilization: platform memory-budget tuning.
• --no-enable-prefix-caching: workload/benchmark cache-policy tuning.
• VLLM_ENGINE_ITERATION_TIMEOUT_S: runtime operational timeout tuning.

Hardware-specific overrides may still use these settings when they are part of a validated platform configuration.

Intel Xeon 6

Choose the tensor parallel size based on the system topology and deployment requirements. Add --tensor-parallel-size <N> when needed; the recipe does not prescribe a Xeon 6 TP value or hard-code NUMA node IDs.

vllm serve meta-llama/Llama-3.2-1B

Docker:

docker run -itd --name llama32-1b-cpu \
  --network host \
  --shm-size 16g \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai-cpu:latest-x86_64 \
    --model meta-llama/Llama-3.2-1B \
    --host 0.0.0.0 \
    --port 8000

Client Usage

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="unused")
response = client.completions.create(
    model="meta-llama/Llama-3.2-1B",
    prompt="Explain tensor parallelism briefly.",
    max_tokens=128,
)
print(response.choices[0].text)

References

• Model card
• vLLM CPU installation