Qwen/Qwen3-8B

Overview

Qwen3-8B is a dense Qwen3 model with hybrid thinking and non-thinking modes. This recipe includes an Intel Xeon 6 CPU configuration validated with BF16 serving.

Prerequisites

• Hardware: 2x Xeon 6 CPUs
• vLLM >= 0.8.5

pip (Intel Xeon 6 CPUs)

For Intel and AMD x86 CPUs, follow the CPU pre-built wheels installation instructions.

Docker (Intel Xeon 6 CPUs)

docker pull vllm/vllm-openai-cpu:latest-x86_64

Intel Xeon 6

Choose the tensor parallel size based on the system topology and deployment requirements. Add --tensor-parallel-size <N> when needed; the recipe does not prescribe a Xeon 6 TP value or hard-code NUMA node IDs.

vllm serve Qwen/Qwen3-8B \
  --max-num-batched-tokens 16384 \
  --gpu-memory-utilization 0.8

Docker:

docker run -itd --name qwen3-8b-cpu \
  --network host \
  --shm-size 16g \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai-cpu:latest-x86_64 \
    --model Qwen/Qwen3-8B \
    --max-num-batched-tokens 16384 \
    --gpu-memory-utilization 0.8 \
    --host 0.0.0.0 \
    --port 8000

Client Usage

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="unused")
response = client.chat.completions.create(
    model="Qwen/Qwen3-8B",
    messages=[{"role": "user", "content": "Explain tensor parallelism briefly."}],
)
print(response.choices[0].message.content)

References

• Model card
• vLLM CPU installation