Qwen/Qwen2.5-VL-7B-Instruct

Qwen2.5-VL dense vision-language model (7B) for image and video understanding — fits on a single TPU v6e chip, one GPU, or Intel Xeon 6 CPU.

Verified on TPU v6e (Trillium) and Intel Xeon 6 CPU with BF16

View on HuggingFace

dense7B128,000 ctxvLLM 0.7.0+multimodaltext

Guide

Overview

Qwen2.5-VL-7B-Instruct is the small dense vision-language model in the Qwen2.5-VL series. At 7B it fits comfortably on a single accelerator — one TPU v6e (Trillium) chip, or a single 24GB+ GPU — and is also validated for Intel Xeon 6 CPU serving. BF16 is the precision used in training, so BF16 inference gives the best accuracy.

For the 72B sibling, see Qwen2.5-VL-72B-Instruct.

Prerequisites

NVIDIA

uv venv
source .venv/bin/activate
uv pip install -U vllm --torch-backend auto

Intel Xeon 6 CPUs

Hardware: Intel Xeon 6 CPUs
vLLM CPU >= 0.8.5 for this validated Xeon 6 configuration

For Intel and AMD x86 CPUs, follow the CPU pre-built wheels installation instructions.

docker pull vllm/vllm-openai-cpu:latest-x86_64

TPU (Trillium / v6e)

Use the official vLLM TPU image. The 7B model fits on a single v6e chip (v6e-1, --topology 1x1).

export HF_HOME=/dev/shm
export HF_TOKEN=<your HF token>
docker run --rm --privileged --net=host \
  --shm-size=16G \
  -e HF_HOME -e HF_TOKEN \
  vllm/vllm-tpu:latest \
  vllm serve ...

Deployment Configurations

Intel Xeon 6

Choose tensor parallelism based on the system topology; the recipe does not prescribe a fixed TP/DP layout or hard-code NUMA node IDs.

vllm serve Qwen/Qwen2.5-VL-7B-Instruct

Docker:

docker run -itd --name qwen25-vl-7b-cpu \
  --network host \
  --shm-size 16g \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai-cpu:latest-x86_64 \
    --model Qwen/Qwen2.5-VL-7B-Instruct \
    --host 0.0.0.0 \
    --port 8000

Runtime and Platform Tuning

The following settings are intentionally not prescribed as portable CPU model defaults because they depend on the workload, hardware, or runtime environment:

--max-num-batched-tokens: scheduler/throughput tuning.
--max-num-seqs: concurrency and scheduler-capacity tuning.
--gpu-memory-utilization: platform memory-budget tuning.
--no-enable-prefix-caching: workload/benchmark cache-policy tuning.
VLLM_ENGINE_ITERATION_TIMEOUT_S: operational runtime timeout.

These settings may still appear in validated hardware-specific overrides. Tune them at deployment time based on platform resources, workload shape, and latency/throughput goals.

TPU v6e (Trillium, single chip)

Verified end-to-end on a single v6e chip.

export HF_HOME=/dev/shm
export HF_TOKEN=<your HF token>
vllm serve Qwen/Qwen2.5-VL-7B-Instruct \
  --tensor-parallel-size 1 \
  --pipeline-parallel-size 1 \
  --dtype bfloat16 \
  --gpu-memory-utilization 0.98 \
  --max-model-len 16384 \
  --limit-mm-per-prompt '{"image":10,"video":0}' \
  --mm-processor-kwargs '{"max_pixels":1003520}' \
  --guided-decoding-backend xgrammar \
  --disable-chunked-mm-input

Single GPU (BF16, TP=1)

export CUDA_VISIBLE_DEVICES=0
vllm serve Qwen/Qwen2.5-VL-7B-Instruct \
  --host 0.0.0.0 \
  --port 8000 \
  --limit-mm-per-prompt '{"image":2,"video":0}'

Multi-GPU (DP=4)

For higher throughput across a node, data parallelism works better than TP for a model this size.

export CUDA_VISIBLE_DEVICES=0,1,2,3
vllm serve Qwen/Qwen2.5-VL-7B-Instruct \
  --host 0.0.0.0 \
  --port 8000 \
  --data-parallel-size 4 \
  --limit-mm-per-prompt '{"image":2,"video":0}'

Configuration Tips

--max-model-len 16384 is a safe default on a single v6e chip (native context is 128K); raise it if you have headroom.
--limit-mm-per-prompt caps incoming multimodal requests per prompt.
--mm-processor-kwargs '{"max_pixels":1003520}' bounds the per-image resolution to control encoder cost.
vLLM uses 90% of device memory by default; on TPU the recipe pushes --gpu-memory-utilization 0.98 to maximize KV cache.
--disable-chunked-mm-input is recommended on TPU for stable multimodal batching.

Benchmarking

Launch the server with --no-enable-prefix-caching to get consistent measurements.

VisionArena-Chat

vllm bench serve \
  --host 0.0.0.0 \
  --port 8000 \
  --backend openai-chat \
  --endpoint /v1/chat/completions \
  --model Qwen/Qwen2.5-VL-7B-Instruct \
  --dataset-name hf \
  --dataset-path lmarena-ai/VisionArena-Chat \
  --num-prompts 128

Random Synthetic

vllm bench serve \
  --host 0.0.0.0 \
  --port 8000 \
  --model Qwen/Qwen2.5-VL-7B-Instruct \
  --dataset-name random \
  --random-input-len 8000 \
  --random-output-len 1000 \
  --num-prompts 128