# Qwen/Qwen3-Reranker-0.6B — vLLM serving recipe

> Historical, single-NPU Qwen3-Reranker-0.6B pooling deployment on Atlas 800 A3, pinned to vLLM Ascend v0.23.0rc1.

Configuration: hardware `ascend_910c` (16× Ascend 910C) · strategy `single_node_tp` (Tensor Parallel) · variant `default` (BF16)

- Interactive page: https://recipes.vllm.ai/Qwen/Qwen3-Reranker-0.6B
- This configuration as JSON: https://recipes.vllm.ai/Qwen/Qwen3-Reranker-0.6B.json?hardware=ascend_910c

The commands below were rendered for exactly this configuration and take precedence over any example in the guide.

## Other configurations

Change the query string (the same parameters as the page URL) and fetch this URL again. Unknown or incompatible values fall back to the recipe default, so check the Configuration line.

- `hardware`: `ascend_910c`
- `variant`: `default`
- `strategy`: `single_node_tp` (multi-node strategies also need `nodes=<n>`)

## Install

Requires vLLM >= 0.23.0rc1.

Docker image: `quay.io/ascend/vllm-ascend:v0.23.0rc1-a3`

## Serve

```bash
export ASCEND_RT_VISIBLE_DEVICES=0
export OMP_PROC_BIND=false
export OMP_NUM_THREADS=1
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
vllm serve Qwen/Qwen3-Reranker-0.6B \
  --runner pooling \
  --hf-overrides '{"architectures":["Qwen3ForSequenceClassification"],"classifier_from_token":["no","yes"],"is_original_qwen3_reranker":true}' \
  --revision e61197ed45024b0ed8a2d74b80b4d909f1255473 \
  --dtype bfloat16 \
  --max-model-len 1024 \
  --port 8000 \
  --tensor-parallel-size 1
```

Docker equivalent:

```bash
docker run --device /dev/davinci0 --device /dev/davinci1 --device /dev/davinci2 --device /dev/davinci3 \
  --device /dev/davinci4 --device /dev/davinci5 --device /dev/davinci6 --device /dev/davinci7 \
  --device /dev/davinci8 --device /dev/davinci9 --device /dev/davinci10 --device /dev/davinci11 \
  --device /dev/davinci12 --device /dev/davinci13 --device /dev/davinci14 --device /dev/davinci15 \
  --device /dev/davinci_manager --device /dev/hisi_hdc \
  -v /usr/local/Ascend/driver:/usr/local/Ascend/driver \
  -v /etc/ascend_install.info:/etc/ascend_install.info \
  -v /usr/local/dcmi:/usr/local/dcmi \
  -v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \
  --privileged --net=host --shm-size=16g \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -e ASCEND_RT_VISIBLE_DEVICES=0 \
  -e OMP_PROC_BIND=false \
  -e OMP_NUM_THREADS=1 \
  -e PYTORCH_NPU_ALLOC_CONF=expandable_segments:True \
  quay.io/ascend/vllm-ascend:v0.23.0rc1-a3 \
  vllm serve Qwen/Qwen3-Reranker-0.6B \
  --runner pooling \
  --hf-overrides '{"architectures":["Qwen3ForSequenceClassification"],"classifier_from_token":["no","yes"],"is_original_qwen3_reranker":true}' \
  --revision e61197ed45024b0ed8a2d74b80b4d909f1255473 \
  --dtype bfloat16 \
  --max-model-len 1024 \
  --port 8000 \
  --tensor-parallel-size 1
```

## Notes for this configuration

### Historical release boundary

Qwen3-Reranker has been discontinued in vLLM Ascend since v0.30.0.
This recipe retains the v0.23.0rc1 historical deployment and does not
establish support on main, nightly, or a later release.

### Pooling clients

Use the Python verification and rerank benchmark in the Guide below
for this pooling model. The command builder's generic Verify and
Bench examples target text-generation services.

## Guide

> Background reference. Commands in it are examples and may target another configuration — use the Serve section above.

### Overview

[Qwen/Qwen3-Reranker-0.6B](https://huggingface.co/Qwen/Qwen3-Reranker-0.6B)
scores the relevance of documents to a query. The original checkpoint uses
the Qwen3 causal-LM architecture; the required HF overrides select
`Qwen3ForSequenceClassification` and the `no`/`yes` classification vectors.

This is a historical Atlas 800 A3 recipe. The
[current Ascend tutorial](https://github.com/vllm-project/vllm-ascend/blob/main/docs/source/tutorials/models/Qwen3-Reranker.md)
states that the model was discontinued in vLLM Ascend v0.30.0 and directs
users to the [v0.23.0 documentation](https://docs.vllm.ai/projects/ascend/en/v0.23.0/tutorials/models/Qwen3-Reranker.html).
The pinned image is retained from the migrated tutorial; no claim about
current-main support is made. The hardware entry is not marked `verified`
by this proposal.

The checkpoint config advertises 40,960 positions (the model card describes
a 32K context). The serving baseline here is 1,024 tokens. This recipe
makes no long-context, 4B, 8B, A2, or Atlas 300I DUO verification claim.

### Checkpoint provenance

The model revision is pinned to
[`e61197ed45024b0ed8a2d74b80b4d909f1255473`](https://huggingface.co/Qwen/Qwen3-Reranker-0.6B/tree/e61197ed45024b0ed8a2d74b80b4d909f1255473).
Its single `model.safetensors` file is 1,191,588,280 bytes with SHA-256
`27cd75a405b9c1b46b59abfd88aaa209e6fed2a1972cde9b70e7659537c5e65b`.
The `config.json` SHA-256 is
`d479c427a9ca5295218063d4f9aca4f297ab4ac27487cca7af42c84643d51ef0`.

Both hashes match the checkpoint staged for the accompanying task. Staging
integrity does not prove that inference or an accuracy benchmark passed.

If the runtime cannot access Hugging Face, stage this exact revision in a
shared cache before starting the container:

```bash
hf download Qwen/Qwen3-Reranker-0.6B --revision e61197ed45024b0ed8a2d74b80b4d909f1255473
```

### Client usage

After the pooling server is ready, install `requests` in the client
environment and run the Python example below. It sends a JSON POST to
`/v1/rerank` with the instruction, query, document and assistant suffix
required by this model. Use this example for rerank verification; the
command builder's generic Verify example targets chat completions.

The examples use port 8000 on the same host as the server. If the server
runs on another host or port, update the Python request URL and the
benchmark `--host` / `--port` arguments accordingly. Changes to the
builder's endpoint controls do not update these Guide examples.

```python
import math

import requests

prefix = '<|im_start|>system\nJudge whether the Document meets the requirements based on the Query and the Instruct provided. Note that the answer can only be "yes" or "no".<|im_end|>\n<|im_start|>user\n'
suffix = "<|im_end|>\n<|im_start|>assistant\n<think>\n\n</think>\n\n"
instruction = "Given a web search query, retrieve relevant passages that answer the query"
query = "What is the capital of China?"
documents = [
    "The capital of China is Beijing.",
    "Gravity is a force that attracts two bodies towards each other. It gives weight to physical objects and is responsible for the movement of planets around the sun.",
]

response = requests.post(
    "http://127.0.0.1:8000/v1/rerank",
    json={
        "model": "Qwen/Qwen3-Reranker-0.6B",
        "query": f"{prefix}<Instruct>: {instruction}\n<Query>: {query}\n",
        "documents": [f"<Document>: {doc}{suffix}" for doc in documents],
    },
    timeout=60,
)
response.raise_for_status()
payload = response.json()
results = payload.get("results")
assert isinstance(results, list) and len(results) == len(documents), payload
assert {item["index"] for item in results} == set(range(len(documents))), payload
scores = {item["index"]: item["relevance_score"] for item in results}
assert all(
    isinstance(score, (int, float))
    and math.isfinite(score)
    and 0.0 <= score <= 1.0
    for score in scores.values()
), payload
assert scores[0] > scores[1], payload
print(payload)
```

A successful result has HTTP 200, one finite probability score per
document, and a higher score for the Beijing passage. This is a functional
smoke check, not an MTEB quality measurement.

### Evaluation and performance

Run this serving benchmark in the same historical vLLM environment as
the pooling server. It targets `/v1/rerank` with the `vllm-rerank` backend
and synthetic `random-rerank` input. The command builder's generic Bench
example targets text generation; use this rerank command instead.

```bash
vllm bench serve \
  --model Qwen/Qwen3-Reranker-0.6B \
  --host localhost \
  --port 8000 \
  --backend vllm-rerank \
  --dataset-name random-rerank \
  --endpoint /v1/rerank \
  --random-input 200 --save-result --result-dir ./
```

This is a synthetic serving benchmark, not a dataset accuracy test. For
MTEB `MultiLongDocReranking`, follow the
[historical instructions](https://docs.vllm.ai/projects/ascend/en/v0.23.0/tutorials/models/Qwen3-Reranker.html)
with a matching vLLM environment. This recipe does not claim to reproduce
model-card quality results.

### References

- [Official model card](https://huggingface.co/Qwen/Qwen3-Reranker-0.6B)
- [Pinned checkpoint](https://huggingface.co/Qwen/Qwen3-Reranker-0.6B/tree/e61197ed45024b0ed8a2d74b80b4d909f1255473)
- [Historical Ascend tutorial](https://docs.vllm.ai/projects/ascend/en/v0.23.0/tutorials/models/Qwen3-Reranker.html)
- [Current support notice](https://github.com/vllm-project/vllm-ascend/blob/main/docs/source/tutorials/models/Qwen3-Reranker.md)