{ "hf_id": "Qwen/Qwen3-Reranker-0.6B", "meta": { "title": "Qwen3-Reranker-0.6B (Ascend historical deployment)", "slug": "qwen3-reranker-0.6b", "provider": "Qwen", "description": "Historical, single-NPU Qwen3-Reranker-0.6B pooling deployment on Atlas 800 A3, pinned to vLLM Ascend v0.23.0rc1.", "date_added": "2026-11-04", "date_updated": "2026-11-05", "tasks": [ "embedding" ], "related_recipes": [], "default_hardware": "ascend_910c", "hardware": { "ascend_910c": "supported" } }, "recommended_command": { "hardware": "ascend_910c", "strategy": "single_node_tp", "variant": "default", "node_count": 1, "deploy_type": "single_node", "env": { "ASCEND_RT_VISIBLE_DEVICES": "0", "OMP_PROC_BIND": "false", "OMP_NUM_THREADS": "1", "PYTORCH_NPU_ALLOC_CONF": "expandable_segments:True" }, "docker_image": "quay.io/ascend/vllm-ascend:v0.23.0rc1-a3", "command": "vllm serve Qwen/Qwen3-Reranker-0.6B \\\n --runner pooling \\\n --hf-overrides '{\"architectures\":[\"Qwen3ForSequenceClassification\"],\"classifier_from_token\":[\"no\",\"yes\"],\"is_original_qwen3_reranker\":true}' \\\n --revision e61197ed45024b0ed8a2d74b80b4d909f1255473 \\\n --dtype bfloat16 \\\n --max-model-len 1024 \\\n --port 8000 \\\n --tensor-parallel-size 1", "argv": [ "vllm", "serve", "Qwen/Qwen3-Reranker-0.6B", "--runner", "pooling", "--hf-overrides", "{\"architectures\":[\"Qwen3ForSequenceClassification\"],\"classifier_from_token\":[\"no\",\"yes\"],\"is_original_qwen3_reranker\":true}", "--revision", "e61197ed45024b0ed8a2d74b80b4d909f1255473", "--dtype", "bfloat16", "--max-model-len", "1024", "--port", "8000", "--tensor-parallel-size", "1" ], "docker_command": "docker run --device /dev/davinci0 --device /dev/davinci1 --device /dev/davinci2 --device /dev/davinci3 \\\n --device /dev/davinci4 --device /dev/davinci5 --device /dev/davinci6 --device /dev/davinci7 \\\n --device /dev/davinci8 --device /dev/davinci9 --device /dev/davinci10 --device /dev/davinci11 \\\n --device /dev/davinci12 --device /dev/davinci13 --device /dev/davinci14 --device /dev/davinci15 \\\n --device /dev/davinci_manager --device /dev/hisi_hdc \\\n -v /usr/local/Ascend/driver:/usr/local/Ascend/driver \\\n -v /etc/ascend_install.info:/etc/ascend_install.info \\\n -v /usr/local/dcmi:/usr/local/dcmi \\\n -v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \\\n --privileged --net=host --shm-size=16g \\\n -v ~/.cache/huggingface:/root/.cache/huggingface \\\n -e ASCEND_RT_VISIBLE_DEVICES=0 \\\n -e OMP_PROC_BIND=false \\\n -e OMP_NUM_THREADS=1 \\\n -e PYTORCH_NPU_ALLOC_CONF=expandable_segments:True \\\n quay.io/ascend/vllm-ascend:v0.23.0rc1-a3 \\\n vllm serve Qwen/Qwen3-Reranker-0.6B \\\n --runner pooling \\\n --hf-overrides '{\"architectures\":[\"Qwen3ForSequenceClassification\"],\"classifier_from_token\":[\"no\",\"yes\"],\"is_original_qwen3_reranker\":true}' \\\n --revision e61197ed45024b0ed8a2d74b80b4d909f1255473 \\\n --dtype bfloat16 \\\n --max-model-len 1024 \\\n --port 8000 \\\n --tensor-parallel-size 1", "docker_argv": [ "docker", "run", "--device", "/dev/davinci0", "--device", "/dev/davinci1", "--device", "/dev/davinci2", "--device", "/dev/davinci3", "--device", "/dev/davinci4", "--device", "/dev/davinci5", "--device", "/dev/davinci6", "--device", "/dev/davinci7", "--device", "/dev/davinci8", "--device", "/dev/davinci9", "--device", "/dev/davinci10", "--device", "/dev/davinci11", "--device", "/dev/davinci12", "--device", "/dev/davinci13", "--device", "/dev/davinci14", "--device", "/dev/davinci15", "--device", "/dev/davinci_manager", "--device", "/dev/hisi_hdc", "-v", "/usr/local/Ascend/driver:/usr/local/Ascend/driver", "-v", "/etc/ascend_install.info:/etc/ascend_install.info", "-v", "/usr/local/dcmi:/usr/local/dcmi", "-v", "/usr/local/bin/npu-smi:/usr/local/bin/npu-smi", "--privileged", "--net=host", "--shm-size", "16g", "-v", "~/.cache/huggingface:/root/.cache/huggingface", "-e", "ASCEND_RT_VISIBLE_DEVICES=0", "-e", "OMP_PROC_BIND=false", "-e", "OMP_NUM_THREADS=1", "-e", "PYTORCH_NPU_ALLOC_CONF=expandable_segments:True", "quay.io/ascend/vllm-ascend:v0.23.0rc1-a3", "vllm", "serve", "Qwen/Qwen3-Reranker-0.6B", "--runner", "pooling", "--hf-overrides", "{\"architectures\":[\"Qwen3ForSequenceClassification\"],\"classifier_from_token\":[\"no\",\"yes\"],\"is_original_qwen3_reranker\":true}", "--revision", "e61197ed45024b0ed8a2d74b80b4d909f1255473", "--dtype", "bfloat16", "--max-model-len", "1024", "--port", "8000", "--tensor-parallel-size", "1" ], "strategy_spec": { "name": "single_node_tp", "deploy_type": "single_node", "display_name": "Tensor Parallel", "short_label": "TP", "orientation": "latency", "description": "Single-node tensor parallel. Splits the model across all local GPUs. TP size is set to the GPU count at deploy time. The simplest multi-GPU strategy — works for all model architectures.\n", "hardware_match": { "min_gpus": 1, "max_gpus": 8, "multi_node": false }, "vllm_args": [], "parallel_flag": "--tensor-parallel-size" }, "hardware_profile": { "brand": "Huawei", "generation": "npu", "display_name": "Ascend 910C", "description": "Huawei Ascend 910C · 16× NPU · 64 GB HBM each · Atlas 800 A3 · vLLM Ascend backend", "gpu_count": 16, "vram_gb": 1024, "multi_node": false, "restricted": true }, "by_hardware": { "ascend_910c": "/Qwen/Qwen3-Reranker-0.6B/hw/ascend_910c.json" } }, "model": { "model_id": "Qwen/Qwen3-Reranker-0.6B", "min_vllm_version": "0.23.0rc1", "docker_image": "quay.io/ascend/vllm-ascend:v0.23.0rc1-a3", "nightly_required": false, "architecture": "dense", "parameter_count": "0.6B", "active_parameters": "0.6B", "context_length": 40960, "base_args": [ "--runner", "pooling", "--hf-overrides", "{\"architectures\":[\"Qwen3ForSequenceClassification\"],\"classifier_from_token\":[\"no\",\"yes\"],\"is_original_qwen3_reranker\":true}", "--revision", "e61197ed45024b0ed8a2d74b80b4d909f1255473", "--dtype", "bfloat16", "--max-model-len", "1024", "--port", "8000" ], "base_env": { "ASCEND_RT_VISIBLE_DEVICES": "0", "OMP_PROC_BIND": "false", "OMP_NUM_THREADS": "1", "PYTORCH_NPU_ALLOC_CONF": "expandable_segments:True" }, "install": { "docker": { "command": "docker pull quay.io/ascend/vllm-ascend:v0.23.0rc1-a3", "note": "Historical image only. Qwen3-Reranker is discontinued in vLLM Ascend v0.30.0 and later." } } }, "features": {}, "opt_in_features": [], "variants": { "default": { "precision": "bf16", "vram_minimum_gb": 2, "description": "Official 0.6B checkpoint, BF16, one Ascend 910C NPU; historical image only.", "supported_hardware": [ "ascend_910c" ], "hardware_overrides": { "ascend_910c": { "docker_image": "quay.io/ascend/vllm-ascend:v0.23.0rc1-a3", "docker_serve_command": "vllm serve", "install": { "pip": false } } } } }, "hardware_overrides": {}, "notes": [ { "title": "Historical release boundary", "when": { "hardware": [ "ascend_910c" ] }, "body": "Qwen3-Reranker has been discontinued in vLLM Ascend since v0.30.0.\nThis recipe retains the v0.23.0rc1 historical deployment and does not\nestablish support on main, nightly, or a later release." }, { "title": "Pooling clients", "when": { "hardware": [ "ascend_910c" ] }, "body": "Use the Python verification and rerank benchmark in the Guide below\nfor this pooling model. The command builder's generic Verify and\nBench examples target text-generation services." } ], "guide": "## Overview\n\n[Qwen/Qwen3-Reranker-0.6B](https://huggingface.co/Qwen/Qwen3-Reranker-0.6B)\nscores the relevance of documents to a query. The original checkpoint uses\nthe Qwen3 causal-LM architecture; the required HF overrides select\n`Qwen3ForSequenceClassification` and the `no`/`yes` classification vectors.\n\nThis is a historical Atlas 800 A3 recipe. The\n[current Ascend tutorial](https://github.com/vllm-project/vllm-ascend/blob/main/docs/source/tutorials/models/Qwen3-Reranker.md)\nstates that the model was discontinued in vLLM Ascend v0.30.0 and directs\nusers to the [v0.23.0 documentation](https://docs.vllm.ai/projects/ascend/en/v0.23.0/tutorials/models/Qwen3-Reranker.html).\nThe pinned image is retained from the migrated tutorial; no claim about\ncurrent-main support is made. The hardware entry is not marked `verified`\nby this proposal.\n\nThe checkpoint config advertises 40,960 positions (the model card describes\na 32K context). The serving baseline here is 1,024 tokens. This recipe\nmakes no long-context, 4B, 8B, A2, or Atlas 300I DUO verification claim.\n\n## Checkpoint provenance\n\nThe model revision is pinned to\n[`e61197ed45024b0ed8a2d74b80b4d909f1255473`](https://huggingface.co/Qwen/Qwen3-Reranker-0.6B/tree/e61197ed45024b0ed8a2d74b80b4d909f1255473).\nIts single `model.safetensors` file is 1,191,588,280 bytes with SHA-256\n`27cd75a405b9c1b46b59abfd88aaa209e6fed2a1972cde9b70e7659537c5e65b`.\nThe `config.json` SHA-256 is\n`d479c427a9ca5295218063d4f9aca4f297ab4ac27487cca7af42c84643d51ef0`.\n\nBoth hashes match the checkpoint staged for the accompanying task. Staging\nintegrity does not prove that inference or an accuracy benchmark passed.\n\nIf the runtime cannot access Hugging Face, stage this exact revision in a\nshared cache before starting the container:\n\n```bash\nhf download Qwen/Qwen3-Reranker-0.6B --revision e61197ed45024b0ed8a2d74b80b4d909f1255473\n```\n\n## Client usage\n\nAfter the pooling server is ready, install `requests` in the client\nenvironment and run the Python example below. It sends a JSON POST to\n`/v1/rerank` with the instruction, query, document and assistant suffix\nrequired by this model. Use this example for rerank verification; the\ncommand builder's generic Verify example targets chat completions.\n\nThe examples use port 8000 on the same host as the server. If the server\nruns on another host or port, update the Python request URL and the\nbenchmark `--host` / `--port` arguments accordingly. Changes to the\nbuilder's endpoint controls do not update these Guide examples.\n\n```python\nimport math\n\nimport requests\n\nprefix = '<|im_start|>system\\nJudge whether the Document meets the requirements based on the Query and the Instruct provided. Note that the answer can only be \"yes\" or \"no\".<|im_end|>\\n<|im_start|>user\\n'\nsuffix = \"<|im_end|>\\n<|im_start|>assistant\\n\\n\\n\\n\\n\"\ninstruction = \"Given a web search query, retrieve relevant passages that answer the query\"\nquery = \"What is the capital of China?\"\ndocuments = [\n \"The capital of China is Beijing.\",\n \"Gravity is a force that attracts two bodies towards each other. It gives weight to physical objects and is responsible for the movement of planets around the sun.\",\n]\n\nresponse = requests.post(\n \"http://127.0.0.1:8000/v1/rerank\",\n json={\n \"model\": \"Qwen/Qwen3-Reranker-0.6B\",\n \"query\": f\"{prefix}: {instruction}\\n: {query}\\n\",\n \"documents\": [f\": {doc}{suffix}\" for doc in documents],\n },\n timeout=60,\n)\nresponse.raise_for_status()\npayload = response.json()\nresults = payload.get(\"results\")\nassert isinstance(results, list) and len(results) == len(documents), payload\nassert {item[\"index\"] for item in results} == set(range(len(documents))), payload\nscores = {item[\"index\"]: item[\"relevance_score\"] for item in results}\nassert all(\n isinstance(score, (int, float))\n and math.isfinite(score)\n and 0.0 <= score <= 1.0\n for score in scores.values()\n), payload\nassert scores[0] > scores[1], payload\nprint(payload)\n```\n\nA successful result has HTTP 200, one finite probability score per\ndocument, and a higher score for the Beijing passage. This is a functional\nsmoke check, not an MTEB quality measurement.\n\n## Evaluation and performance\n\nRun this serving benchmark in the same historical vLLM environment as\nthe pooling server. It targets `/v1/rerank` with the `vllm-rerank` backend\nand synthetic `random-rerank` input. The command builder's generic Bench\nexample targets text generation; use this rerank command instead.\n\n```bash\nvllm bench serve \\\n --model Qwen/Qwen3-Reranker-0.6B \\\n --host localhost \\\n --port 8000 \\\n --backend vllm-rerank \\\n --dataset-name random-rerank \\\n --endpoint /v1/rerank \\\n --random-input 200 --save-result --result-dir ./\n```\n\nThis is a synthetic serving benchmark, not a dataset accuracy test. For\nMTEB `MultiLongDocReranking`, follow the\n[historical instructions](https://docs.vllm.ai/projects/ascend/en/v0.23.0/tutorials/models/Qwen3-Reranker.html)\nwith a matching vLLM environment. This recipe does not claim to reproduce\nmodel-card quality results.\n\n## References\n\n- [Official model card](https://huggingface.co/Qwen/Qwen3-Reranker-0.6B)\n- [Pinned checkpoint](https://huggingface.co/Qwen/Qwen3-Reranker-0.6B/tree/e61197ed45024b0ed8a2d74b80b4d909f1255473)\n- [Historical Ascend tutorial](https://docs.vllm.ai/projects/ascend/en/v0.23.0/tutorials/models/Qwen3-Reranker.html)\n- [Current support notice](https://github.com/vllm-project/vllm-ascend/blob/main/docs/source/tutorials/models/Qwen3-Reranker.md)", "source": { "source": "vllm-ascend", "repo": "vllm-project/vllm-ascend", "ref": "main", "sha": "931bfd948e04a89b735a2f78520093aa9dbc49f0", "path": "recipes/models/Qwen/Qwen3-Reranker-0.6B.yaml", "edit_url": "https://github.com/vllm-project/vllm-ascend/edit/main/recipes/models/Qwen/Qwen3-Reranker-0.6B.yaml", "overlays": [] } }