# Qwen/Qwen3.8-27B — vLLM serving recipe

> 27B-parameter dense hybrid-attention model with linear attention on 48 of 64 layers, a vision tower, a built-in MTP draft head, 262K native context window and extensible to 1M context

Configuration: hardware `h200` (8× H200) · strategy `single_node_tp` (Tensor Parallel) · variant `default` (BF16) · features `tool_calling`, `reasoning`, `encoder_parallel` · frontend `rust`

- Interactive page: https://recipes.vllm.ai/Qwen/Qwen3.8-27B
- This configuration as JSON: https://recipes.vllm.ai/Qwen/Qwen3.8-27B.json?hardware=h200

The commands below were rendered for exactly this configuration and take precedence over any example in the guide.

## Other configurations

Change the query string (the same parameters as the page URL) and fetch this URL again. Unknown or incompatible values fall back to the recipe default, so check the Configuration line.

- `hardware`: `h100`, `h200`, `b200`, `gb200`, `b300`, `gb300`, `dgx_spark_gb10`, `rtx_pro_6000`, `rtx_5090`, `rtx_5090_2x`, `mi300x`, `mi325x`, `mi350x`, `mi355x`, `arc_pro_b60`, `arc_pro_b70`, `ascend_950pr`
- `variant`: `default`, `fp8`, `nvfp4`, `nvfp4_nvidia`, `int4`, `mxfp4`, `ascend_w8a8`
- `strategy`: `single_node_tp` (multi-node strategies also need `nodes=<n>`)
- `features`: comma-separated list of `tool_calling`, `reasoning`, `spec_decoding`, `text_only`, `encoder_parallel`, `long_context` (default: `tool_calling`, `reasoning`, `encoder_parallel`)

## Install

Requires vLLM >= 0.17.0.

```bash
uv venv
source .venv/bin/activate
uv pip install -U vllm --torch-backend auto
```

Docker image: `vllm/vllm-openai:qwen38`

Extra dependencies:

- Matches the transformers version config.json was written by (5.8.0), for the Qwen3-VL processor classes. vLLM parses the config with its own Qwen3_5Config.

```bash
uv pip install -U "transformers>=5.8.0"
```

## Serve

```bash
export VLLM_USE_RUST_FRONTEND=1
vllm serve Qwen/Qwen3.8-27B \
  --tensor-parallel-size 1 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --reasoning-parser qwen3 \
  --mm-encoder-tp-mode data
```

Docker equivalent:

```bash
docker run --gpus all \
  --privileged --ipc=host -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -e VLLM_USE_RUST_FRONTEND=1 \
  vllm/vllm-openai:qwen38 Qwen/Qwen3.8-27B \
  --tensor-parallel-size 1 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --reasoning-parser qwen3 \
  --mm-encoder-tp-mode data
```

## Guide

> Background reference. Commands in it are examples and may target another configuration — use the Serve section above.

### Overview

Qwen3.8-27B is the 27-billion-parameter dense member of the Qwen3.8 family, on the same
hybrid-attention backbone as the 2.4T MoE flagship.

**The layer mix is the interesting part.** Only 16 of the 64 layers run full attention
(`full_attention_interval: 4`); the other 48 run linear attention with a constant
recurrent state. Unlike the 2.4T this is a multimodal model: the architecture is
`Qwen3_5ForConditionalGeneration` and `config.json` carries a `vision_config`. Text
serving is what this recipe covers and what has been verified. Quantized builds (NVFP4, FP8,
INT4, and Red Hat AI's MXFP4) are offered as variants alongside the base checkpoint.

### Prerequisites

- **transformers >= 5.8.0**, matching the version `config.json` was written by. vLLM parses
  the config with its own `Qwen3_5Config`, so this is really about the Qwen3-VL processor.

### Launch commands

#### Low latency

NVFP4, TP1:

```bash
vllm serve Inferact/Qwen3.8-27B-NVFP4 \
  --tensor-parallel-size 1 \
  --max-model-len 262144 \
  --kv-cache-dtype fp8 \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice --tool-call-parser qwen3_coder
```

FP8, TP4 (one GB300 tray) for the largest KV cache:

```bash
vllm serve Qwen/Qwen3.8-27B-FP8 \
  --tensor-parallel-size 4 \
  --max-model-len 262144 \
  --kv-cache-dtype fp8 \
  --reasoning-parser qwen3
```

Add `--speculative-config '{"method":"mtp","num_speculative_tokens":3}'` for MTP.

#### MXFP4 (W4A4) — Blackwell

```bash
vllm serve RedHatAI/Qwen3.8-27B-MXFP4 \
  --tensor-parallel-size 1 \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice --tool-call-parser qwen3_coder
```

True W4A4 on Blackwell. The in-checkpoint MTP head, long-context YaRN and tool calling all
carry over from the base recipe.

#### 2x RTX 5090 (consumer Blackwell, sm120)

Verified on vLLM `0.26.1rc1.dev608+g99a10304d`, TP2 across two cards.

**NVFP4 uses the real kernel here.** vLLM selects
`FlashInferCutlassNvFp4LinearKernel for NVFP4 GEMM` on sm120 — a cutlass path, not an
emulation fallback — for both the `Inferact` build above and `unsloth/Qwen3.8-27B-NVFP4`.

**The block-scaled FP8 checkpoint needs no workaround.** vLLM auto-disables DeepGemm for
`model_type=qwen3_5_text` on Blackwell and falls back to CUTLASS, so it loads unaided.
Verified by running *without* `VLLM_USE_DEEP_GEMM=0`: identical 377,456-token KV pool.

```bash
vllm serve unsloth/Qwen3.8-27B-NVFP4 \
  --tensor-parallel-size 2 \
  --max-model-len 262144 \
  --kv-cache-dtype fp8 \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice --tool-call-parser qwen3_coder
```

At 262,144 context, GPU KV cache size as reported at startup:

| precision | KV tokens | weights/GPU | MTP acceptance |
|---|---|---|---|
| FP8 | 377,456 | 14.28 GiB | 0.771 |
| NVFP4 (Inferact) | 445,875 | 12.02 GiB | 0.897 |
| NVFP4 (unsloth) | 920,517 | 10.64 GiB | 0.788 |

The two NVFP4 builds are not interchangeable on 32 GB cards: `unsloth` is
mixed-precision (FP8 channel-wise alongside the 4-bit groups) and leaves room for roughly
twice the KV cache, while the uniform-W4A4 `Inferact` build drafts better. Both serve
correctly.

The in-checkpoint MTP head works in every precision above. Acceptance is measured from
`vllm:spec_decode_num_{accepted,draft}_tokens_total`, since throughput alone cannot
distinguish a working drafter from one that loaded and was ignored.

#### 1x RTX 5090 — NVFP4 needs `--enforce-eager`

One card has **31.4 GiB usable**, not 32, and NVFP4 fits only with CUDA graphs off:

```bash
vllm serve Inferact/Qwen3.8-27B-NVFP4 \
  --tensor-parallel-size 1 \
  --max-model-len 32768 \
  --kv-cache-dtype fp8 \
  --enforce-eager \
  --reasoning-parser qwen3
```

Without `--enforce-eager`, startup dies in CUDA graph capture with
`torch.OutOfMemoryError: Tried to allocate 784.00 MiB`. **Raising or lowering
`--gpu-memory-utilization` does not help** — 0.80 and 0.93 both leave the same 47.06 MiB
free, because that budget covers weights and KV while graph capture allocates outside it.

KV pool at 32K context: 91,022 tokens with `--enforce-eager` alone; 135,926 adding
`--language-model-only`; 152,917 also capping `--max-num-seqs 8`; 76,458 with bf16 KV
instead of fp8 — so fp8 KV is a choice here, not a requirement. The MTP head still fits
(90,112 tokens, 0.754 acceptance). Those two flags are levers for a bigger pool, not fixes
for the OOM: only `--enforce-eager` decides whether the server starts.

#### Intel XPU B60 / B70

Validated on 4x Intel Arc Pro B60 / B70 (B60 24 GB, B70 32 GB per card) with the
official vLLM XPU image `vllm/vllm-openai-xpu:latest`. `--enforce-eager` is required on this
backend. Select Arc Pro B60/B70 in the hardware picker above for the exact command.

#### Huawei Ascend 950PR-w8a8 (vLLM Ascend)

Verified on a single 950PR, TP1, vLLM Ascend 0.23.0 from the
`quay.io/ascend/vllm-ascend:qwen3.8-a5` image.

The ModelSlim INT8 build is the checkpoint to use here. `--quantization ascend` is what
selects the ModelSlim scheme — without it the loader treats the weights as float. The MTP
head inside the checkpoint works and is spelled `qwen3_5_mtp` on this backend.

Both the weight and the image come from ModelScope, not HuggingFace: set
`VLLM_USE_MODELSCOPE=True` or pass a local path.

```bash
export VLLM_USE_MODELSCOPE=True
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export HCCL_BUFFSIZE=512
export OMP_PROC_BIND=false
export OMP_NUM_THREADS=1

vllm serve Eco-Tech/Qwen3.8-27B-w8a8 \
  --tensor-parallel-size 1 \
  --quantization ascend \
  --max-num-batched-tokens 16384 \
  --enable-prefix-caching \
  --reasoning-parser qwen3 \
  --tool-call-parser qwen3_coder \
  --mm-encoder-tp-mode data \
  --speculative-config '{"method":"qwen3_5_mtp","num_speculative_tokens":3,"enforce_eager":true}' \
  --compilation-config '{"cudagraph_mode":"FULL_DECODE_ONLY"}' \
  --additional-config '{"enable_cpu_binding":true}'
```

`--reasoning-parser qwen3` is not optional in practice: the chat template opens every
assistant turn with `<think>`, so without it the entire reasoning block lands in
`message.content` and a 2048-token budget can be spent before the answer starts.

Measured single-stream decode: 64 tok/s (2048-token completion, one request, MTP on).
Read MTP acceptance from `vllm:spec_decode_num_{accepted,draft}_tokens_total` before
tuning `num_speculative_tokens` — throughput alone cannot tell a working drafter from
one that loaded and was ignored.

#### Huawei Ascend 950PR-Native block-scaled FP8 (vLLM Ascend)

Verified on a single 950PR, TP1, vLLM Ascend 0.23.0 from the
`quay.io/ascend/vllm-ascend:nightly-main-a5` image.

The official `Qwen/Qwen3.8-27B-FP8` checkpoint serves directly on a 950PR — no offline
re-quantization and **no `--quantization ascend`**. vLLM Ascend reads `quant_method: fp8`
and `weight_block_size` straight from the checkpoint, expands the `weight_scale_inv`
tiles at load time and re-quantizes to MXFP8 on 950.

Requires a vLLM Ascend build that includes
[vllm-ascend#14852](https://github.com/vllm-project/vllm-ascend/pull/14852) (merged
2026-08-26); no tagged release carries it yet. The W8A8 path above is unaffected.

```bash
vllm serve Qwen/Qwen3.8-27B-FP8 \
  --tensor-parallel-size 1 \
  --max-model-len 128000 \
  --max-num-seqs 16 \
  --gpu-memory-utilization 0.85 \
  --trust-remote-code \
  --enable-prefix-caching \
  --reasoning-parser qwen3 \
  --tool-call-parser qwen3_coder \
  --mm-encoder-tp-mode data \
  --speculative-config '{"method": "qwen3_5_mtp", "num_speculative_tokens": 3, "enforce_eager": true}'
```

#### DFlash2 (opt-in)

DFlash2 is not a standalone model. Serve `Qwen/Qwen3.8-27B` and attach
[incoai/Qwen3.8-27B-DFlash2](https://huggingface.co/incoai/Qwen3.8-27B-DFlash2)
as the drafter. Requires vLLM >=0.28.0, which shipped
[vllm#52816](https://github.com/vllm-project/vllm/pull/52816).
```bash
vllm serve Qwen/Qwen3.8-27B \
  --speculative-config '{"method":"dflash","model":"incoai/Qwen3.8-27B-DFlash2","num_speculative_tokens":7}'
```

#### DSpark (opt-in)

DSpark is an external confidence-scheduled drafter. Serve `Qwen/Qwen3.8-27B` and attach
[RedHatAI/Qwen3.8-27B-speculator.dspark](https://huggingface.co/RedHatAI/Qwen3.8-27B-speculator.dspark)
with eight speculative tokens:

```bash
vllm serve Qwen/Qwen3.8-27B \
  --speculative-config '{"method":"dspark","model":"RedHatAI/Qwen3.8-27B-speculator.dspark","num_speculative_tokens":8}'
```

### Client usage

`generation_config.json` ships `temperature: 1.0`, `top_p: 0.95`, `top_k: 20`.

```python
from openai import OpenAI

client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1", timeout=3600)

resp = client.chat.completions.create(
    model="Qwen/Qwen3.8-27B",
    messages=[{"role": "user", "content": "Give me three primes above 100."}],
    temperature=1.0, top_p=0.95, max_tokens=2048,
)
print(resp.choices[0].message.content)
```

#### Thinking modes

The model supports no-think and adaptive thinking through `chat_template_kwargs`, per
request or server-wide via `--default-chat-template-kwargs`:

- `{"enable_thinking": false}` — no thinking, the model answers directly.
- `{"reasoning_effort": "low"}` — adaptive thinking. `xhigh` (default), `medium`, `low`.

### Processing Ultra-Long Texts

The model is native at 262k. To extend it to 1M, turn on the **Long Context**
feature above: it applies the model card's YaRN RoPE configuration via
`--hf-overrides` and raises `--max-model-len` to 1,000,000. Static YaRN holds
the scaling factor constant regardless of input length, so leave it off unless
you actually serve long prompts; for a ~524K workload, halve the card's
`factor` to `2.0`.

Note the override is nested under `text_config` here, where the 2.4T takes it flat.

See the [model card](https://huggingface.co/Qwen/Qwen3.8-27B#processing-ultra-long-texts) for the full parameter reference.

### Troubleshooting

**Two different "MXFP4" paths.** The *native* `Mxfp4Config` path (gpt-oss-style weight-only
checkpoints) has no linear-layer method and falls back to `UnquantizedLinearMethod`, so it does
not run. This recipe's **RedHatAI MXFP4** variant instead uses the *compressed-tensors*
`mxfp4-pack-quantized` format, which maps to `CompressedTensorsW4A4Mxfp4` — a real W4A4 method
on Blackwell (SM100+) and W4A16-via-Marlin elsewhere. That path loads and serves correctly on
Nvidia; pick the **MXFP4** variant (or NVFP4) when you want the 4-bit build.

### References

- Model card: https://huggingface.co/Qwen/Qwen3.8-27B
- FP8 checkpoint: https://huggingface.co/Qwen/Qwen3.8-27B-FP8
- NVFP4 build (NVIDIA): https://huggingface.co/nvidia/Qwen3.8-27B-NVFP4
- NVFP4 build (Inferact): https://huggingface.co/Inferact/Qwen3.8-27B-NVFP4
- MXFP4 build (RedHatAI): https://huggingface.co/RedHatAI/Qwen3.8-27B-MXFP4
- Ascend INT8 checkpoint (ModelScope): https://www.modelscope.cn/models/Eco-Tech/Qwen3.8-27B-w8a8
- vLLM documentation: https://docs.vllm.ai/
- vLLM Ascend tutorial: https://docs.vllm.ai/projects/ascend/en/latest/tutorials/models/Qwen3.8-27B.html
- DFlash2 draft: https://huggingface.co/incoai/Qwen3.8-27B-DFlash2
- DSpark draft: https://huggingface.co/RedHatAI/Qwen3.8-27B-speculator.dspark
- DFlash2 blog: https://inco.ai/blog/dflash2/
- vLLM DFlash2 PR: https://github.com/vllm-project/vllm/pull/52816
- Native FP8 support on Ascend: https://github.com/vllm-project/vllm-ascend/pull/14852