vLLM/Recipes
Qwen

Qwen/Qwen3.8-27B

27B-parameter dense hybrid-attention model with linear attention on 48 of 64 layers, a vision tower, a built-in MTP draft head, 262K native context window and extensible to 1M context

Fits one Blackwell GPU in every precision: NVFP4 in 24.6 GiB, 6.6M KV tokens at 1M context

View on HuggingFace
dense27B262,144 ctxvLLM 0.17.0+multimodaltext
Guide

Overview

Qwen3.8-27B is the 27-billion-parameter dense member of the Qwen3.8 family, on the same hybrid-attention backbone as the 2.4T MoE flagship.

The layer mix is the interesting part. Only 16 of the 64 layers run full attention (full_attention_interval: 4); the other 48 run linear attention with a constant recurrent state. Unlike the 2.4T this is a multimodal model: the architecture is Qwen3_5ForConditionalGeneration and config.json carries a vision_config. Text serving is what this recipe covers and what has been verified.

Prerequisites

  • transformers >= 5.8.0, matching the version config.json was written by. vLLM parses the config with its own Qwen3_5Config, so this is really about the Qwen3-VL processor.

Launch commands

Low latency

NVFP4, TP1:

vllm serve Inferact/Qwen3.8-27B-NVFP4 \
  --tensor-parallel-size 1 \
  --max-model-len 262144 \
  --kv-cache-dtype fp8 \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice --tool-call-parser qwen3_coder

FP8, TP4 (one GB300 tray) for the largest KV cache:

vllm serve Qwen/Qwen3.8-27B-FP8 \
  --tensor-parallel-size 4 \
  --max-model-len 262144 \
  --kv-cache-dtype fp8 \
  --reasoning-parser qwen3

Add --speculative-config '{"method":"mtp","num_speculative_tokens":3}' for MTP.

Client usage

generation_config.json ships temperature: 1.0, top_p: 0.95, top_k: 20.

from openai import OpenAI

client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1", timeout=3600)

resp = client.chat.completions.create(
    model="Qwen/Qwen3.8-27B",
    messages=[{"role": "user", "content": "Give me three primes above 100."}],
    temperature=1.0, top_p=0.95, max_tokens=2048,
)
print(resp.choices[0].message.content)

Thinking modes

The model supports no-think and adaptive thinking through chat_template_kwargs, per request or server-wide via --default-chat-template-kwargs:

  • {"enable_thinking": false} — no thinking, the model answers directly.
  • {"reasoning_effort": "low"} — adaptive thinking. xhigh (default), medium, low.

Processing Ultra-Long Texts

The model has 262k native context length and it can be extended to 1M with --max-model-len flag. The best value can be picked based on the use case and GPU ram usage tradeoff.

For example, to enable full 1M context length on this model:

vllm serve Qwen/Qwen3.8-27B \
  --max-model-len 1010000 \
  --hf-overrides '{"text_config": {"max_position_embeddings": 1010000}}' \
  ...

Note the override is nested under text_config here, where the 2.4T takes it flat.

Troubleshooting

MXFP4 does not load on Nvidia devices. The vLLM MXFP4 implementation on Nvidia device is currently missing linear method support so it doesn't run as intended. Use NVFP4 quantization on Nvidia instead.

References