Qwen/Qwen3.8-27B

Overview

Qwen3.8-27B is the 27-billion-parameter dense member of the Qwen3.8 family, on the same hybrid-attention backbone as the 2.4T MoE flagship.

The layer mix is the interesting part. Only 16 of the 64 layers run full attention (full_attention_interval: 4); the other 48 run linear attention with a constant recurrent state. Unlike the 2.4T this is a multimodal model: the architecture is Qwen3_5ForConditionalGeneration and config.json carries a vision_config. Text serving is what this recipe covers and what has been verified.

Prerequisites

• transformers >= 5.8.0, matching the version config.json was written by. vLLM parses the config with its own Qwen3_5Config, so this is really about the Qwen3-VL processor.

Launch commands

Low latency

NVFP4, TP1:

vllm serve Inferact/Qwen3.8-27B-NVFP4 \
  --tensor-parallel-size 1 \
  --max-model-len 262144 \
  --kv-cache-dtype fp8 \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice --tool-call-parser qwen3_coder

FP8, TP4 (one GB300 tray) for the largest KV cache:

vllm serve Qwen/Qwen3.8-27B-FP8 \
  --tensor-parallel-size 4 \
  --max-model-len 262144 \
  --kv-cache-dtype fp8 \
  --reasoning-parser qwen3

Add --speculative-config '{"method":"mtp","num_speculative_tokens":3}' for MTP.

2x RTX 5090 (consumer Blackwell, sm120)

Verified on vLLM 0.26.1rc1.dev608+g99a10304d, TP2 across two cards.

NVFP4 uses the real kernel here. vLLM selects FlashInferCutlassNvFp4LinearKernel for NVFP4 GEMM on sm120 — a cutlass path, not an emulation fallback — for both the Inferact build above and unsloth/Qwen3.8-27B-NVFP4.

The block-scaled FP8 checkpoint needs no workaround. vLLM auto-disables DeepGemm for model_type=qwen3_5_text on Blackwell and falls back to CUTLASS, so it loads unaided. Verified by running without VLLM_USE_DEEP_GEMM=0: identical 377,456-token KV pool.

vllm serve unsloth/Qwen3.8-27B-NVFP4 \
  --tensor-parallel-size 2 \
  --max-model-len 262144 \
  --kv-cache-dtype fp8 \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice --tool-call-parser qwen3_coder

At 262,144 context, GPU KV cache size as reported at startup:

precision	KV tokens	weights/GPU	MTP acceptance
FP8	377,456	14.28 GiB	0.771
NVFP4 (Inferact)	445,875	12.02 GiB	0.897
NVFP4 (unsloth)	920,517	10.64 GiB	0.788

The two NVFP4 builds are not interchangeable on 32 GB cards: unsloth is mixed-precision (FP8 channel-wise alongside the 4-bit groups) and leaves room for roughly twice the KV cache, while the uniform-W4A4 Inferact build drafts better. Both serve correctly.

The in-checkpoint MTP head works in every precision above. Acceptance is measured from vllm:spec_decode_num_{accepted,draft}_tokens_total, since throughput alone cannot distinguish a working drafter from one that loaded and was ignored.

1x RTX 5090 — NVFP4 needs --enforce-eager

One card has 31.4 GiB usable, not 32, and NVFP4 fits only with CUDA graphs off:

vllm serve Inferact/Qwen3.8-27B-NVFP4 \
  --tensor-parallel-size 1 \
  --max-model-len 32768 \
  --kv-cache-dtype fp8 \
  --enforce-eager \
  --reasoning-parser qwen3

Without --enforce-eager, startup dies in CUDA graph capture with torch.OutOfMemoryError: Tried to allocate 784.00 MiB. Raising or lowering --gpu-memory-utilization does not help — 0.80 and 0.93 both leave the same 47.06 MiB free, because that budget covers weights and KV while graph capture allocates outside it.

KV pool at 32K context: 91,022 tokens with --enforce-eager alone; 135,926 adding --language-model-only; 152,917 also capping --max-num-seqs 8; 76,458 with bf16 KV instead of fp8 — so fp8 KV is a choice here, not a requirement. The MTP head still fits (90,112 tokens, 0.754 acceptance). Those two flags are levers for a bigger pool, not fixes for the OOM: only --enforce-eager decides whether the server starts.

Client usage

generation_config.json ships temperature: 1.0, top_p: 0.95, top_k: 20.

from openai import OpenAI

client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1", timeout=3600)

resp = client.chat.completions.create(
    model="Qwen/Qwen3.8-27B",
    messages=[{"role": "user", "content": "Give me three primes above 100."}],
    temperature=1.0, top_p=0.95, max_tokens=2048,
)
print(resp.choices[0].message.content)

Thinking modes

The model supports no-think and adaptive thinking through chat_template_kwargs, per request or server-wide via --default-chat-template-kwargs:

• {"enable_thinking": false} — no thinking, the model answers directly.
• {"reasoning_effort": "low"} — adaptive thinking. xhigh (default), medium, low.

Processing Ultra-Long Texts

The model has 262k native context length and it can be extended to 1M with --max-model-len flag. The best value can be picked based on the use case and GPU ram usage tradeoff.

For example, to enable full 1M context length on this model:

vllm serve Qwen/Qwen3.8-27B \
  --max-model-len 1010000 \
  --hf-overrides '{"text_config": {"max_position_embeddings": 1010000}}' \
  ...

Note the override is nested under text_config here, where the 2.4T takes it flat.

Troubleshooting

MXFP4 does not load on Nvidia devices. The vLLM MXFP4 implementation on Nvidia device is currently missing linear method support so it doesn't run as intended. Use NVFP4 quantization on Nvidia instead.

References

• Model card: https://huggingface.co/Qwen/Qwen3.8-27B
• FP8 checkpoint: https://huggingface.co/Qwen/Qwen3.8-27B-FP8
• NVFP4 build (NVIDIA): https://huggingface.co/Inferact/Qwen3.8-27B-NVFP4
• vLLM documentation: https://docs.vllm.ai/