vLLM/Recipes
StepFun

stepfun-ai/Step-5-Preview

Multimodal MoE with hybrid sparse/sliding-window attention, a 1M-token context window, and MTP.

View on HuggingFace
moe604B1,048,576 ctxvLLM +multimodal
Guide

Overview

Step-5 Preview is a multimodal MoE model with hybrid sparse/sliding-window attention, a 1M-token context window, and multi-token prediction (MTP).

The NVFP4 checkpoint is available through the NVFP4 variant below.

Docker

docker pull vllm/vllm-openai:step-5-preview

docker run --rm -it \
    --gpus all \
    --ipc=host \
    --network=host \
    --volume "$HOME/.cache/huggingface:/root/.cache/huggingface" \
    --entrypoint /bin/bash \
    vllm/vllm-openai:step-5-preview

Run the following commands inside the container. Model access is required; use hf auth login if the checkpoint requires authentication.

FP8: single node on 4 B300 GPUs

The examples use four B300 GPUs. Choose TP4, TEP4, or DEP4:

ModeTensor parallelData parallelExpert parallel
TP441Disabled
TEP4414
DEP4144
export CUDA_VISIBLE_DEVICES=0,1,2,3

On Blackwell, TEP4 and DEP4 use MegaMoE. TP4 uses the default MoE backend because MegaMoE requires expert parallelism.

For FP8 on H200, use eight GPUs: TP8, TEP8, or DEP8, respectively. Set CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 and change the parallel size from 4 to 8 (DEP keeps TP=1). Omit the MegaMoE flag on Hopper.

These are separate layouts: TP4 and TEP4 use DP=1; DEP4 uses TP=1. --max-num-seqs applies per DP rank. The 1M-token context limit includes input and output; reduce --max-model-len or concurrency if KV cache memory is insufficient.

NVFP4: single node on 4 B300 GPUs

Use stepfun-ai/Step-5-Preview-NVFP4 with the same Step-5 image. The checkpoint includes both NVFP4 and BF16 layers; quantization is detected from its config.

docker pull vllm/vllm-openai:step-5-preview

docker run --rm -it \
    --gpus 4 \
    --ipc=host \
    --network=host \
    --volume "$HOME/.cache/huggingface:/root/.cache/huggingface" \
    --entrypoint /bin/bash \
    vllm/vllm-openai:step-5-preview

Inside the container, reuse one of the TP4, TEP4, or DEP4 commands above, replacing the checkpoint with stepfun-ai/Step-5-Preview-NVFP4 and setting --max-model-len 320000. Keep --reasoning-parser step5 and --tool-call-parser step5 unchanged.

ModeParallel arguments
TP4--tensor-parallel-size 4
TEP4--tensor-parallel-size 4 --enable-expert-parallel --moe-backend deep_gemm_mega_moe
DEP4--tensor-parallel-size 1 --data-parallel-size 4 --enable-expert-parallel --moe-backend deep_gemm_mega_moe

TP4 uses automatic MoE backend selection for the mixed-precision layers. TEP4 and DEP4 use MegaMoE for routed experts; shared experts run separately. Send requests with "model": "stepfun-ai/Step-5-Preview-NVFP4".

Available context and concurrency depend on the remaining KV cache memory.

Send a request

curl http://127.0.0.1:8000/v1/chat/completions \
    --header 'Content-Type: application/json' \
    --data '{
      "model": "stepfun-ai/Step-5-Preview",
      "messages": [{"role": "user", "content": "Hello!"}],
      "max_tokens": 4096,
      "stream": true
    }'

References