# openai/gpt-oss-120b — vLLM serving recipe

> OpenAI's gpt-oss family (20B / 120B) with MXFP4 MoE, attention-sinks, built-in tools via Responses API

Configuration: hardware `h200` (8× H200) · strategy `single_node_tp` (Tensor Parallel) · variant `default` (MXFP4) · features `tool_calling`

- Interactive page: https://recipes.vllm.ai/openai/gpt-oss-120b
- This configuration as JSON: https://recipes.vllm.ai/openai/gpt-oss-120b.json?hardware=h200

The commands below were rendered for exactly this configuration and take precedence over any example in the guide.

## Other configurations

Change the query string (the same parameters as the page URL) and fetch this URL again. Unknown or incompatible values fall back to the recipe default, so check the Configuration line.

- `hardware`: `h100`, `h200`, `b200`, `gb200`, `b300`, `gb300`, `mi300x`, `mi325x`, `mi350x`, `mi355x`, `ironwood`, `arc_pro_b70`
- `variant`: `default`, `amd_fp8`
- `strategy`: `single_node_tp`, `multi_node_tp`, `multi_node_tp_pp`, `multi_node_tep`, `multi_node_dep` (multi-node strategies also need `nodes=<n>`)
- `features`: comma-separated list of `tool_calling`, `spec_decoding` (default: `tool_calling`)

## Install

Requires vLLM >= 0.10.1.

```bash
uv venv
source .venv/bin/activate
uv pip install -U vllm --torch-backend auto
```

Docker image: `vllm/vllm-openai:latest`

## Serve

```bash
vllm serve openai/gpt-oss-120b \
  --tensor-parallel-size 1 \
  --tool-call-parser openai \
  --enable-auto-tool-choice
```

Docker equivalent:

```bash
docker run --gpus all \
  --privileged --ipc=host -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai:latest openai/gpt-oss-120b \
  --tensor-parallel-size 1 \
  --tool-call-parser openai \
  --enable-auto-tool-choice
```

## Guide

> Background reference. Commands in it are examples and may target another configuration — use the Serve section above.

### Overview

`gpt-oss-20b` and `gpt-oss-120b` are open-weight reasoning models from OpenAI. vLLM
supports NVIDIA H100/H200/B200, AMD MI300X/MI325X/MI355X, and Radeon AI PRO R9700,
with ongoing work for Ampere/Ada/RTX 5090.

Optimizations:
- Flexible parallelism (TP 2/4/8)
- Attention kernels for attention-sinks and sliding-window shapes
- Asynchronous scheduling for CPU/GPU overlap

### Prerequisites

- Hardware: NVIDIA H100/H200/B200 (or A100 80GB for single-GPU), AMD MI300+
- vLLM >= 0.10.1
- CUDA >= 12.8 if building from source (must match between install and serving)

#### Install vLLM

```bash
uv venv
source .venv/bin/activate
uv pip install vllm --torch-backend=auto
```

Docker quickstart:

```bash
docker run --gpus all -p 8000:8000 --ipc=host vllm/vllm-openai --model openai/gpt-oss-20b
```

AMD ROCm wheels:

```bash
uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm
```

Intel XPU (Arc Pro B70):

```bash
docker pull vllm/vllm-openai-xpu:latest
```

### Launch commands

A100 (single card, default TRITON_ATTN + Marlin MXFP4 MoE):

```bash
vllm serve openai/gpt-oss-120b
vllm serve openai/gpt-oss-120b --tensor-parallel-size 4
```

Blackwell (B200) with FlashInfer MXFP4+MXFP8 MoE:

```bash
vllm serve openai/gpt-oss-120b --config GPT-OSS_Blackwell.yaml --tensor-parallel-size 1
```

On vLLM v0.28.0+ the FlashInfer TRTLLM MXFP4 MoE path is selected automatically on Blackwell; it can be pinned explicitly with `--moe-backend flashinfer_trtllm --quantization-config.moe.activation mxfp8`.

Hopper (H100/H200): same as Blackwell without `kv-cache-dtype` and without the FlashInfer MoE flags.

TPU v7 (Ironwood), verified on a `tpu7x-standard-4t` host (topology `2x2x1`, TP=2 × DP=4). Served via the OpenAI API-server entrypoint from the nightly TPU image (e.g. `vllm/vllm-tpu:nightly-20260316-e778980-57a314d`):

```bash
export TPU_BACKEND_TYPE=jax
export MODEL_IMPL_TYPE=vllm
export HF_HOME=/data
export HUGGING_FACE_HUB_TOKEN=<your HF token>
vllm serve openai/gpt-oss-120b \
  --host 0.0.0.0 \
  --port 8000 \
  --seed 42 \
  --tensor-parallel-size 2 \
  --data-parallel-size 4 \
  --max-model-len 9216 \
  --max-num-batched-tokens 16384 \
  --max-num-seqs 2048 \
  --gpu-memory-utilization 0.86 \
  --kv-cache-dtype fp8 \
  --load-format runai_streamer \
  --no-enable-prefix-caching \
  --async-scheduling
```

AMD MI300X/MI325X:

```bash
export HSA_NO_SCRATCH_RECLAIM=1
export AMDGCN_USE_BUFFER_OPS=0
export VLLM_ROCM_USE_AITER=1
export VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4

vllm serve openai/gpt-oss-120b \
  --tensor-parallel-size 8 \
  --attention-backend ROCM_AITER_UNIFIED_ATTN \
  -cc.pass_config.fuse_rope_kvcache=True \
  -cc.use_inductor_graph_partition=True \
  --gpu-memory-utilization 0.95 \
  --block-size 64
```

Intel Arc Pro B70 (4x 32 GB, TP=4), MXFP4 weights via the official XPU image:

```bash
docker run --device /dev/dri \
  -v /dev/dri/by-path:/dev/dri/by-path --shm-size=16g \
  --privileged --ipc=host -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -e TORCH_LLM_ALLREDUCE=1 \
  -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 \
  -e VLLM_WORKER_MULTIPROC_METHOD=spawn \
  vllm/vllm-openai-xpu:latest \
  openai/gpt-oss-120b \
  --tensor-parallel-size 4 \
  --block-size 64 \
  --enforce-eager \
  --disable-sliding-window \
  --no-enable-prefix-caching \
  --max-model-len 9216 \
  --max-num-batched-tokens 9216 \
  --gpu-memory-utilization 0.9
```

### Tool Use

`/v1/responses` endpoint supports built-in tools (browsing, python, MCP). Setup
requires `uv pip install gpt-oss` and either docker (for Python sandbox) or
`PYTHON_EXECUTION_BACKEND=dangerously_use_uv`. For demo tools:

```bash
vllm serve ... --tool-server demo
```

For user-defined function calling:

```bash
vllm serve ... --tool-call-parser openai --enable-auto-tool-choice
```

### Client Usage

```python
from openai import OpenAI

client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1")
response = client.chat.completions.create(
    model="openai/gpt-oss-120b",
    messages=[{"role": "user", "content": "Explain sinks attention."}],
)
print(response.choices[0].message.content)
```

### Accuracy Evaluation

OpenAI recommends evaluating with the gpt-oss reference library:

```bash
vllm serve openai/gpt-oss-120b \
  --tensor_parallel_size 8 --max-model-len 131072 \
  --max-num-batched-tokens 10240 --max-num-seqs 128 \
  --gpu-memory-utilization 0.85 --no-enable-prefix-caching

mkdir -p /tmp/gpqa_openai
OPENAI_API_KEY=empty python -m gpt_oss.evals \
  --model openai/gpt-oss-120b --eval gpqa --n-threads 128
```

Reproduced scores (120B): Low 65.3 / 51.2; Mid 72.4 / 79.6; High 79.4 / 93.0 (GPQA / AIME25).

### Troubleshooting

- **Attention sinks dtype error on Blackwell:** pass `--quantization-config.moe.activation mxfp8` and `--kv-cache-dtype fp8`.
- **`tl.language not defined`:** make sure no extra Triton (e.g., pytorch-triton) is installed.
- **H100 TP1 OOM:** `--gpu-memory-utilization 0.95 --max-num-batched-tokens 1024`.
- **Harmony vocab download failure:** pre-download tiktoken files and set `TIKTOKEN_ENCODINGS_BASE`.

### Known Limitations

- Responses API: streaming is basic, annotations/citations unsupported, usage accounting returns zeros.
- Function calling currently supports only `tool_choice="auto"`.

### References

- [gpt-oss-120b](https://huggingface.co/openai/gpt-oss-120b)
- [gpt-oss-20b](https://huggingface.co/openai/gpt-oss-20b)
- [amd/gpt-oss-120b-w-mxfp4-a-fp8](https://huggingface.co/amd/gpt-oss-120b-w-mxfp4-a-fp8)
- [Eagle3 draft model](https://huggingface.co/nvidia/gpt-oss-120b-Eagle3-v3)
- [gpt-oss-120b on Ironwood (TPU v7)](https://github.com/AI-Hypercomputer/tpu-recipes/tree/main/inference/ironwood/vLLM/GPT-OSS)