# adventists-ai/Sophon-Realtime-32B-Tag — vLLM serving recipe

> Open full-duplex voice model that hears your tone and speaks with one: listens while you talk, takes interruptions, and laughs, sighs or stresses words when the conversation calls for it. Zero-shot voice cloning from a 3-10 s clip. One vLLM-Omni model (thinker -> expressive talker -> vocoder) on a single 80 GB GPU, with an OpenAI-Realtime-style full-duplex WebSocket and /v1/chat/completions (audio in, text + audio out).

Configuration: hardware `h200` (8× H200) · variant `default` (FP8)

- Interactive page: https://recipes.vllm.ai/adventists-ai/Sophon-Realtime-32B-Tag
- This configuration as JSON: https://recipes.vllm.ai/adventists-ai/Sophon-Realtime-32B-Tag.json?hardware=h200

The commands below were rendered for exactly this configuration and take precedence over any example in the guide.

## Other configurations

Change the query string (the same parameters as the page URL) and fetch this URL again. Unknown or incompatible values fall back to the recipe default, so check the Configuration line.

- `hardware`: `h100`, `h200`, `b200`, `gb200`, `b300`, `gb300`, `mi300x`, `mi325x`, `mi350x`, `mi355x`
- `variant`: `default`

## Install

Requires vLLM >= 0.30.0.

```bash
uv venv
source .venv/bin/activate
uv pip install -U vllm --torch-backend auto
```

Extra dependencies:

- Out-of-tree model: launch with `sophon-realtime serve`, not `vllm serve`. The Sophon-Realtime plugin pins vLLM 0.30.0 and vLLM-Omni 0.30.0 exactly (newer versions are not supported yet), registers the model, and `sophon-realtime serve` downloads the weights and starts the server with the right deploy config. Keep the `[trt]` extra: it installs TensorRT for the talker; the latency table below was measured with it.

```bash
uv pip install "sophon-realtime[trt]"
```

## Serve

vLLM-Omni recipe — the generation endpoints and request examples are in the guide below.

```bash
vllm serve adventists-ai/Sophon-Realtime-32B-Tag \
  --omni
```

## Guide

> Background reference. Commands in it are examples and may target another configuration — use the Serve section above.

### Overview

[Sophon-Realtime-32B-Tag](https://huggingface.co/adventists-ai/Sophon-Realtime-32B-Tag) is the recommended build of
Sophon-Realtime v1, an open full-duplex voice model. A 32B thinker listens to the user's audio, decides when it is its
turn and writes the reply with inline sound tags (`[laughter]`, `[sigh]`, `<strong>…</strong>`); a streaming talker
performs those tags while the reply is still being written. The whole pipeline is one vLLM-Omni model.
VoiceBench 82.8, Big Bench Audio 81.8%, NVV-SuperBench 0.86 (zh) / 0.80 (en).

**Out-of-tree model.** Sophon-Realtime is not built into vLLM-Omni. It is served through its own plugin package,
`sophon-realtime`, which pins vLLM 0.30.0 and vLLM-Omni 0.30.0 exactly (newer versions are not supported yet) and
registers the model architecture. A plain
`vllm serve ... --omni` on a stock vLLM-Omni install will not load it: install the plugin and start with
`sophon-realtime serve` as shown below.

### Prerequisites

- One 80 GB GPU (H100 / H800 / A100 80G); about 65 GB per instance.
- Linux, CUDA, Python 3.12.

### Installation and launch

```bash
uv venv --python 3.12 && source .venv/bin/activate
uv pip install "sophon-realtime[trt]"   # [trt] = TensorRT for the talker; without it the talker runs in PyTorch and is slower under load
sophon-realtime serve --port 8000       # downloads the weights (~42 GB) on first run
```

`sophon-realtime serve` starts a CUDA MPS daemon for its GPU by default, so the three stages share the GPU
concurrently. MPS applies to every CUDA process on that GPU, and the daemon keeps running after the server exits
(`serve` prints the command to stop it). On a shared GPU, or with other CUDA workloads on the same device, set
`SOPHON_MPS=0`. After the weights are downloaded, the first start builds kernels and TensorRT engines (about 7–8 min
on one H100 with sophon-realtime 1.1.2); later starts take about 4–5 min.

### Zero-shot voice cloning

Put a 3–10 s clip of one speaker at `<model dir>/voices/<name>.wav` (wav or flac, any sample rate; or point
`SOPHON_VOICES` at another folder) and set
`"voice": "<name>"` in the realtime session or the chat request. No fine-tuning or restart; new clips are picked up
on the next request. Only clone your own voice or voices you have permission to use.

### Client usage

- Full duplex: `ws://localhost:8000/v1/realtime` (OpenAI-Realtime-style: `input_audio_buffer.append`; the model decides
  when the user has finished and handles barge-in).
- One-shot: `POST /v1/chat/completions` with `input_audio`, `"modalities": ["text","audio"]`.
- OpenAI GPT-Live / OpenAI Realtime compatible gateway (LiveKit, Pipecat, function calling):
  [live/](https://github.com/adventists-ai/sophon-realtime/tree/main/live).

### Performance (one H100 80 GB)

`sophon-realtime` 1.1.2 from a clean `pip install "sophon-realtime[trt]"`, default `serve` (TensorRT and CUDA MPS on),
`/v1/chat/completions` with a short spoken question (replies of about 5 s of speech), users arriving within 10 s,
measured after warm-up. First audio = time from sending the request to the first audio chunk.

| concurrent users | first audio median / max |
|---|---|
| 1 | 0.61 s |
| 8 | 0.64 / 0.70 s |
| 16 | 0.69 / 0.98 s |
| 24 | 0.72 / 1.02 s |

With longer replies (16 public spoken questions, about 20 s of speech per reply), first audio at 16 users is
0.87 / 1.76 s. The first one or two requests after a start can take several seconds while the server finishes
warming up.

### References

- [Model card](https://huggingface.co/adventists-ai/Sophon-Realtime-32B-Tag) · [GitHub](https://github.com/adventists-ai/sophon-realtime)
- Apache-2.0