{
  "hf_id": "mistralai/Voxtral-4B-TTS-2603",
  "meta": {
    "title": "Voxtral-4B-TTS-2603",
    "slug": "voxtral-4b-tts-2603",
    "provider": "Mistral AI",
    "description": "Mistral's 4B text-to-speech model served via vLLM-Omni with built-in voice presets, exposed through the OpenAI /v1/audio/speech API (24 kHz mono).",
    "date_added": "2026-06-17",
    "date_updated": "2026-06-15",
    "tasks": [
      "omni"
    ],
    "related_recipes": [
      "Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice",
      "mistralai/Voxtral-Mini-4B-Realtime-2602",
      "fishaudio/s2-pro"
    ]
  },
  "model": {
    "model_id": "mistralai/Voxtral-4B-TTS-2603",
    "min_vllm_version": "0.20.0",
    "architecture": "dense",
    "parameter_count": "4B",
    "active_parameters": "4B",
    "context_length": 0,
    "base_args": [
      "--omni"
    ],
    "base_env": {},
    "install": {
      "pip": {
        "command": "uv venv\nsource .venv/bin/activate\nuv pip install -U vllm --torch-backend auto"
      },
      "docker": {
        "command": "docker pull vllm/vllm-openai:latest"
      },
      "extras": [
        {
          "note": "vLLM-Omni provides the TTS serving backend and the `--omni` serve path. See the installation guide for pre-built wheels or building from source.",
          "command": "uv pip install git+https://github.com/vllm-project/vllm-omni.git"
        },
        {
          "note": "Voxtral's tokenizer/audio path requires mistral_common >= 1.10.0",
          "command": "uv pip install -U \"mistral_common>=1.10.0\""
        }
      ]
    }
  },
  "omni": {
    "tasks": [
      {
        "id": "t2a",
        "label": "Text → Speech",
        "endpoint": "/v1/audio/speech",
        "description": "Voxtral TTS with a built-in voice preset via /v1/audio/speech",
        "curl": "curl -X POST http://localhost:8000/v1/audio/speech \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\n    \"model\": \"mistralai/Voxtral-4B-TTS-2603\",\n    \"input\": \"Hello, this is Voxtral TTS running with vLLM-Omni.\",\n    \"voice\": \"casual_female\",\n    \"language\": \"English\",\n    \"response_format\": \"wav\"\n  }' --output voxtral.wav\n"
      }
    ]
  },
  "features": {},
  "opt_in_features": [],
  "variants": {
    "default": {
      "precision": "bf16",
      "vram_minimum_gb": 24,
      "description": "BF16 weights — fits a single 24 GB GPU (observed server-startup peak ~20.5 GiB on an RTX 4090)"
    }
  },
  "hardware_overrides": {},
  "guide": "## Overview\n\n[Voxtral-4B-TTS-2603](https://huggingface.co/mistralai/Voxtral-4B-TTS-2603) is\nMistral AI's 4B text-to-speech model, served through **vLLM-Omni** with the\nOpenAI-compatible `/v1/audio/speech` API. It generates **24 kHz mono** audio\nfrom text using model-provided voice presets.\n\n## Prerequisites\n\n- **Hardware**: a single CUDA GPU with **≥ 24 GB** VRAM. Reference run: 1×\n  RTX 4090 24 GB — Stage 0 (`audio_generation`) ~18.95 GiB, Stage 1\n  (`audio_tokenizer`) ~1.55 GiB, server-startup peak ~20.5 GiB (~85%). Both\n  stages share GPU 0 via the bundled deploy config.\n- **vLLM-Omni** targeting vLLM >= 0.20, with `mistral_common >= 1.10.0`.\n\n## Installation\n\n```bash\nuv venv && source .venv/bin/activate\nuv pip install git+https://github.com/vllm-project/vllm-omni.git\nuv pip install -U \"mistral_common>=1.10.0\"\n```\n\n## Launch the server\n\nThe deploy config (`vllm_omni/deploy/voxtral_tts.yaml`) is auto-discovered:\n\n```bash\nvllm serve mistralai/Voxtral-4B-TTS-2603 --omni --port 8000\n```\n\n## Client usage\n\n### Voice preset (curl)\n\n```bash\ncurl -X POST http://127.0.0.1:8000/v1/audio/speech \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\n    \"model\": \"mistralai/Voxtral-4B-TTS-2603\",\n    \"input\": \"Hello, this is Voxtral TTS running with vLLM-Omni.\",\n    \"voice\": \"casual_female\",\n    \"language\": \"English\",\n    \"response_format\": \"wav\"\n  }' --output voxtral.wav\n```\n\n### Streaming PCM\n\n```bash\ncurl -X POST http://127.0.0.1:8000/v1/audio/speech \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\n    \"model\": \"mistralai/Voxtral-4B-TTS-2603\",\n    \"input\": \"Hello, this is Voxtral TTS streaming PCM.\",\n    \"voice\": \"casual_female\",\n    \"language\": \"English\",\n    \"stream\": true,\n    \"response_format\": \"pcm\"\n  }' --output voxtral_stream.pcm\n\n# Convert raw 24 kHz mono PCM to WAV\nffmpeg -f s16le -ar 24000 -ac 1 -i voxtral_stream.pcm voxtral_stream.wav -y\n```\n\n### Examples\n\n- [Offline end-to-end](https://github.com/vllm-project/vllm-omni/blob/main/examples/offline_inference/text_to_speech/voxtral_tts/end2end.py)\n- [Online Gradio demo](https://github.com/vllm-project/vllm-omni/blob/main/examples/online_serving/text_to_speech/voxtral_tts/gradio_demo.py)\n\n## Known limitations\n\n- Output is **mono 24 kHz**.\n- **Voice cloning is gated upstream.** With the public\n  `mistralai/Voxtral-4B-TTS-2603` checkpoint, supplying a `ref_audio` (or\n  running the voice-clone path) fails with\n  `RuntimeError: encode_waveforms requires encoder weights which are not\n  available in the open-source checkpoint.` — the public weights omit the\n  encoder needed to turn reference audio into conditioning features. Use the\n  built-in voice presets instead.\n\n## References\n\n- [Model card](https://huggingface.co/mistralai/Voxtral-4B-TTS-2603)\n- [vLLM-Omni text-to-speech examples](https://github.com/vllm-project/vllm-omni/tree/main/examples/online_serving/text_to_speech/voxtral_tts)\n",
  "source": {
    "source": "vllm-omni",
    "repo": "vllm-project/vllm-omni",
    "ref": "main",
    "sha": "9dd5c74af1e32095129085055988517f1369f210",
    "path": "recipes/models/mistralai/Voxtral-4B-TTS-2603.yaml",
    "edit_url": "https://github.com/vllm-project/vllm-omni/edit/main/recipes/models/mistralai/Voxtral-4B-TTS-2603.yaml",
    "overlays": []
  }
}