DOCSEARCHbyAlgolia
Submit Docs

Wyoming OpenAI

repository·main·Indexed 41 days ago

https://github.com/roryeckel/wyoming_openai

An OpenAI-compatible proxy middleware for the Wyoming protocol (v0.6.1) that enables Wyoming clients, such as Home Assistant, to use various ASR and TTS services. It supports service consolidation, asynchronous processing, and streaming compatibility via yasbd. It can be deployed with official OpenAI services or local alternatives including Speaches, LocalAI, Kokoro-FastAPI, Mistral AI Voxtral, Microsoft Edge TTS, Chatterbox, and Groq.

Tokens
13.2K
Snippets
27
Records
52
Agent score
51%

What's inside Wyoming OpenAI

  1. Overview of Wyoming OpenAI

    main

    Wyoming OpenAI is an OpenAI-compatible proxy middleware for the Wyoming protocol. It acts as a bridge between Wyoming clients (such as the Home Assistant Wyoming Integration) and OpenAI-compatible endpoints for Automatic Speech Recognition (ASR/STT) and Text-to-Speech (TTS).

    Key features include:

    • Service Consolidation: Allows multiple programs (e.g., Open WebUI and Home Assistant) to share a single inference server.
    • Streaming Compatibility: Uses yasbd to chunk text at sentence boundaries, enabling responsive incremental audio delivery even for APIs that do not natively support streaming text input.
    • Asynchronous Processing: Supports concurrent pipelining (defaulting to three in-flight requests) to keep audio playback smooth while maintaining event order.

  2. How Wyoming OpenAI communicates with Home Assistant

    main

    Wyoming OpenAI acts as a proxy between Home Assistant and an OpenAI-compatible API.

    Speech-to-Text (STT/ASR) Flow

    Home Assistant sends audio via the Wyoming Protocol. The proxy buffers the WAV PCM chunks and then communicates with the OpenAI-compatible API:

    • Non-Streaming: The proxy uploads the complete audio file; the API returns a full text transcript.
    • Streaming: The proxy sends audio with stream=true; the API returns transcript deltas, which the proxy forwards to Home Assistant as TranscriptChunk events.

    Text-to-Speech (TTS) Flow

    • Non-Streaming (Synthesize): Home Assistant sends a Synthesize event. The proxy requests speech synthesis from the API and streams the resulting audio chunks back to HA.
    • Streaming (SynthesizeStart/Chunk/Stop): This uses yasbd for sentence boundary detection. As Home Assistant sends SynthesizeChunk events, the proxy accumulates text, detects complete sentences, and prefetches audio for those sentences (up to three concurrent OpenAI TTS requests) to provide low-latency incremental audio playback.

  3. Understand the Wyoming OpenAI Compatibility Contract

    main

    Wyoming OpenAI acts as a proxy for the Wyoming protocol by targeting the OpenAI-compatible surface of upstream providers.

    It does not attempt to support every custom API, route, or dialect offered by a provider. Instead, it relies on the standard OpenAI endpoints. If a provider only offers a feature (like streaming) via a custom, non-OpenAI route, Wyoming OpenAI will treat that feature as out of scope. Compatibility is determined by whether the provider's OpenAI-compatible routes implement the expected request schemas, response formats, and streaming behaviors.

  4. How SSML is handled during synthesis

    main

    When using Wyoming with text_format=ssml, Wyoming OpenAI projects the SSML to backend-safe plain text before sending it upstream.

    Key behaviors:

    • Unsupported Tags: Markup for prosody, voice, emphasis, say-as, and audio are not implemented or forwarded.
    • Preserved Boundaries: The tags break, p, and s (as well as complete unknown/custom tags) preserve one spoken word boundary.
    • Removal: Known inline tags are removed transparently.
    • Incremental Projection: The projection is incremental, meaning splitting SSML across synthesis chunks does not change the resulting text.

  5. Deploy Wyoming OpenAI with Mistral AI Voxtral (STT Only)

    main

    Use Mistral AI's Voxtral for high-quality, multilingual speech transcription. This is an STT-only solution.

    Key Details:

    • Capabilities: High accuracy, multilingual support, and designed for long-form inputs (32k token context).
    • Requirements: Requires a Mistral API key.
    • Integration: Uses OpenAI-compatible API endpoints.

    Deployment Command:

    docker compose -f docker-compose.voxtral.yml up -d

  6. Deploy Wyoming OpenAI with Official OpenAI Services

    main

    To use official OpenAI APIs for Speech-to-Text (STT) and Text-to-Speech (TTS), follow these steps:

    1. Environment Variables: Create a .env file in your project directory containing your API keys:
      • STT_OPENAI_KEY: Your OpenAI API key for STT.
      • TTS_OPENAI_KEY: Your OpenAI API key for TTS.
    2. Configuration: Use the provided docker-compose.yml template. This setup binds the Wyoming server to port 10300.
    3. Deployment: Run the following command to start the services in detached mode.
    docker compose -f docker-compose.yml up -d

  7. Use Wyoming OpenAI in Home Assistant

    main

    To integrate Wyoming OpenAI with Home Assistant (HA), follow these steps:

    1. Deploy the instance: Install and set up your Wyoming OpenAI instance using your preferred deployment method.
    2. Add the Integration: In Home Assistant, navigate to Settings > Devices & Services > Add Integration. Search for and add the Wyoming Protocol integration, providing the URI of your Wyoming OpenAI instance.
    3. Configure Voice Assistant: Update your Home Assistant Voice Assistant pipeline to use the STT (Speech-to-Text) and TTS (Text-to-Speech) services provided by the Wyoming OpenAI instance.

    Reloading Configuration: If you update models, voices, or URLs, you must reload the integration in Home Assistant to apply changes:

    1. Go to Settings > Devices & Services.
    2. Select the Wyoming OpenAI integration.
    3. Click Reload.

  8. Deploy Wyoming OpenAI with Groq (Fast Inference)

    main

    Use Groq's LPU architecture for ultra-fast cloud-based inference for both STT and TTS.

    Key Details:

    • STT: Uses Whisper Large V3.
    • TTS: Uses Orpheus TTS, which supports emotion control via inline tags like [cheerful] or [whisper].
    • Free Tier: Supports up to 2,000 STT requests/day and 100 TTS requests/day.
    • Requirements: Requires a Groq API key.
    • Environment Variables: Set your Groq API key as STT_OPENAI_KEY and/or TTS_OPENAI_KEY in a .env file.

    Deployment Command:

    docker compose -f docker-compose.groq.yml up -d

  9. Deploy Wyoming OpenAI with Chatterbox TTS

    main

    Chatterbox provides a local, OpenAI-compatible API for high-quality text-to-speech with advanced voice cloning.

    Key Details:

    • Capabilities: Supports custom voice training and personalized speech generation.
    • Environment: Runs completely offline with no external API dependencies.
    • Performance: Supports GPU acceleration and real-time streaming.

    Deployment Command:

    docker compose -f docker-compose.chatterbox.yml up -d

  10. Deploy Wyoming OpenAI with LocalAI Local Service

    main

    LocalAI provides a completely local, privacy-focused replacement for OpenAI, supporting Whisper (STT) and Piper (TTS).

    Key Details:

    • Models: Automatically downloads whisper-base and multiple Piper TTS voices on the first run.
    • API: Provides OpenAI-compatible endpoints.
    • Keys: No API keys are required.
    • GPU: The example uses a GPU-accelerated image with NVIDIA CUDA 12 support.

    Deployment Command:

    docker compose -f docker-compose.localai.yml up -d

  11. Configure Wyoming OpenAI via environment variables

    main

    The Wyoming OpenAI proxy server can be configured using either command-line arguments or environment variables. Environment variables are recommended for containerized deployments.

    STT Transport Modes

    • Standard STT: Uses /v1/audio/transcriptions with response streaming after Wyoming AudioStop is received. Controlled by --stt-models.
    • Streaming STT: Uses /v1/audio/transcriptions with response streaming. Controlled by --stt-streaming-models.
    • Realtime STT: Opens a /v1/realtime session, sending 24 kHz mono PCM16 audio chunks as they arrive and emitting TranscriptChunk deltas. Controlled by --stt-realtime-models.

    Extra Body Configuration

    Both STT_EXTRA_BODY and TTS_EXTRA_BODY must be valid JSON objects. The values are merged directly into the outgoing request body.

    • STT: response_format must remain json. A boolean stream override is supported.
    • TTS: Can override response_format, speed, and instructions, but stream and stream_format are rejected as the handler expects audio bytes.

  12. Develop Wyoming OpenAI with Docker

    main

    To build the Wyoming OpenAI proxy server from source for development, combine the base configuration with the development configuration.

    Deployment Command:

    docker compose -f docker-compose.yml -f docker-compose.dev.yml up -d --build

    You can also combine the development configuration with any of the local service templates (e.g., Speaches, LocalAI, Groq, etc.).

    Example (Development + Speaches):

    docker compose -f docker-compose.speaches.yml -f docker-compose.dev.yml up -d --build