Speaches Documentation

Speaches is an OpenAI API-compatible server designed for streaming transcription, translation, and speech generation. It functions similarly to Ollama but is specialized for TTS (Text-to-Speech) and STT (Speech-to-Text) models.

Key capabilities include:

• OpenAI Compatibility: Works with any tool or SDK designed for OpenAI's API.
• Audio Generation: Supports text-to-audio summaries, sentiment analysis on recordings, and async speech-to-speech interactions via the chat completions endpoint.
• Streaming Support: Transcription is delivered via Server-Sent Events (SSE) as it happens, eliminating the need to wait for the full audio file to process.
• Dynamic Model Management: Models are loaded automatically based on the request and unloaded after a period of inactivity.
• Engine Support: Uses faster-whisper for STT, and piper or Kokoro for TTS.
• Hardware Support: Compatible with both GPU and CPU.

You can specify which models are used for transcription and speech generation by passing additional parameters to the chat completion endpoint.

When using the OpenAI Python SDK, these parameters must be passed via the extra_body argument:

• transcription_model: The model used to transcribe input audio.
• speech_model: The model used to generate output audio.

openai_client.chat.completions.create(
        model="gpt-4o-mini",
        modalities=["text", "audio"],
        audio={"voice": "alloy", "format": "wav"},
        stream=False,
        messages=[
            {
                "role": "user",
                "content": [
                    {"type": "text", "text": "What is in this recording?"},
                    {"type": "input_audio", "input_audio": {"data": "<bytes ommitted>", "format": "wav"}},
                ],
            },
        ],
        extra_body={"transcription_model": "Systran/faster-whisper-tiny.en", "speech_model": "hexgrad/Kokoro-82M"}
    )

To run Speaches using Docker Compose, download the appropriate YAML configuration files based on your hardware (CUDA or CPU) and set the COMPOSE_FILE environment variable before starting the service.

CUDA Setup

curl --silent --remote-name https://raw.githubusercontent.com/speaches-ai/speaches/master/compose.yaml
curl --silent --remote-name https://raw.githubusercontent.com/speaches-ai/speaches/master/compose.cuda.yaml
export COMPOSE_FILE=compose.cuda.yaml

CUDA with CDI feature enabled

curl --silent --remote-name https://raw.githubusercontent.com/speaches-ai/speaches/master/compose.yaml
curl --silent --remote-name https://raw.githubusercontent.com/speaches-ai/speaches/master/compose.cuda.yaml
curl --silent --remote-name https://raw.githubusercontent.com/speaches-ai/speaches/master/compose.cuda-cdi.yaml
export COMPOSE_FILE=compose.cuda-cdi.yaml

CPU Setup

curl --silent --remote-name https://raw.githubusercontent.com/speaches-ai/speaches/master/compose.yaml
curl --silent --remote-name https://raw.githubusercontent.com/speaches-ai/speaches/master/compose.cpu.yaml
export COMPOSE_FILE=compose.cpu.yaml

After setting the environment variable, start the service with:

docker compose up --detach

Use the speaches-cli to list, download, and verify Speech-to-Text (STT) models. Ensure SPEACHES_BASE_URL is set to your server address before running commands.

• List available STT models: Use registry ls with the --task automatic-speech-recognition flag.
• Download a model: Use model download <model_id>.
• Verify installation: Use model ls with the --task text-to-speech flag (Note: the documentation uses text-to-speech in the example to check for the model, ensure you use the correct task for your needs) and filter the output.

export SPEACHES_BASE_URL="http://localhost:8000"

# Listing all available STT models
uvx speaches-cli registry ls --task automatic-speech-recognition | jq '.data | [].id'

# Downloading a Systran/faster-distil-whisper-small.en model
uvx speaches-cli model download Systran/faster-distil-whisper-small.en

# Check that the model has been installed
uvx speaches-cli model ls --task text-to-speech | jq '.data | map(select(.id == "Systran/faster-distil-whisper-small.en"))'