DOCSEARCHbyAlgolia
Submit Docs

unmute

repository·main·Indexed 23 days ago

https://github.com/kyutai-labs/unmute

A low-latency system that integrates high-performance Speech-to-Text (STT) and Text-to-Speech (TTS) models with text-based LLMs to create real-time voice interaction experiences. It supports any text LLM via OpenRouter or self-hosted VLLM and uses a WebSocket communication protocol based on the OpenAI Realtime API (ORA). Version 0.1.0.

Tokens
14.8K
Snippets
27
Records
87
Agent score
80%

What's inside unmute

  1. What is Unmute?

    main

    Unmute is a system that enables text-based Large Language Models (LLMs) to listen and speak by integrating Kyutai's Speech-to-Text (STT) and Text-to-Speech (TTS) models.

    Workflow:

    1. STT: Transcribes user audio in real time.
    2. LLM: Generates a text response once speech is detected to have stopped.
    3. TTS: Reads the generated text aloud and streams the audio back to the user.

    The system is designed for low latency and can work with any text LLM (e.g., via OpenRouter or self-hosted VLLM).

  2. Understand the WebSocket Communication Protocol

    main

    The Unmute backend and frontend communicate via WebSockets using a protocol based on the OpenAI Realtime API (ORA).

    While Unmute attempts to match the ORA format, there are deviations including additional message types and simplified parameters.

    • Protocol Details: For specific message types and deviations, refer to unmute/openai_realtime_api_events.py.
    • Full Documentation: Detailed information on the WebSocket protocol, message types, and the audio processing pipeline is located in docs/browser_backend_communication.md.

  3. Understand the Unmute connection lifecycle

    main

    A standard integration with the Unmute backend follows this lifecycle:

    1. Health Check: The client verifies connectivity by checking the /v1/health endpoint.
    2. WebSocket Connection: Establish the connection using the realtime subprotocol.
    3. Session Setup: Send a session.update message containing the desired voice and instructions.
    4. Audio Streaming: Engage in bidirectional real-time audio communication.
    5. Graceful Shutdown: Handle disconnection and perform necessary cleanup.

  4. Deploy Unmute using Docker Swarm

    main

    For production-scale deployments, Unmute uses Docker Swarm. All commands should be executed from a client machine within the repository directory, not directly on the swarm nodes.

    Initial Node Setup

    If you are setting up a new machine, first transfer and run the setup script:

    # On your client machine
scp setup_gpu_swarm_node.py llm-wrapper-gpu000:/root/
ssh llm-wrapper-gpu000 python3 /root/setup_gpu_swarm_node.py

    Initializing the Swarm

    To declare a node as the manager (only if a swarm is not already running):

    docker -H ssh://llm-wrapper-gpu000 swarm init

    Connecting Workers

    To add a new worker node to an existing manager:

    1. On the manager, retrieve the join token: docker -H ssh://llm-wrapper-gpu000 swarm join-token worker
    2. Run the resulting command on the new worker node.

    Running the Deployment

    Before running the bake scripts, ensure the following environment variables are set:

    • HUGGING_FACE_HUB_TOKEN: Required to access gated models (e.g., google/gemma-3-12b-it).
    • PROVIDERS_GOOGLE_CLIENT_SECRET: Required for authentication to observability services (Grafana, Traefik).
    • NEWSAPI_API_KEY: (Optional) Used for the "Dev (news)" character.

    Execute the deployment using:

    ./bake_deploy_prod.sh
# OR
./bake_deploy_staging.sh
    # If new machine
scp setup_gpu_swarm_node.py llm-wrapper-gpu000:/root/
ssh llm-wrapper-gpu000 python3 /root/setup_gpu_swarm_node.py

# Manager only
docker -H ssh://llm-wrapper-gpu000 swarm init

# Connect worker
docker -H ssh://llm-wrapper-gpu000 swarm join-token worker

./bake_deploy_prod.sh

  5. Restart or Update Unmute services

    main

    You can manage individual services within the swarm without performing a full docker stack deploy.

    Force Restart a Service

    Use this to trigger a restart (e.g., to pick up new voices in llm-wrapper_tts):

    docker -H ssh://llm-wrapper-gpu000 service update --force llm-wrapper_something

    Update a Single Service

    To update a specific service's image or configuration (like adding a volume) without affecting the rest of the stack:

    Update Image:

    docker service update --image rg.fr-par.scw.cloud/namespace-unruffled-tereshkova/llm-wrapper-frontend:latest --with-registry-auth llm-wrapper_frontend

    Add a Volume:

    docker service update --mount-add type=volume,source=other-volume,target=/somewhere-else llm-wrapper_frontend
    docker -H ssh://llm-wrapper-gpu000 service update --force llm-wrapper_something

docker service update --image rg.fr-par.scw.cloud/namespace-unruffled-tereshkova/llm-wrapper-frontend:latest --with-registry-auth llm-wrapper_frontend

docker service update --mount-add type=volume,source=other-volume,target=/somewhere-else llm-wrapper_frontend

  6. Hardware and OS Requirements for Unmute

    main

    To run Unmute, your environment must meet the following specifications:

    Hardware:

    • GPU: Must have CUDA support.
    • VRAM: At least 16 GB minimum.
    • Architecture: x86_64 (Note: aarch64 is not supported).

    Operating System:

    • Linux: Fully supported.
    • Windows: Supported via WSL only. Native Windows is not supported.
    • Mac: Not supported.

  7. Change Characters and Voices

    main

    You can customize the chatbot's personality and voice by modifying the voices.yaml configuration file.

    • Voices: To use a different voice, update the path_on_server field in voices.yaml with the relative path to the audio file (e.g., voice-donations/Haku.wav). Available voices can be found in the Kyutai voice repository.
    • Prompts: Character prompts are defined in voices.yaml. Note that some system prompts (like 'Quiz show') use dynamic elements defined in unmute/llm/system_prompt.py.

    Important: The voices.yaml file is loaded and cached when the backend starts. You must restart the backend for any changes to take effect.

  8. Configure Hugging Face Hub access for LLMs

    main

    The default local setup uses google/gemma-3-1b-it. Because this model is gated, you must follow these steps to provide access:

    1. Create a Hugging Face account.
    2. Accept the conditions on the Mistral Small 3.2 24B model page (required for access).
    3. Create a Hugging Face access token with "Read access to contents of all public gated repos you can access".
      • Security Note: Do not use tokens with write access for public deployments.
    4. Set the token as an environment variable in your shell (e.g., ~/.bashrc): export HUGGING_FACE_HUB_TOKEN=hf_...your token here...

  9. Enable Subtitles and Dev Mode

    main

    Unmute provides built-in features for debugging and accessibility:

    • Subtitles: Press S in the UI to toggle subtitles for both the user and the chatbot.
    • Dev Mode: This mode is disabled by default. To enable it, modify useKeyboardShortcuts.ts and set ALLOW_DEV_MODE to true. Once enabled, press D to view the debug interface. You can extend the debug view by adding information to self.debug_dict within unmute_handler.py.

  10. Scale the Unmute Swarm

    main

    To handle heavy loads by adding more resources (GPU, Disk, RAM, CPU), follow these two steps:

    1. Add new machines: On the manager node, run docker swarm join-token worker to get the join command, then execute that command on the new node.
    2. Scale services: Once the node is part of the swarm, increase the container count for specific services using docker service scale. For example, to scale the LLM wrapper:
    docker service scale llm-wrapper_llm=10

    Note: Swarm will not automatically rebalance containers to new nodes to avoid downtime. To force a rebalance across all nodes, you must manually force-restart the service.

    docker service scale llm-wrapper_llm=10

  11. Run Unmute without Docker (Dockerless)

    main

    This method is for Linux or WSL users and requires manual dependency management.

    Software Requirements:

    • uv (via curl -LsSf https://astral.sh/uv/install.sh | sh)
    • cargo (via curl https://sh.rustup.rs -sSf | sh)
    • pnpm (via curl -fsSL https://get.pnpm.io/install.sh | sh -)
    • cuda 12.1 (via conda or Nvidia website)

    Execution: Start each service in a separate tmux session or terminal:

    ./dockerless/start_frontend.sh
./dockerless/start_backend.sh
./dockerless/start_llm.sh        # Needs 6.1GB of vram
./dockerless/start_stt.sh        # Needs 2.5GB of vram
./dockerless/start_tts.sh        # Needs 5.3GB of vram

    The website will be available at http://localhost:3000.

  12. Access Unmute Voice Donation recordings

    main

    The voice recordings collected during the Unmute Voice Donation Project are publicly available on the Hugging Face Hub. You can find the raw recordings in the voice-donations/ directory of the kyutai/tts-voices repository. These voices are intended for use with the Kyutai TTS 1.6B model.

    https://huggingface.co/kyutai/tts-voices/tree/main/voice-donations