DOCSEARCHbyAlgolia
Submit Docs

Voicebox AI Voice Studio

repository·main·Indexed 13 days ago

https://github.com/jamiepine/voicebox

A local-first, open-source AI voice studio providing a complete voice I/O stack, including zero-shot voice cloning, text-to-speech (TTS) in 23 languages, and Whisper-based speech-to-text (STT) dictation. It features a FastAPI backend with hardware acceleration for Apple Silicon (MLX) and NVIDIA/AMD/Intel GPUs (PyTorch), and includes a Model Context Protocol (MCP) server for integration with AI agents like Claude Code, Cursor, and Windsurf.

Tokens
84.1K
Snippets
195
Records
415
Agent score
99%

What's inside Voicebox

  1. Overview of Voicebox features

    main

    Voicebox is an open-source, local-first AI voice studio designed to run entirely on your hardware. Key capabilities include:

    • Speech Synthesis: Access to 7 TTS engines (including Qwen3-TTS, LuxTTS, and Kokoro) with zero-shot voice cloning and 50+ preset voices across 23 languages.
    • Dictation: Global dictation via a hotkey chord that transcribes and pastes text into any focused application.
    • Voice Personalities: Uses a local LLM to compose or rewrite text based on specific persona profiles.
    • Expressive Control: Supports paralinguistic tags like [laugh] and [sigh] for natural delivery.
    • Post-Processing: Built-in effects including pitch shift, reverb, delay, chorus, compression, and filters.
    • Stories Editor: A multi-track timeline for creating complex audio projects like podcasts or narratives.
    • Agent Integration: Provides an MCP (Model Context Protocol) server, allowing MCP-aware agents to call Voicebox to speak using your cloned voices.
    • API Access: Features both REST and WebSocket APIs for application integration.

  2. Overview of Voicebox Voice I/O

    main

    Voicebox is designed as a local voice I/O layer for both humans and AI agents. It bridges the gap between speech input (STT/Dictation) and speech output (TTS/Cloning).

    Core Capabilities:

    • Input: Speech-to-text (STT) using Whisper, local LLM refinement, and global dictation.
    • Intelligence: Local LLM processing for text refinement and persona management.
    • Output: High-fidelity text-to-speech (TTS), voice cloning, and post-processing effects.
    • Routing: A pipeline that can route captured audio through STT, then an LLM, and finally to a cloned voice (the 'persona loop').

  3. What is Voicebox?

    main

    Voicebox is a local-first, open-source AI voice studio designed to handle the full voice I/O stack (input and output) on your own machine. It serves as a private alternative to services like ElevenLabs (output) and WisprFlow (input).

    Core Capabilities:

    • Voice Cloning: Zero-shot cloning from a few seconds of audio.
    • Speech Generation: Supports 23 languages across 7 TTS engines (Qwen3-TTS, Qwen CustomVoice, LuxTTS, Chatterbox Multilingual, Chatterbox Turbo, HumeAI TADA, and Kokoro).
    • Voice Input (STT): Global dictation via hotkey using Whisper-based speech-to-text.
    • Agent Integration: Allows MCP-aware AI agents (like Claude Code, Cursor, or Cline) to speak using your cloned voices via the voicebox.speak tool call.
    • Privacy: All models, voice data, and captures remain local.
    • Extensibility: API-first design with a REST API and a built-in MCP server.

  4. Review Voicebox Project Status and Roadmap

    main
    The Voicebox project status is tracked via PROJECT_STATUS.md. This document provides a high-level overview of the project's current state, including recent releases (like the v0.5.0 Capture release), upcoming roadmap items, and a detailed triage of open Pull Requests (PRs) categorized by theme (e.g., i18n, new engines, GPU/hardware support, and hardening).

  5. Map of recording and transcription workflows in Voicebox

    main

    Voicebox provides three distinct workflows for recording and transcribing audio, all sharing the same Speech-to-Text (STT) backend but serving different purposes:

    1. Dictation: Use a global hotkey to speak and have text automatically pasted into your currently focused application. Transcripts are cleaned up by a local LLM if auto-refine is enabled. Captures are also saved to the Captures tab.
    2. Captures: Use the Captures tab to record audio in-app, upload files (.wav, .mp3, .m4a, .webm, .opus, .flac), or manage existing dictations. This is for creating a clean archive of audio and transcripts.
    3. Voice Profile Samples: Use the Voices tab to record clips specifically for voice cloning. Unlike standard captures, these require a reference_text that matches the audio verbatim.

    You can promote an existing capture to a voice sample via the Captures tab's Send-to menu by selecting "Use as voice sample…".

  6. Overview of Voicebox Engine Architecture

    main

    Voicebox utilizes a multi-engine architecture designed to support a wide variety of TTS (Text-to-Speech) backends. This allows for easy integration of new models and engines.

    Key Architectural Concepts:

    • Engine Sprawl Management: The project distinguishes between first-class (stable) and experimental TTS backends to manage complexity.
    • Platform Gating: While the architecture is flexible, some engines may be gated to specific platforms (e.g., certain engines may be CUDA-only).
    • Plugin Interface: There is an ongoing effort to formalize a community engine/provider plugin API to allow third-party developers to add new engines easily.

  7. Architecture Overview for Adding TTS Engines

    main

    Voicebox uses a layered backend architecture. When adding a new Text-to-Speech (TTS) engine, you primarily interact with the backends/ directory and the models.py configuration file. The system uses a model configuration registry to handle automatic dispatch, meaning you typically do not need to modify the routes/ (HTTP handlers) or services/ (business logic) layers.

    LayerPurposeFiles Touched
    routes/Thin HTTP handlersNone (auto-dispatch)
    services/Business logicNone (auto-dispatch)
    backends/Engine implementationsyour_engine_backend.py
    utils/Shared utilitiesAs needed

  8. Integrate a Local LLM backend via LLMBackend protocol

    main

    Voicebox supports local LLM backends using an LLMBackend protocol, alongside existing TTSBackend and STTBackend protocols. You can register models like Qwen3 (0.6B, 1.7B, or 4B) via ModelConfig.

    Implementation details:

    • Apple Silicon: Uses MLX with 4-bit community quants.
    • Other Platforms: Uses PyTorch (transformers AutoModelForCausalLM).
    • Model Management: Reuses the existing Hugging Face (HF) download path, cache directory, and model management UI, ensuring no fragmented model caches or new runtimes (like llama.cpp or ollama) are required.

  9. Understand the difference between Cloned and Preset voice profiles

    main

    Voicebox uses two types of profiles to enable speech generation. Choosing the right one depends on whether you need a specific identity or a quick, high-quality voice.

    Profile typeWhat it storesUse when...
    ClonedOne or more reference audio samples + a voice embeddingYou want to replicate a specific person's voice
    PresetA reference to a pre-built voice in a specific engineYou want a curated, production-ready voice with no audio prep

    Key distinction: Cloned profiles require clean audio samples for a specific person, while Preset profiles use existing engine catalogs (like Kokoro or Qwen) and require no audio preparation. Note that Preset profiles are locked to their source engine; if you select a profile in the generation UI, the engine must match the profile's origin.

  10. Identify upcoming engine and model support

    main

    The project is actively expanding its supported TTS (Text-to-Speech) and STT (Speech-to-Text) engines. Developers looking for specific model capabilities can monitor the following categories of work in the PR queue:

    • New Engines/Models: Includes candidates like MOSS-TTS-Nano (CPU realtime), MiniMax Cloud TTS (External API), Parakeet (Whisper alternative), and 4-bit quantized Qwen.
    • Custom Models: Support for Custom HuggingFace voice models and Per-profile LoRA fine-tuning is under development.
    • Streaming: Work is ongoing for stream=true immediate-audio mode for GET /tts and MLX TTS audio chunk streaming.

  11. Understand the Voice I/O Pipeline Abstraction

    main

    Voice I/O operations follow a dataflow pattern consisting of three stages: Source → Transforms → Sink(s). While this is an internal architectural concept, it defines how all voice features (like Dictation or Agent voice reply) are constructed.

    • Source: The trigger or input method (e.g., Hold to speak, Tap to toggle, File drop, API call).
    • Transform: The processing chain applied to the audio or text (e.g., STT model, Refinement LLM, Persona LLM, Translation).
    • Sink: The destination for the processed data (e.g., Clipboard + paste, File on disk, HTTP webhook, MCP server sink, TTS loopback).

    Users do not configure these individual steps manually; instead, they use Presets that bundle a specific source, a transform chain, and one or more sinks to achieve a goal (e.g., a 'Dictation' preset uses a hold-to-speak source, an STT transform, and a clipboard sink).

  12. How Voicebox Cloud Backup & Sync works

    main

    Phase 1 of the roadmap introduces cloud accounts for backup and synchronization. To maintain a local-first privacy model, the server acts as a storage provider for encrypted blobs without ever having access to plaintext data.

    Key Features

    • End-to-End Encryption (E2E): All audio and transcript blobs are encrypted client-side using a key derived from your device-pairing flow. The server only sees encrypted ciphertext.
    • Sync Scope: The service syncs captures (audio + transcripts), generations, voice profiles (as ciphertext), and settings.
    • Sync Exclusions: To protect privacy, the server does not sync voice profile audio in plaintext or refinement LLM context.
    • Multi-device Recovery: You can decrypt backups on a second device using the same paired-device key. Recovery is handled via a printable key generated during the initial pairing.
    • Quota Model: Usage is metered by the number of generations rather than storage GB, making tiering more predictable.