DOCSEARCHbyAlgolia
Submit Docs

OpenVoice Instant Voice Cloning Framework

repository·main·Indexed 12 days ago

https://github.com/myshell-ai/openvoice

An instant voice cloning framework for accurate tone color cloning, granular control over voice styles (emotion and rhythm), and zero-shot cross-lingual capabilities. It features OpenVoice V1 and V2, with V2 providing improved audio quality and native support for English, Spanish, French, Chinese, Japanese, and Korean. The system utilizes BaseSpeakerTTS for speech generation and ToneColorConverter for applying target voice characteristics.

Tokens
4.8K
Snippets
14
Records
22
Agent score
95%

What's inside OpenVoice

  1. Overview of OpenVoice V1 and V2

    main

    OpenVoice is a versatile instant voice cloning system. It provides accurate tone color cloning, flexible voice style control (emotion, accent, rhythm, pauses, intonation), and zero-shot cross-lingual voice cloning.

    OpenVoice V1

    • Accurate Tone Color Cloning: Clones reference tone color across multiple languages and accents.
    • Flexible Style Control: Granular control over emotion, accent, rhythm, pauses, and intonation.
    • Zero-shot Cross-lingual Cloning: Supports generating speech in languages not present in the reference speaker's training data.

    OpenVoice V2

    • Improved Audio Quality: Uses a different training strategy for better output.
    • Native Multi-lingual Support: Natively supports English, Spanish, French, Chinese, Japanese, and Korean.
    • Commercial Use: Both V1 and V2 are released under the MIT License, making them free for commercial use.

  2. Quick Use of OpenVoice via Deployed Services

    main

    If you want to test OpenVoice's cross-lingual voice cloning capabilities without local installation, you can use pre-deployed services for various languages. OpenVoice can clone a voice from any input speech audio and use it to speak in multiple languages.

    Available language demos include:

    • British English
    • American English
    • Indian English
    • Australian English
    • Spanish
    • French
    • Chinese
    • Japanese
    • Korean

  5. Use OpenVoice for multi-lingual and cross-lingual tasks

    main

    OpenVoice supports any language as long as a compatible base speaker for that language is available. The tone color converter is designed to work across languages.

    To implement multi-lingual or cross-lingual usage, refer to the demo_part2.ipynb notebook in the repository. If you do not wish to train your own base speaker TTS model, you can use the OpenAI TTS model as the base speaker.

  6. Install OpenVoice on Linux

    main

    For developers and researchers using Linux, Python, and PyTorch, follow these steps to install the repository in editable mode using Conda.

    Note: The installation steps are identical for both OpenVoice V1 and V2.

    conda create -n openvoice python=3.9
conda activate openvoice
git clone git@github.com:myshell-ai/OpenVoice.git
cd OpenVoice
pip install -e .

  7. Set up OpenVoice V2

    main

    OpenVoice V2 requires specific checkpoints and the installation of MeloTTS. V2 natively supports English, Spanish, French, Chinese, Japanese, and Korean.

    1. Download the V2 checkpoints from this link.
    2. Extract the contents into a folder named checkpoints_v2 within the project root.
    3. Install MeloTTS and its required dictionary:

    Demo Usage: Refer to demo_part3.ipynb for example usage of V2 features.

    pip install git+https://github.com/myshell-ai/MeloTTS.git
python -m unidic download

  8. Set up OpenVoice V1

    main

    To use OpenVoice V1, you must download the specific checkpoints and place them in the correct directory.

    1. Download the V1 checkpoints from this link.
    2. Extract the contents into a folder named checkpoints within the project root.

    Key Features & Examples:

    • Flexible Voice Style Control: Refer to demo_part1.ipynb for usage.
    • Cross-Lingual Voice Cloning: Refer to demo_part2.ipynb for cloning languages seen or unseen in the MSML training set.
    • Local Gradio Demo: Launch a minimalist local interface using the command below.
    python -m openvoice_app --share

  9. Control Speech Style and Speed in BaseSpeakerTTS

    main

    The base_speaker_tts.tts method allows you to modify the emotional delivery and tempo of the generated speech.

    • speaker: Controls the style. Available options include: friendly, cheerful, excited, sad, angry, terrified, shouting, and whispering.
    • speed: A float controlling the playback speed (e.g., 0.9 for slower speech).

    Note: If you change the speaker style, you must also update the source_se (tone color embedding) to match that specific style (e.g., using a specific en_style_se.pth file) to ensure consistent results during conversion.

  10. Perform Voice Cloning Inference

    main

    Voice cloning is a two-step process:

    1. Generate base speech: Use base_speaker_tts.tts() to generate an initial audio file from text. You can control the speaker style and speed here.
    2. Convert tone color: Use tone_color_converter.convert() to transform the base audio into the target voice using the source embedding (src_se) and the target embedding (tgt_se).
    save_path = f'{output_dir}/output_en_default.wav'
text = "This audio is generated by OpenVoice."
src_path = f'{output_dir}/tmp.wav'

# Step 1: Run the base speaker tts
base_speaker_tts.tts(text, src_path, speaker='default', language='English', speed=1.0)

# Step 2: Run the tone color converter
encode_message = "@MyShell"
tone_color_converter.convert(
    audio_src_path=src_path, 
    src_se=source_se, 
    tgt_se=target_se, 
    output_path=save_path,
    message=encode_message
)

  11. Initialize BaseSpeakerTTS and ToneColorConverter

    main

    To use OpenVoice, you must initialize two main components: BaseSpeakerTTS for generating the initial speech and ToneColorConverter for applying the target voice characteristics. Both require a configuration JSON and a checkpoint file. You should also specify the compute device (e.g., cuda:0 or cpu).

    import os
import torch
from openvoice import se_extractor
from openvoice.api import BaseSpeakerTTS, ToneColorConverter

ckpt_base = 'checkpoints/base_speakers/EN'
ckpt_converter = 'checkpoints/converter'
device="cuda:0" if torch.cuda.is_available() else "cpu"
output_dir = 'outputs'

# Initialize Base Speaker TTS
base_speaker_tts = BaseSpeakerTTS(f'{ckpt_base}/config.json', device=device)
base_speaker_tts.load_ckpt(f'{ckpt_base}/checkpoint.pth')

# Initialize Tone Color Converter
tone_color_converter = ToneColorConverter(f'{ckpt_converter}/config.json', device=device)
tone_color_converter.load_ckpt(f'{ckpt_converter}/checkpoint.pth')

os.makedirs(output_dir, exist_ok=True)

  12. Use MeloTTS as base speakers for multi-lingual cloning

    main

    You can use MeloTTS to generate high-quality multi-lingual source audio for the cloning process. MeloTTS supports various languages and accents including English (American, British, Indian, Australian), Spanish, French, Chinese, Japanese, and Korean.

    To use the pre-trained base speaker embeddings from OpenVoice, load the .pth files from the checkpoints_v2/base_speakers/ses/ directory corresponding to the MeloTTS speaker key.

    from melo.api import TTS

# Example: Using English (American) as a base
language = 'EN'
device = 'cuda:0' if torch.cuda.is_available() else 'cpu'
model = TTS(language=language, device=device)

# Get speaker ID from MeloTTS model
speaker_ids = model.hps.data.spk2id
speaker_key = 'en_us' # Example key
speaker_id = speaker_ids[speaker_key]

# Load the corresponding source SE from OpenVoice checkpoints
source_se = torch.load(f'checkpoints_v2/base_speakers/ses/{speaker_key.lower().replace("_", "-")}.pth', map_location=device)

# Generate source audio
src_path = 'outputs_v2/tmp.wav'
model.tts_to_file("Your text here", speaker_id, src_path, speed=1.0)