repository·master·Indexed 27 days ago
https://github.com/jasonppy/voicecraftA token infilling neural codec language model for zero-shot speech editing and text-to-speech (TTS) using in-the-wild data. It supports speech substitution, insertion, and deletion, and provides tools for training, inference via command line or Gradio UI, and integration with the Montreal Forced Aligner (MFA) for audio alignment.
VoiceCraft is subject to specific licensing terms:
Important Disclaimer: Using this technology to generate or edit someone's speech (including government leaders, political figures, and celebrities) without their explicit consent is prohibited and may violate copyright laws.
To use VoiceCraft as a standalone script, use tts_demo.py for text-to-speech or speech_editing_demo.py for speech editing. Ensure your environment is set up first. You can pass command line arguments to specify input audios, target transcripts, and inference hyperparameters. Use the -h flag to view available arguments.
python3 tts_demo.py -hVoiceCraft can be run using Docker on Linux or Windows. This requires Docker and the NVIDIA Container Toolkit installed.
Follow these steps:
inference_tts.ipynb in the browser to run inference.# 1. clone the repo
git clone git@github.com:jasonppy/VoiceCraft.git
cd VoiceCraft
# 2. Build the docker image
docker build --tag "voicecraft" .
# 3. Start the container
./start-jupyter.sh # linux
start-jupyter.bat # windows
# 4. Check logs for the Jupyter URL
docker logs jupyter
# 5. Optionally enter the container
docker exec -it jupyter /bin/bashThe simplest way to try VoiceCraft's speech editing or TTS inference is via Google Colab. Use the dedicated notebooks for specific tasks:
To install VoiceCraft locally without Docker, create a Conda environment and install the required dependencies. Note that specific versions of torch, torchaudio, and xformers are required for compatibility.
conda create -n voicecraft python=3.9.16
conda activate voicecraft
pip install -e git+https://github.com/facebookresearch/audiocraft.git@c5157b5bf14bf83449c17ea1eeb66c19fb4bc7f0#egg=audiocraft
pip install xformers==0.0.22
pip install torchaudio==2.0.2 torch==2.0.1
apt-get install ffmpeg
apt-get install espeak-ng
pip install tensorboard==2.16.2
pip install phonemizer==3.2.1
pip install datasets==2.16.0
pip install torchmetrics==0.11.1
pip install huggingface_hub==0.22.2
# Install MFA (Montreal Forced Aligner)
conda install -c conda-forge montreal-forced-aligner=2.2.17 openfst=1.8.2 kaldi=5.5.1068
mfa model download dictionary english_us_arpa
mfa model download acoustic english_us_arpa
# For Jupyter support
conda install -n voicecraft ipykernel --no-deps --force-reinstallOnce the data is prepared (codes, phonemes, vocab.txt, and manifest), you can start training using the provided shell scripts in the z_scripts directory.
conda activate voicecraft
cd ./z_scripts
bash e830M.shTo train a VoiceCraft model, you must encode utterances into codes and convert transcripts into phoneme sequences. Use the phonemize_encodec_encode_hf.py script to automate this process using the Encodec model.
Note: If you encounter Out-of-Memory (OOM) errors, decrease --batch_size or --max_len.
conda activate voicecraft
export CUDA_VISIBLE_DEVICES=0
cd ./data
python phonemize_encodec_encode_hf.py \
--dataset_size xs \
--download_to path/to/store_huggingface_downloads \
--save_dir path/to/store_extracted_codes_and_phonemes \
--encodec_model_path path/to/encodec_model \
--mega_batch_size 120 \
--batch_size 32 \
--max_len 30000When using the Gradio interface, observe the following constraints to avoid errors:
sample_batch_size, stop_repetition, and seed. Changing other advanced settings like kvache or temperature may trigger JSON warnings.prompt end time between 6-9 seconds. Setting it too long may cause JSON errors or cut off generated audio.prompt end time + generated audio) should not exceed 16 or 17 seconds..snd extension. You must manually convert these to .wav or .mp3, or rename the file extension manually.After installing system dependencies in your Docker container, activate the voicecraft conda environment and install the necessary Python packages using gradio_requirements.txt.
!source ~/.bashrc && \
conda activate voicecraft && \
pip install -r gradio_requirements.txtBefore running inference, you must install the Montreal Forced Aligner (MFA) models and dictionaries. This is required for audio alignment. If you are not using a pre-configured Docker environment, run the following commands in your terminal:
source ~/.bashrc && \
conda activate voicecraft && \
mfa model download dictionary english_us_arpa && \
mfa model download acoustic english_us_arpaLaunch the Gradio web interface using the gradio_app.py script. You must specify the --demo-path, --tmp-path, and --models-path to point to the correct directories within the cloned repository. Use the --share flag to generate a public URL.
!python /content/VoiceCraft/gradio_app.py --demo-path=/content/VoiceCraft/demo --tmp-path=/content/VoiceCraft/demo/temp --models-path=/content/VoiceCraft/pretrained_models --shareTo perform speech cloning, you must first align your reference audio with its transcript using MFA. This generates the necessary alignment files to identify the prompt segment.
Place your .wav file and a corresponding .txt file (with the same name) in a temporary folder, then run:
source ~/.bashrc && \
conda activate voicecraft && \
mfa align -v --clean -j 1 --output_format csv {temp_folder} \
english_us_arpa english_us_arpa {align_temp}Troubleshooting: If alignment fails, try increasing the beam size:
mfa align ... --beam 1000 --retry_beam 2000