BreezyVoice

Use batch_inference.py to process multiple synthesis tasks from a CSV file.

Arguments:

• --csv_file: Path to the input CSV file.
• --speaker_prompt_audio_folder: Folder containing the speaker prompt audio files.
• --output_audio_folder: Folder where the generated .wav files will be saved.

CSV File Structure: Your CSV must include these columns:

• speaker_prompt_audio_filename: The filename (without extension) of the reference audio in the prompt folder.
• speaker_prompt_text_transcription: (Optional) Transcription of the reference audio.
• content_to_synthesize: The text to be synthesized.
• output_audio_filename: The desired filename (without extension) for the output audio.

python batch_inference.py \
  --csv_file ./data/batch_files.csv \
  --speaker_prompt_audio_folder ./data \
  --output_audio_folder ./results

You can export Matcha checkpoints to ONNX format for optimized inference.

Requirements:

• pip install onnx
• torch>=2.1.0 (required for scaled_product_attention support during export).

Export Command

python3 -m matcha.onnx.export matcha.ckpt model.onnx --n-timesteps 5

Options

• --n-timesteps <int>: Specifies the number of timesteps (default is 5). This is a hyper-parameter fixed at export time.
• --vocoder-name <name>: Optionally embed a vocoder in the graph.
• --vocoder-checkpoint <path>: Optionally embed a specific vocoder checkpoint to enable end-to-end waveform generation.

Fine-tuning allows you to adapt the model to new data using mel-spectrograms generated by models like Tacotron2.

Steps:

1. Generate mel-spectrograms in .npy format using teacher-forcing. The filename must match the original audio filename (e.g., LJ001-0001.wav $\rightarrow$ LJ001-0001.npy).
2. Create a directory named ft_dataset and copy all .npy files into it.
3. Run the training script with the --fine_tuning flag set to True.

python train.py --fine_tuning True --config config_v1.json

To use HiFi-GAN, ensure you have Python >= 3.6 installed. Follow these steps to set up the environment and dataset:

1. Clone the repository.
2. Install the required Python dependencies listed in requirements.txt.
3. Download and extract the LJ Speech dataset.
4. Move all .wav files from the dataset into the LJSpeech-1.1/wavs directory.

Once a model is exported to ONNX, you can run inference using onnxruntime.

Setup

• For CPU: pip install onnxruntime
• For GPU: pip install onnxruntime-gpu

Inference Command

python3 -m matcha.onnx.infer model.onnx --text "hey" --output-dir ./outputs

Parameters and Modes

• GPU Acceleration: Add the --gpu flag to use onnxruntime-gpu.
• Synthesis Control: Use --temperature <float>, --speaking_rate <float>, and --spk <int>.
• Vocoder Integration:
  • If you exported only Matcha: The output will be mel-spectrograms (numpy arrays). To get audio, pass a vocoder path: --vocoder hifigan.small.onnx.
  • If you exported Matcha + Vocoder: The output will be .wav files directly.

python3 -m matcha.onnx.infer model.onnx --text "hey" --output-dir ./outputs --gpu

To perform mel-spectrogram inversion (converting existing audio/spectrograms back to waveforms):

1. Create a test_files directory and copy your .wav files into it.
2. Run inference.py pointing to your generator checkpoint.

Options:

• --checkpoint_file: Path to the generator checkpoint file.
• --output_dir: Custom directory for generated files (defaults to generated_files).

python inference.py --checkpoint_file [path_to_checkpoint]

To generate audio directly from mel-spectrograms (e.g., from Tacotron2 or Glow-TTS):

1. Create a test_mel_files directory and copy your generated mel-spectrogram files into it.
2. Run inference_e2e.py pointing to your generator checkpoint.

Options:

• --checkpoint_file: Path to the generator checkpoint file.
• --output_dir: Custom directory for generated files (defaults to generated_files_from_mel).

python inference_e2e.py --checkpoint_file [path_to_checkpoint]

You can train the HiFi-GAN model using the train.py script. By default, it uses the V1 generator configuration. To train different versions (V2 or V3), swap the configuration file.

Options:

• --config: Path to the JSON configuration file (e.g., config_v1.json, config_v2.json, or config_v3.json).
• --checkpoint_path: Custom path where checkpoints and configuration copies are saved (defaults to cp_hifigan).

# Train V1 Generator
python train.py --config config_v1.json

# Train V2 Generator
python train.py --config config_v2.json

To install BreezyVoice, clone the repository and install the required Python dependencies. This project requires Python 3.10.

Note on GPU/CPU: The model can run on CPU. If you do not have a GPU, you must change onnxruntime-gpu to onnxruntime in the requirements.txt file before installing. If you have a GPU, ensure you have cudnn installed matching your CUDA version (e.g., sudo apt-get -y install cudnn9-cuda-11).

# Clone the repo
git clone https://github.com/mtkresearch/BreezyVoice.git
cd BreezyVoice

# Install Requirements (requires Python3.10)
# If using GPU, ensure onnxruntime-gpu is in requirements.txt
pip uninstall onnxruntime
pip install -r requirements.txt

You can install Matcha-TTS via pip or directly from the GitHub source. It is recommended to use a Conda environment with Python 3.10.

Using pip

pip install matcha-tts

From source

pip install git+https://github.com/shivammehta25/Matcha-TTS.git
cd Matcha-TTS
pip install -e .