DOCSEARCHbyAlgolia
Submit Docs

FireRedTTS-2 Documentation

repository·main·Indexed 23 days ago

https://github.com/fireredteam/fireredtts2

A long-form streaming text-to-speech (TTS) system optimized for multi-speaker dialogue generation, such as podcasts and chatbots. It features ultra-low latency streaming, multilingual support (English, Chinese, Japanese, Korean, French, German, and Russian), and zero-shot voice cloning. The system utilizes a 12.5Hz streaming speech tokenizer and dual-transformer architecture, supporting both monologue and dialogue generation via a Gradio web UI or Python API.

Tokens
6.5K
Snippets
13
Records
29
Agent score
76%

What's inside FireRedTTS-2

  1. Overview of FireRedTTS-2 capabilities

    main

    FireRedTTS-2 is a long-form streaming TTS system designed for multi-speaker dialogue generation.

    Key features include:

    • Long Conversational Speech: Supports 3-minute dialogues with 4 speakers (scalable).
    • Multilingual & Zero-Shot: Supports English, Chinese, Japanese, Korean, French, German, and Russian. Includes zero-shot voice cloning for cross-lingual and code-switching scenarios.
    • Ultra-Low Latency: Uses a 12.5Hz streaming speech tokenizer and dual-transformer architecture. On an L20 GPU, first-packet latency can be as low as 140ms.
    • Stability: High similarity and low WER/CER in monologue and dialogue.
    • Random Timbre Generation: Useful for generating ASR/speech interaction data.
    • Efficient Inference: Supports bf16 inference, reducing VRAM usage from 14GB to 9GB for consumer-grade GPU deployment.

  2. Usage Disclaimer for Voice Cloning

    main
    The FireRedTTS2 project includes zero-shot voice cloning capabilities. This feature is intended solely for academic research purposes. Users are strictly prohibited from using this model for any illegal activities. The developers assume no liability for misuse.

  3. Convert processed data to .arrow format with step3_write_arrow.py

    main

    To prevent I/O bottlenecks during large model training, convert your tokenized .jsonl data into the .arrow format using the Datasets library. This generates an out_datasets folder containing the .arrow data file along with dataset_info.json and state.json required for seamless loading.

    python step3_write_arrow.py --jsonl ./ljspeech_token.jsonl --pretrained_dir xxx/pretrained_models --dataset_dir ./out_datasets --prefix ljspeech

  4. Install FireRedTTS-2 via Conda

    main

    To install FireRedTTS-2 locally using Conda, follow these steps to set up a Python 3.11 environment, install PyTorch with CUDA 12.6 support, and install the project dependencies.

    1. Clone the repository.
    2. Create and activate a Conda environment named fireredtts2.
    3. Install PyTorch.
    4. Install the package and its requirements in editable mode.
    git clone https://github.com/FireRedTeam/FireRedTTS2.git
cd FireRedTTS2

conda create --name fireredtts2 python==3.11
conda activate fireredtts2

# Step 1. PyTorch Installation (if required)
pip install torch==2.7.1 torchvision==0.22.1 torchaudio==2.7.1 --index-url https://download.pytorch.org/whl/cu126

# Step 2. Install Dependencies
pip install -e .
pip install -r requirements.txt

  5. Download FireRedTTS-2 Pretrained Models

    main

    Pretrained models must be downloaded from Hugging Face using Git LFS and placed in the pretrained_models/FireRedTTS2 directory.

    git lfs install
git clone https://huggingface.co/FireRedTeam/FireRedTTS2 pretrained_models/FireRedTTS2

  6. Extract audio tokens with step2_extract_token.py

    main

    To accelerate training, audio tokens should be pre-extracted rather than generated dynamically. This script performs quantization on the audio information retrieved from your .jsonl metadata file.

    Input: A .jsonl file from Step 1. Output: A new .jsonl file containing an additional audio_token field.

    python step2_extract_token.py --jsonl ./ljspeech.jsonl --pretrained_dir xxx/pretrained_models

  7. Run fine-tuning with accelerate launch

    main

    Once the configuration is set, initiate the training process using accelerate launch. You must provide the path to your configuration file and the path to the pretrained checkpoint.

    accelerate launch --mixed_precision "fp16" posttrain.py --config_path ./config_finetune_1.5b_0.2b.json --checkpoint_path xxx/pretrained_models/llm_posttrain.pt

  8. Install FireRedTTS-2 using Docker

    main

    You can run FireRedTTS-2 in a containerized environment using Docker. This involves building the image from the docker directory and running the container with GPU support and volume mounting for your workspace.

    # Build docker image
docker build -t fireredtts2:v1.0 docker

# Launch docker container
docker run -v=${PWD}:/workspace/FireRedTTS2 --ipc=host --net=host --gpus=all -it fireredtts2:v1.0 bash

  9. Prepare dataset metadata with step1_create_meta.py

    main

    Before training, you must aggregate dataset information into a unified .jsonl format. This step creates a metadata file containing essential information for each audio clip, such as duration, path, speaker, and text. This facilitates subsequent feature extraction.

    For dialogue data, ensure each dialogue turn is added as a separate dictionary entry within the segments list for that audio clip.

    python step1_create_meta.py --data_dir xxx/LJSpeech-1.1 --out_jsonl ./ljspeech.jsonl

  10. Launch the FireRedTTS2 Web UI

    main

    You can generate dialogue using an easy-to-use web interface that supports both voice cloning and randomized voices. Run the following command to start the Gradio demo:

    python gradio_demo.py --pretrained-dir "./pretrained_models/FireRedTTS2"

  11. Interleave audio and text tokens for training

    main

    The interleave function transforms flat audio and text tokens into an interleaved format suitable for training. It processes segments to fit within specific length constraints:

    • AUDIO_MAX_LEN: 2300
    • TOTAL_MAX_LEN: 3100

    Process:

    1. Indexing: Uses get_index to map segment lengths to start/end indices.
    2. Segment Selection: If there are more than 16 segments, it randomly selects a subset.
    3. Length Constraint: If the total audio length exceeds AUDIO_MAX_LEN, it iteratively shrinks the segment range until it fits.
    4. Interleaving: It reshapes audio tokens to [AUDIO_NUM_CODEBOOKS, seq_len] and creates frames where:
      • Audio frames: Shape [seq_len, AUDIO_NUM_CODEBOOKS + 1]. The first AUDIO_NUM_CODEBOOKS columns contain the audio tokens, and the last column is reserved for text (set to 0).
      • Text frames: Shape [text_seq_len, AUDIO_NUM_CODEBOOKS + 1]. The last column contains the text tokens, and the first AUDIO_NUM_CODEBOOKS columns are set to 0.
    5. Output: Returns concatenated interleave_tokens_concat and interleave_masks_concat tensors.

  12. Configure fine-tuning parameters in the JSON config file

    main

    Fine-tuning is controlled via a JSON configuration file (e.g., config_finetune_1.5b_0.2b.json). You must modify the following key sections:

    • train: Set logs_folder (output directory for logs and the final model), batch_size (adjust based on GPU memory), and other hyperparameters like lr, n_epochs, and accumulate_num.
    • models: Define model architecture details like backbone_flavor, decoder_flavor, and vocabulary sizes.
    • dataset: Point train_dataset_dir and valid_dataset_dir to the folder containing your .arrow files generated in the data preparation phase.
    {
    "train": {
        "batch_size": 12,
        "lr": 0.000003,
        "n_epochs": 10,
        "warmup_steps": 12000,
        "weight_decay": 0.002,
        "lr_decay": "linear",
        "accumulate_num": 2,
        "max_grad_norm": 1.3,
        "keep_ckpts": 10,
        "log_every": 10,
        "val_every": 1000,
        "save_every": 1000,
        "gen_every": 1000,
        "num_workers": 4,
        "logs_folder": "xxx/finetune_logs"
    },
    "models": {
        "sample_rate": 16000,
        "backbone_flavor": "qwen-1.5b",
        "decoder_flavor": "qwen-200m",
        "text_vocab_size": 151936,
        "audio_vocab_size": 2051,
        "audio_num_codebooks": 16,
        "decoder_loss_weight": 0.6
    },
    "dataset": {
        "train_dataset_dir": "xxx/out_datasets",
        "valid_dataset_dir": "xxx/out_datasets"
    }
}