DOCSEARCHbyAlgolia
Submit Docs

LLaMA-Omni 2

repository·main·Indexed 42 days ago

https://github.com/ictnlp/llama-omni2

A series of speech-language models designed for real-time, low-latency spoken chat, enabling simultaneous text and speech response generation via a streaming autoregressive speech decoder. Version 1.0.0 supports English-only models (0.5B to 14B) and bilingual English/Chinese models (0.5B to 32B-Bilingual). The system integrates Whisper-large-v3 as a speech encoder and CosyVoice 2 as a flow-matching model and vocoder.

Tokens
13.7K
Snippets
51
Records
58
Agent score
55%

What's inside LLaMA-Omni 2

  1. Download required models for LLaMA-Omni 2

    main

    LLaMA-Omni 2 requires three sets of models to function:

    1. Whisper-large-v3: The speech encoder.
    2. CosyVoice 2: The flow-matching model and vocoder.
    3. LLaMA-Omni2 series: The core speech-language models.

    Note: LLaMA-Omni2-0.5B through 14B support English only. LLaMA-Omni2-0.5B through 32B-Bilingual support both English and Chinese.

    If you are in China and have connection issues with Hugging Face, set the HF_ENDPOINT environment variable to https://hf-mirror.com before downloading.

    # 1. Download Whisper
import whisper
model = whisper.load_model("large-v3", download_root="models/speech_encoder/")

# 2. Download CosyVoice 2 components
huggingface-cli download --resume-download ICTNLP/cosy2_decoder --local-dir models/cosy2_decoder

# 3. Download LLaMA-Omni2 models (Example: 7B Bilingual)
export HF_ENDPOINT=https://hf-mirror.com
model_name=LLaMA-Omni2-7B-Bilingual
huggingface-cli download --resume-download ICTNLP/$model_name --local-dir models/$model_name

  2. Fine-tune HiFi-GAN

    main

    Fine-tuning allows you to adapt the model to new data. Follow these steps:

    1. Prepare Mel-spectrograms: Generate mel-spectrograms in .npy (numpy) format using a tool like Tacotron2 with teacher-forcing. The filename must match the original audio file (e.g., LJ001-0001.wav $\rightarrow$ LJ001-0001.npy).
    2. Organize Data: Create a directory named ft_dataset and copy all generated .npy files into it.
    3. Execute Fine-tuning: Run the training script with the --fine_tuning True flag.
    python train.py --fine_tuning True --config config_v1.json

  3. Run ONNX inference for Matcha-TTS

    main

    After exporting to ONNX, you can run inference using onnxruntime or onnxruntime-gpu.

    Installation

    pip install onnxruntime
pip install onnxruntime-gpu  # For GPU acceleration

    Inference Command

    python3 -m matcha.onnx.infer model.onnx --text "hey" --output-dir ./outputs

    Parameters and Features

    • GPU Acceleration: Add the --gpu flag to use the GPU.
    • Synthesis Control: Use --temperature <float>, --speaking_rate <float>, and --spk <int> (speaker ID).
    • Vocoder Integration:
      • If you exported only Matcha, the output will be mel-spectrograms (numpy arrays).
      • To generate .wav files, either export with an embedded vocoder OR pass a path to an ONNX-formatted vocoder model:
        python3 -m matcha.onnx.infer model.onnx --text "hey" --output-dir ./outputs --vocoder hifigan.small.onnx
    python3 -m matcha.onnx.infer model.onnx --text "hey" --output-dir ./outputs --gpu

  4. Perform end-to-end speech synthesis from mel-spectrograms

    main

    To generate audio directly from mel-spectrogram files (e.g., from Tacotron2 or Glow-TTS):

    1. Create a directory named test_mel_files and copy your .npy mel-spectrogram files into it.
    2. Run inference_e2e.py providing the path to your generator checkpoint.
    3. Output: Generated files are saved in generated_files_from_mel by default. Use the --output_dir option to specify a different directory.
    python inference_e2e.py --checkpoint_file [generator_checkpoint_path]

# Example with custom output directory
python inference_e2e.py --checkpoint_file checkpoints/v1.pth --output_dir ./my_mel_results

  5. Install LLaMA-Omni 2

    main

    To install LLaMA-Omni 2, clone the repository and set up a Conda environment with Python 3.10. Install the package in editable mode using pip install -e ..

    git clone https://github.com/ictnlp/LLaMA-Omni2
cd LLaMA-Omni2

conda create -n llama-omni2 python=3.10
conda activate llama-omni2
pip install -e .

  6. Perform inference from wav files

    main

    To generate audio from existing .wav files (mel-spectrogram inversion):

    1. Create a directory named test_files and copy your source .wav files into it.
    2. Run inference.py providing the path to your generator checkpoint.
    3. Output: Generated files are saved in generated_files by default. Use the --output_dir option to specify a different directory.
    python inference.py --checkpoint_file [generator_checkpoint_path]

# Example with custom output directory
python inference.py --checkpoint_file checkpoints/v1.pth --output_dir ./my_results

  7. Train Matcha-TTS with a custom dataset

    main

    To train Matcha-TTS on your own data (e.g., LJ Speech), follow these steps:

    1. Prepare Data: Extract your dataset and create filelists (train/val) pointing to the audio and text.
    2. Configure: Update configs/data/ljspeech.yaml with your train_filelist_path and valid_filelist_path.
    3. Calculate Statistics: Generate normalization statistics using the dataset configuration:
      matcha-data-stats -i ljspeech.yaml
      Update the data_statistics section in configs/data/ljspeech.yaml with the resulting mel_mean and mel_std values.
    4. Run Training: Use make train-ljspeech or the python entrypoint:
      python matcha/train.py experiment=ljspeech

    Training Variants

    • Minimum memory: python matcha/train.py experiment=ljspeech_min_memory
    • Multi-GPU: python matcha/train.py experiment=ljspeech trainer.devices=[0,1]
    python matcha/train.py experiment=ljspeech

  8. Run the Gradio Demo

    main

    To launch the interactive Gradio web interface, you must run three separate components in order: the controller, the web server, and the model worker.

    1. Controller: Manages the service orchestration.
    2. Gradio Web Server: Provides the UI.
    3. Model Worker: Handles the actual model inference.

    Ensure you have downloaded the models as described in the Quick Start guide before running these commands.

    # 1. Launch controller
python -m llama_omni2.serve.controller --host 0.0.0.0 --port 10000

# 2. Launch gradio web server (in a new terminal)
python -m llama_omni2.serve.gradio_web_server --controller http://localhost:10000 --port 8000 --vocoder-dir models/cosy2_decoder

# 3. Launch model worker (in a new terminal)
model_name=LLaMA-Omni2-7B-Bilingual
python -m llama_omni2.serve.model_worker --host 0.0.0.0 --controller http://localhost:10000 --port 40000 --worker http://localhost:40000 --model-path models/$model_name --model-name $model_name

  11. Export Matcha-TTS models to ONNX

    main

    You can export Matcha checkpoints to ONNX format for optimized inference.

    Requirements:

    • pip install onnx
    • torch >= 2.1.0 (required for scaled_product_attention support during export).

    Export Command

    python3 -m matcha.onnx.export matcha.ckpt model.onnx --n-timesteps 5

    Options

    • --n-timesteps <int>: Specifies the number of timesteps. This is treated as a hyper-parameter and must be fixed during export. Defaults to 5.
    • Embedding a Vocoder: You can pass --vocoder-name and --vocoder-checkpoint to embed the vocoder directly into the ONNX graph, allowing for single-run waveform generation.
    python3 -m matcha.onnx.export matcha.ckpt model.onnx --n-timesteps 5

  12. Train HiFi-GAN

    main

    You can train the HiFi-GAN model using the train.py script. By default, checkpoints and configuration copies are saved in the cp_hifigan directory.

    • Standard Training (V1): Use config_v1.json.
    • V2 or V3 Training: Replace the config file with config_v2.json or config_v3.json.
    • Custom Output Path: Use the --checkpoint_path option to change where checkpoints are saved.
    # Train V1 Generator
python train.py --config config_v1.json

# Example with custom checkpoint path
python train.py --config config_v1.json --checkpoint_path /path/to/save