DOCSEARCHbyAlgolia
Submit Docs

Opus Documentation

repository·main·Indexed 25 days ago

https://github.com/xiph/opus

Documentation for the Opus high-quality audio coding project. Includes guides on building the project using CMake for Windows, macOS, Linux, Android, and iOS, as well as detailed instructions for training and implementing deep learning models including FARGAN, LPCNet, OSCE (Opus Speech Coding Enhancement), and packet loss generative models.

Tokens
7.9K
Snippets
34
Records
55
Agent score
84%

What's inside Opus

  1. Prepare LPCNet for xcorr extraction

    main

    To enable xcorr extraction in LPCNet, you must modify the source and recompile:

    1. Add lpcnet_extractor.c to the project.
    2. Add relevant functions to lpcnet_enc.c.
    3. Add the source for the new headers and .c files to Makefile.am.
    4. Compile the project to generate the ./lpcnet_xcorr_extractor object.

  2. Run FARGAN inference

    main

    Inference involves two steps: generating features from audio and then synthesizing speech from those features.

    1. Generate Features

    Use the fargan_demo executable to generate .f32 features from a .pcm file.

    2. Synthesize Speech

    You can synthesize speech using either PyTorch or the compiled C code.

    PyTorch Synthesis: Use the test_fargan.py script with your model checkpoint and the generated features.

    C Code Synthesis: Use the fargan_demo executable with the -fargan-synthesis flag.

  3. Run Opus tests with Meson

    main

    Opus includes a test suite that can be executed via Meson's built-in testing command. Run the following command from the project root to execute tests against your build directory:

    meson test -C builddir

  4. Export a trained OSCE model to C

    main

    To integrate a trained model into the Opus codebase, export the weights to the opus/dnn directory and rebuild Opus with OSCE enabled.

    python export_model_weigths.py <path_to_checkpoint> ../../ [--quantize]

    Warning: Running autogen.sh will overwrite any local changes made to the model weights. After exporting, ensure you build Opus with the --enable-osce flag.

    python export_model_weigths.py <path_to_checkpoint> ../../ [--quantize]

  5. Train a FARGAN model

    main

    Training FARGAN is a two-step process: pre-training followed by adversarial training.

    1. Pre-training

    Run the pre-training script with the extracted features and speech files.

    2. Adversarial Training

    Once pre-training is complete, run the adversarial training script. You must provide the initial checkpoint from the pre-training step using the --initial-checkpoint flag. The final model will be saved in the output_dir/checkpoints/ directory.

  6. Train OSCE models using regression loss

    main

    To train a model using regression loss, first create a default setup for either lace or nolace using make_default_setup.py, then execute the training script.

    1. Create setup

    python make_default_setup.py model.yml --model lace/nolace --path2dataset <path2dataset>

    2. Run training

    Run in the foreground:

    python train_model.py model.yml <output folder> --no-redirect

    Or run in the background (output is written to <output folder>/out.txt):

    nohup python train_model.py model.yml <output folder> &
    python make_default_setup.py model.yml --model lace/nolace --path2dataset <path2dataset>
python train_model.py model.yml <output folder> --no-redirect

  8. Generate training data for OSCE

    main

    Generating training data involves two main steps: preparing the audio files and building a specialized version of the opus_demo binary.

    1. Prepare and concatenate audio

    Convert training items to 16 kHz, 16-bit PCM, and concatenate them using scripts/concatenator.py:

    python scripts/concatenator.py filelist 16000 dataset/clean.s16 --db_min -40 --db_max 0

    2. Build opus_demo with training data support

    Configure and build the opus_demo binary with the --enable-osce-training-data flag to enable writing decoder features to disk:

    ./configure --disable-shared --enable-osce-training-data && make clean && make -j

    3. Create the training data

    Run the built opus_demo binary against your concatenated file:

    cd dataset && <path_to_opus_demo_with_training_data>/opus_demo voip 16000 1 9000 -silk_random_switching 249 clean.s16 coded.s16

    Note: The -silk_random_switching argument specifies the number of frames after which parameters are switched randomly.

    python scripts/concatenator.py filelist 16000 dataset/clean.s16 --db_min -40 --db_max 0
./configure --disable-shared --enable-osce-training-data && make clean && make -j
cd dataset && <path_to_opus_demo_with_training_data>/opus_demo voip 16000 1 9000 -silk_random_switching 249 clean.s16 coded.s16

  9. Convert FARGAN model to C code

    main

    You can convert a trained PyTorch FARGAN model into C source files (fargan_data.c and fargan_data.h). After conversion, copy these files to the opus/dnn/ directory (replacing existing ones) and recompile Opus to use the new model.

    python dump_fargan_weights.py output_dir/checkpoints/fargan_adv_50.pth fargan_c_dir

  10. Export RDO-VAE weights to C

    main

    To use a trained PyTorch model within the Opus C codebase, convert the .pth checkpoint to C source and header files using export_rdovae_weights.py. After exporting, copy the generated files to the opus/dnn/ directory (replacing existing files) and recompile Opus.

    python export_rdovae_weights.py output_dir/checkpoints/chechpoint_400.pth dred_c_dir

  11. Install the PTDB Dataset

    main

    To prepare the PTDB dataset for neural pitch estimation, download the zipped files, unzip them, and process the speech data to combine male and female samples.

    1. Download and unzip the PTDB-TUG dataset:
      wget https://www2.spsc.tugraz.at/databases/PTDB-TUG/SPEECH_DATA_ZIPPED.zip
unzip SPEECH_DATA_ZIPPED.zip
    2. Process the data: Navigate into the resulting SPEECH DATA directory and run the processing script:
      ./ptdb_process.sh

  12. Exchange weights between PyTorch and TensorFlow/Keras

    main

    The weight-exchange module facilitates exchanging weights between torch and tensorflow.keras modules using an intermediate NumPy format.

    • To load or dump PyTorch weights, use the exchange.torch submodule.
    • To load or dump TensorFlow weights, use the exchange.tf submodule.

    Note that import exchange does not automatically import the heavy dependencies. You must explicitly import the submodule corresponding to the framework you are using. exchange.torch requires torch to be installed, and exchange.tf requires tensorflow to be installed.