UTMOSv2: UTokyo-SaruLab MOS Prediction System

By default, the model predicts all .wav files in the --input_dir. To limit prediction to a specific subset, provide a list of utterance IDs (utt-id).

Format for list files: The file should contain one utt-id per line. The .wav extension is optional.

sys00691-utt0682e32
sys00691-utt31fd854
sys00691-utt33a4826

Usage:

• Python API: Use val_list_path (path to a text file) or val_list (a direct list of strings).
• Inference Script: Use the --val_list_path flag.

# Python API via file path
mos = model.predict(input_dir="/path/to/wav/dir/", val_list_path="/path/to/your/val/list.txt")

# Python API via direct list
mos = model.predict(
    input_dir="/path/to/wav/dir/",
    val_list=["sys00691-utt0682e32", "sys00691-utt31fd854"]
)

# Inference Script
python inference.py --input_dir /path/to/wav/dir/ --out_path /path/to/output/file.csv --val_list_path /path/to/your/val/list.txt

To use your own trained weights, provide the path to the checkpoint file. The path can be a configuration file name or a direct path to a .pth file.

Note: The weights must be compatible with the model configuration specified. If you have different weights for different folds, you must run inference separately for each fold.

Usage:

• Python API: Use the checkpoint_path argument in utmosv2.create_model().
• Inference Script: Use the --weight flag.

Looping through multiple fold weights (CLI):

for i in {0..5}; do
    python inference.py --input_path /path/to/wav/file.wav --out_path /path/to/output/file.csv --weight /path/to/your/weight_fold${i}.pth --fold $i
done

# Python API
model = utmosv2.create_model(checkpoint_path="/path/to/your/weight.pth")
mos = model.predict(input_dir="/path/to/wav/dir/")

# Inference Script
python inference.py --input_dir /path/to/wav/dir/ --out_path /path/to/output/file.csv --weight /path/to/your/weight.pth

Training is performed using train.py. The training process often involves multiple stages, especially for the fusion models.

Important: Stages must be trained in ascending numerical order. Fusion models require both the spec_only and ssl_only models to be trained beforehand.

Paper Reproduction

Use the following commands to reproduce the paper's results:

• Without SSL: python train.py --reproduce --config spec_only
• SSL Stage 1 (w/o spec): python train.py --reproduce --config ssl_only_stage1
• SSL Stage 2 (w/o spec): python train.py --reproduce --config ssl_only_stage2 --weight ssl_only_stage1
• UTMOSv2 Stage 2 (Fusion): python train.py --reproduce --config fusion_stage2
• UTMOSv2 Stage 3 (Fusion): python train.py --reproduce --config fusion_stage3 --weight fusion_stage2

Competition Reproduction

To reproduce the VoiceMOS Challenge 2024 Track 1 system:

• python train.py --reproduce --config c_spec_only_stage1
• python train.py --reproduce --config c_spec_only_stage2 --weight c_spec_only_stage1
• python train.py --reproduce --config c_ssl_only_stage1
• python train.py --reproduce --config c_ssl_only_stage2 --weight ssl_only_stage1
• python train.py --reproduce --config c_fusion_stage2
• python train.py --reproduce --config c_fusion_stage3 --weight c_fusion_stage2

# Example: Training Stage 3 of UTMOSv2
python train.py --reproduce --config fusion_stage3 --weight fusion_stage2

Training requires two main data components: the BVCC datasets and additional datasets (sarulab-data, Blizzard Challenges, and SOMOS). They must be organized into specific directory structures in the project root.

1. BVCC Datasets

Organize the BVCC datasets into a data directory as follows:

data
|-- DATA
|   `-- sets
|       `-- val_list.txt
|-- data-main.zip
|-- data-sets.zip
`-- main
    `-- DATA
        |-- mydata_system.csv
        |-- sets
        |   |-- DEVSET
        |   |-- test_mos_list.txt
        |   |-- TESTSET
        |   |-- train_mos_list.txt
        |   |-- TRAINSET
        |   `-- val_mos_list.txt
        `-- wav
            |-- [wav_files...]

2. Additional Datasets

Organize other datasets (Blizzard, SOMOS, sarulab) into a data2 directory. Each dataset should contain its corresponding MOS CSV file and its audio files in a specific sub-folder structure. For example:

• Blizzard: data2/blizzardYYYY/blizzardYYYY_mos.csv and data2/blizzardYYYY/blizzardYYYY_wavs.
• SOMOS: data2/somos/audios and data2/somos/training_files.
• Sarulab: data2/sarulab/VMC2024_MOS.csv.

To train the model on custom datasets, you must provide a JSON configuration file that maps dataset domains to their respective directories and Mean Opinion Score (MOS) lists.

Data Configuration Format

The JSON file should follow this structure:

{
  "data": [
    {
      "name": "dataset1",
      "dir": "/path/to/your/dataset1",
      "mos_list": "/path/to/your/moslist1.txt"
    }
  ]
}

• name: A unique identifier for the data-domain ID.
• dir: The directory containing the .wav audio files.
• mos_list: A text file containing filename/ID and MOS values in a comma-separated format:

  filename-id,mos_value
  sys64e2f-utt491a78a,2.375

The training process will use the intersection of files found in the dir and those listed in the mos_list.

Execution

Run the training script using train.py with the --config and --data_config flags.

python train.py --config spec_only --data_config data_config.json