Brouhaha: Multi-task Audio Analysis Model

Brouhaha is a multi-task model designed to extract three specific acoustic features from audio segments:

• Speech/non-speech segments: Voice Activity Detection (VAD).
• Speech-to-Noise Ratio (SNR): Measures the speech level relative to the noise level.
• C50: Measures the extent of environmental reverberation.

Train the model using the train command. You must specify an experimental directory, a task profile (-p), a model type, the number of epochs, and the data directory.

To use custom hyper-parameters and task parameters, place a config.yaml in your experimental directory and include the --config flag in your command.

Example (Basic):

python main.py train /path/to/experimental/directory \
    -p Brouhaha.SpeakerDiarization.NoisySpeakerDiarization \
    --model_type pyannet \
    --epoch 35 \
    --data_dir path/to/your/database

Example (With config.yaml):

python main.py train /path/to/experimental/directory \
    -p Brouhaha.SpeakerDiarization.NoisySpeakerDiarization \
    --model_type pyannet \
    --epoch 35 \
    --data_dir path/to/your/database \
    --config

To install Brouhaha, clone the repository, create a Conda environment with Python 3.8, and install the package using pip. Depending on your environment, you may also need to install libsndfile via conda-forge.

# clone brouhaha
git clone https://github.com/marianne-m/brouhaha-vad.git
cd brouhaha-vad

# creating a conda environment
conda create -n brouhaha python=3.8
conda activate brouhaha

# install brouhaha
pip install .

# optional: install libsndfile if necessary
conda install -c conda-forge libsndfile

Your database must include a train/dev/test split. Each split must follow this specific directory structure:

train/
├── audio_16k
│   └── file_1.flac
│   └── ...
├── detailed_snr_labels
│   └── file_1_snr.npy
│   └── ...
├── reverb_labels.txt
└── rttm_files
    └── file_1.rttm
    └── ...

To register the database, add a Brouhaha entry to your ~/.pyannote/database.yml file using the following pattern:

Databases:
  Brouhaha: Path/to/your/database/*/audio_16k/{uri}.flac

Use the score command to evaluate the model. This computes:

• F-Score for Voice Activity Detection (VAD)
• Mean Squared Error (MSE) for SNR
• Mean Squared Error (MSE) for C50 room acoustics estimation.

python main.py score \
    -p Brouhaha.SpeakerDiarization.NoisySpeakerDiarization \
    --model_path path/to/the/model/checkpoint \
    --data_dir path/to/your/database \
    --out_dir path/to/the/inference/output/folder \
    --report_path path/to/score/files

To loop through Brouhaha data, you must install the pyannote-brouhaha-db package. Ensure your conda environment is activated before installation.

conda activate brouhaha
pip install https://github.com/marianne-m/pyannote-brouhaha-db.git

Use the apply command to run inference on the test portion of your database.

If --params is not specified, the system defaults to the parameters tuned on the Brouhaha database. Use --ext to specify the output file extension (e.g., wav).

python main.py apply \
    -p Brouhaha.SpeakerDiarization.NoisySpeakerDiarization \
    --model_path path/to/the/model/checkpoint \
    --data_dir path/to/your/database \
    --out_dir path/to/the/inference/output/folder \
    --ext wav \
    --params path/to/best/params/yaml/file

Use the brouhaha/main.py script with the apply command to run the pretrained model on a directory of audio files. The model extracts Voice Activity Detection (speech/non-speech), Speech-to-Noise Ratio (SNR), and C50 reverberation metrics.

python brouhaha/main.py apply \
      --data_dir path/to/data \
      --out_dir path/to/predictions \
      --model_path models/best/checkpoints/best.ckpt \
      --ext wav

After training, use the tune command to optimize the threshold for voice activity detection (VAD).

If the --params flag is omitted, the best parameters will be automatically saved to best_params.yml within your experimental directory.

python main.py tune /path/to/experimental/directory \
    -p Brouhaha.SpeakerDiarization.NoisySpeakerDiarization \
    --model_path path/to/the/model/checkpoint \
    --data_dir path/to/your/database \
    --params path/to/best/params/yaml/file

The Brouhaha project provides a command-line interface for four primary tasks: training models, tuning hyperparameters, applying models to data (inference), and scoring inference results against ground truth.

Available commands:

• train: Train a new model using a specified architecture and dataset.
• tune: Optimize pipeline hyperparameters (like thresholds) using Optuna.
• apply: Run inference on audio files to generate RTTM, SNR, and C50 predictions.
• score: Evaluate the accuracy of previously generated inference results using metrics like F-score, MSE for SNR, and MSE for C50.