inaSpeechSegmenter

To install inaSpeechSegmenter using pip, first ensure you have ffmpeg installed on your system (e.g., sudo apt-get install ffmpeg on Ubuntu). It is recommended to use a Python 3 virtual environment. inaSpeechSegmenter supports Python versions 3.8 through 3.13.

# create a python 3 virtual environment and activate it
$ python -m venv env
$ source env/bin/activate

# install framework and dependencies
$ pip install inaSpeechSegmenter

If you need to install from the git repository, clone the repo and use pip to install the local directory. Note that you should use pip install . instead of setup.py for the installation process.

# clone git repository
$ git clone https://github.com/ina-foss/inaSpeechSegmenter.git

# create a python 3 virtual environment and activate it
$ python -m venv env
$ source env/bin/activate

# install framework and dependencies
$ cd inaSpeechSegmenter
$ pip install .

# check program behavior
$ python setup.py test

The output of the Segmenter call is a list of tuples. Each tuple represents a detected segment and contains three elements in the following order:

1. label: A string indicating the detected content. Supported labels are 'male', 'female', 'music', and 'noEnergy'.
2. start time: The start time of the segment.
3. end time: The end time of the segment.

# Example output structure
# [('male', 0.0, 5.2), ('music', 5.2, 10.5), ...]
print(segmentation)

Voice Femininity Scoring (VFS) uses a VBx extraction pipeline to extract features from an audio file, followed by a pre-trained gender detection model applied to those descriptors.

To ensure accuracy, the process includes:

1. Voice Activity Detection (VAD): Uses inaSpeechSegmenter.Segmenter to identify speech segments. The femininity score is computed using gender predictions only on these speech segments.
2. Single Speaker Requirement: For the most accurate score, the recording should contain only one speaker.

Score Interpretation:

• VFS = 1: Voice gender prediction is "female".
• VFS = 0: Voice gender prediction is "male".

Technical Requirement: If using tensorflow-2.12, you must use cudNN version 8.6 or higher.

from inaSpeechSegmenter.vbx_segmenter import VoiceFemininityScoring

To perform segmentation, import the Segmenter class, initialize it (which loads the neural networks), and call the instance with the path to your media file. The segmenter supports any media format compatible with ffmpeg, including video, audio, and URLs.

from inaSpeechSegmenter import Segmenter

# Initialize the segmenter (loads neural networks)
seg = Segmenter()

# media can be a local file path or a URL
media = './media/musanmix.mp3'

# Perform segmentation
segmentation = seg(media)

Install the library using pip. Note that in some environments (like Google Colab), you may also need to install the ffmpeg command line utility to handle media files.

! apt install ffmpeg
! pip install inaSpeechSegmenter

Once exported to CSV, you can use pandas to perform data analytics on the segments. Common tasks include:

1. Calculating segment length: Subtract the start time from the stop time.
2. Aggregating by label: Group the data by the labels column and sum the lengths to find the total duration of each category (e.g., 'music', 'noEnergy', 'male', 'female').

import pandas as pd

# Read the results
df = pd.read_table("myseg.csv")

# Compute the length of each sequence
df["length"] = df['stop'] - df['start']

# Compute the aggregated length of all sequences by label
df_aggregated = df[['labels', 'length']].groupby("labels").sum()

The Segmenter class constructor accepts the following optional arguments:

• vad_engine (default: 'smn'): Chooses the Voice Activity Detection engine.
  • 'smn': The recent engine that splits signals into speech, music, and noise.
  • 'sm': An older engine that splits signals into speech and music (noise is categorized as either speech or music).
• detect_gender (default: True): If True, performs gender segmentation on speech segments, outputting labels 'female' or 'male'. If False, outputs the label 'speech' (this mode is faster).
• ffmpeg: Allows you to provide a path to a specific ffmpeg binary instead of using the system default.

The Segmenter class is the primary entry point for the inaSpeechSegmenter Python API. It is used to split audio signals into zones of speech, music, and noise, and can optionally perform gender segmentation on speech segments.

# Note: The following is a conceptual usage based on the Segmenter class description
from inaSpeechSegmenter import Segmenter

# Initialize the segmenter with specific options
segmenter = Segmenter(vad_engine='smn', detect_gender=True)

# Perform segmentation (actual method call depends on implementation)
# segments = segmenter(audio_file)

The gd_model_criteria parameter determines which gender detection model is used during the scoring process. Available options are:

• "bgc" (default): A Multi-layer Perceptron trained on all data, providing the best BGC (as per the Interspeech 2023 paper).
• "vfp": A Multi-layer Perceptron trained on French CommonVoice, providing the best VFP (as per the Interspeech 2023 paper).

Pass a media file (e.g., an .mp3 file path or URL) to the Segmenter instance. The returned segmentation object contains the detected segments and their labels.

media = 'https://github.com/ina-foss/inaSpeechSegmenter/raw/master/media/musanmix.mp3'
segmentation = seg(media)

The ina_speech_segmenter.py binary allows you to segment multimedia archives in any format supported by ffmpeg. The tool provides two output formats: csv (compatible with Sonic Visualiser) and TextGrid (Praat format). Use the --help flag to see all available command-line options.

# get help
$ ina_speech_segmenter.py --help