DOCSEARCHbyAlgolia
Submit Docs

OpenSoundscape (OPSO)

repository·master·Indexed 42 days ago

https://github.com/kitzeslab/opensoundscape

An open-source Python library for scalable acoustic data analysis in ecology and conservation. Version 0.13.2 provides a pipeline for loading audio, generating spectrograms, training deep learning models (CNNs), and detecting bioacoustic vocalizations.

Tokens
42.8K
Snippets
138
Records
175
Agent score
53%

What's inside OpenSoundscape

  1. Overview of OpenSoundscape (OPSO) capabilities

    master

    OpenSoundscape (OPSO) is an open-source Python library designed for analyzing bioacoustic data. It provides a suite of utilities that can be composed into data analysis pipelines.

    Key capabilities include:

    • Audio Processing: Loading and manipulating audio files, including loading audio starting at specific real-world timestamps.
    • Spectrograms: Creating and manipulating spectrograms.
    • Machine Learning:
      • Training convolutional neural networks (CNNs) on spectrograms using PyTorch.
      • Running pre-trained CNNs to detect vocalizations.
      • Training and using models from the Bioacoustics Model Zoo (e.g., BirdNET or Perch).
      • Using Lightning for training.
    • Specialized Detection: Detecting periodic vocalizations using RIBBIT.
    • Annotation Management: Loading and manipulating Raven annotations.
    • Acoustic Localization: Estimating the location of sound sources from synchronized recordings.

  2. Understand CNN training concepts: Epochs, Batches, and Tensors

    master

    When training a Convolutional Neural Network (CNN), the following terms define the learning process:

    • Tensor: An array of numbers representing a single sample (e.g., a spectrogram).
    • Batch: A collection of samples (e.g., 128 samples) processed together. CNNs learn more effectively from batches than from individual samples.
    • Epoch: One complete pass through the entire set of training samples. The model typically needs to see the data multiple times (multiple epochs) to learn.
    • Validation Dataset: A separate dataset used at the end of each epoch to assess performance. The model's weights are not updated based on validation data; it serves as a fair, 'unseen' test to prevent memorization (overfitting).

  3. Evaluate performance using Score Histograms

    master

    Score histograms are used to visualize the distribution of scores for true positives and true negatives in a test set. This helps determine how well the classifier differentiates between target sounds and background noise.

    Interpreting Histograms

    • Good Performance: Clear separation between the true positive distribution and the true negative distribution.
    • Poor Performance: Significant overlap between the two distributions, or a bimodal distribution where true positives cluster at the same score as true negatives.
    • Common Issues:
      • Left Skew in True Positives: May be caused by partial vocalizations in the test set clips.
      • Bimodal Distributions: May indicate the presence of a different vocalization type that was not well-represented in the training dataset.

  4. Evaluate performance using Precision and Recall

    master

    Classifier performance is often measured by calculating precision and recall at various score thresholds.

    Key Metrics

    • Score Threshold: The value above which a clip is predicted as 'present' and below which it is 'absent'.
    • True Positive (TP): Correctly identified target sound.
    • False Positive (FP): Clip predicted as present that actually lacks the target sound.
    • False Negative (FN): Clip predicted as absent that actually contains the target sound.
    • True Negative (TN): Correctly identified absence of the sound.

    Calculation

    • Precision: $\frac{\text{True Positives}}{\text{All Predicted Positives}}$ (How many of the predicted 'hits' were actually correct?)
    • Recall: $\frac{\text{True Positives}}{\text{All Actual Positives}}$ (How many of the actual target sounds did the model find?)

    Goal: A general rule of thumb is to aim for $>0.7$ for both precision and recall. Performance can be visualized using PR curves (Precision-Recall curves) or by plotting precision and recall lines against the threshold.

  5. Handling confusion species in CNN training

    master

    When training a CNN, it is critical to account for confusion species—neighboring species that have similar songs to your target species. To minimize interference and improve accuracy, you should include both the target species and their respective confusion species as distinct classes in your CNN training set.

    For example, if studying the Wood Thrush, you should include the Hermit Thrush (its confusion species) as a separate class so the model learns to distinguish between them.

  6. Structure training data frames for classifiers

    master

    Training requires a data frame (a CSV file loaded into memory) that maps audio file paths to their respective class labels. The structure of this data frame depends on the type of model you are training.

    Model Types and Data Formats

    Labels are represented as 1 (presence) or 0 (absence).

    1. Single-class: Identifies only one specific class (e.g., a single species). The data frame contains a path column and one class column.
    2. Multi-class, Single-target: Identifies one of several possible species. Each row must contain exactly one 1. If a clip fits no target classes, it is assigned to an additional class (e.g., Other, absent, or negative).
    3. Multi-class, Multi-target: Identifies multiple species simultaneously. Each row can have zero, one, or multiple 1s.

    Data Frame Examples

    Multi-class, Single-target (Exactly one '1' per row):

    fileSpecies_ASpecies_BOther
    /Path/to/clip1.wav100
    /Path/to/clip2.wav010
    /Path/to/clip3.wav001

    Multi-class, Multi-target (Zero, one, or multiple '1's per row):

    fileSpecies_ASpecies_B
    /Path/to/clip1.wav10
    /Path/to/clip2.wav01
    /Path/to/clip3.wav11
    /Path/to/clip4.wav00

  7. Understand training schedules: steps vs epochs

    master

    OpenSoundscape v0.13.0+ uses a step-based training schedule. Instead of tracking training by epochs (the number of times the model sees the entire dataset), the duration of training and the learning rate schedule are determined by the number of steps (where one step equals one batch of clips).

    • Default training steps: 1000
    • Epochs: While the system is step-based, the concept of an epoch (multiple batches viewed once) is still relevant for understanding how many times the model has seen the data.
    • Overfitting: Setting training duration too high may cause the model to overfit (learning only the specific clips in the training set). Setting it too low may prevent the model from learning sufficient features.

  8. Understanding sound classification types

    master

    Sound classification involves assigning a 'class' (e.g., a specific bird species, a chainsaw, or a frog chorus) to audio data. OpenSoundscape and related tools typically use one of the following approaches:

    • Convolutional Neural Networks (CNNs): The most common method for complex or highly variable sounds (like songbird songs). CNNs use spectrograms (visual representations of sound showing timing, pitch, and loudness) to assign confidence scores to specific classes.
    • Signal Processing Algorithms: Best for simple, highly consistent, or periodic sounds (e.g., certain frog calls or Ruffed Grouse drums). An example is RIBBIT (Repeat-Interval Based Bioacoustic Identification Tool).
    • Clustering: Groups similar sounds together for manual review (e.g., Kaleidoscope).
    • Template Matching: Slides a template spectrogram over audio to score matches (e.g., RavenPro).

  9. Configure preprocessing and data augmentation

    master

    Preprocessing and augmentation define how audio is converted into tensors for the CNN. These parameters are applied during the training process.

    Preprocessing Parameters

    • bandpass range: Frequency limits applied to the audio.
    • window samples: Number of samples per window.
    • overlap samples: Number of overlapping samples between windows.
    • overlays & overlay weights: The weight (e.g., 0.5) or a random range (e.g., 0.3-0.7) applied when averaging an overlay image with the underlying training image.

    Data Augmentation

    To improve generalization, OpenSoundscape applies random augmentations to images in each epoch.

    • Default Augmentations: Horizontal and vertical gray bars are overlaid, and the tensor is slightly trimmed around the edges. These shifts and overlays change slightly from epoch to epoch.
    • Customization: You can modify augmentation by removing defaults or adding pre-built augmentations like custom overlays.

  10. How Convolutional Neural Networks (CNNs) work

    master

    A CNN is a specialized neural network designed for 2D data (like spectrogram images) that leverages spatial relationships.

    Neural Networks

    Consist of layers of nodes and connections called weights (or parameters). Training involves adjusting these weights so that input data, when multiplied by the weights and summed, produces an output score (often passed through a sigmoid function to scale between 0 and 1) that matches the target label.

    Convolutional Layers

    Instead of having a unique weight for every single pixel, CNNs use kernels (small matrices of parameters, e.g., 3x3 or 7x7).

    1. A kernel slides (convolves) across the input image.
    2. At each position, the kernel's values are multiplied by the underlying pixel values and summed.
    3. This allows the network to recognize spatial patterns (like vertical lines or specific shapes) regardless of where they appear in the image.
    4. Subsequent layers combine these simple shapes into increasingly complex features.

    Fully Connected Layers

    After the convolutional layers have extracted features, the final layers of the CNN act like a standard neural network, mapping those features to specific output nodes (one for each target class).

  11. Assemble an appropriate test set for CNN evaluation

    master

    To accurately assess a classifier, you must use a test set that mimics real-world settings. This set is used for prediction outside of the training process and should not be used to decide which model version is 'best' during training (that is the role of the validation set).

    Best Practices for Test Sets

    • Avoid targeted recordings: Using recordings similar to your training data will not provide an accurate estimate of real-world performance.
    • Use annotated field data: Ideally, use data representative of the environment where the model will be deployed.
    • Composition: A good test set should include several dozen positive examples of the target sound. You can combine field data (for target species) with fully-annotated soundscapes (for non-target sounds) to test performance across diverse data.