DOCSEARCHbyAlgolia
Submit Docs

DeepSqueak v3 Documentation

repository·master·Indexed 19 days ago

https://github.com/drcoffey/deepsqueak

A machine vision-based bioacoustics research tool for automating the detection of ultrasonic vocalizations in audio recordings. DeepSqueak v3 utilizes neural networks like YOLO V2 for detection, Variational Auto Encoders (VAEs) for clustering, and provides capabilities for refining detections and training custom species detectors. Includes documentation on the DeepSqueakFeed utility for converting JSON data to .mat files and detailed specifications for the audiodata and Calls data structures.

Tokens
1.4K
Snippets
2
Records
4
Agent score
15%

What's inside DeepSqueak

  1. Overview of DeepSqueak v3

    master

    DeepSqueak v3 is a bioacoustics research tool that uses machine vision to accelerate the detection and analysis of ultrasonic vocalizations. It utilizes neural network architectures (such as YOLO V2) to identify acoustic features within spectrograms.

    Key capabilities include:

    • Detection: Using high-precision neural networks to detect vocalizations.
    • Refinement: Navigating entire audio files to refine existing detections or manually add new bounding boxes.
    • Training: Retraining existing networks with custom recordings or training new species detectors from scratch using manual bounding box annotations.
    • Clustering: Using Variational Auto Encoders (VAEs) for contour-invariant clustering of detections, supported by a dedicated clustering GUI.

  2. Use DeepSqueakFeed to convert JSON to .mat files

    master

    DeepSqueakFeed is a utility designed to facilitate interaction between DeepSqueak and other programs. It uses JSON files as intermediaries. If you provide data in the specific DeepSqueak JSON format, this script will transform it into the required .mat files used by the software.

    To prepare data for conversion, you can use a Python workflow where each audio.wav file is accompanied by an audio_labels.txt file. The label file should be a tab-separated value (TSV) file containing the start timestamp, end timestamp, and a label for each call.

    Example audio_labels.txt content:

    0.1234  0.5678  call
1.2345  1.6789  call
    # No direct command provided in source, but describes a workflow of providing JSON to transform into .mat

  3. JSON schema for DeepSqueak data

    master

    The JSON file used by DeepSqueakFeed must follow a specific structure that mirrors the internal .mat file format. It consists of two primary objects: audiodata and Calls.

    audiodata object

    Contains metadata describing the source .wav file:

    • Filename: Full path to the .wav file (string).
    • CompressionMethod: Compression method used (string).
    • NumChannels: Number of audio channels (number).
    • SampleRate: Samples per second (number).
    • TotalSamples: Total number of samples (number).
    • Duration: Duration in seconds (number).
    • Title: Title of the audio file (string).
    • Comment: File comment (string).
    • Artist: The artist or rodent that produced the file (string).
    • BitsPerSample: Number of bits per sample (number).

    Calls object

    Contains arrays representing labeled calls. Crucially, the arrays Box, Score, Type, and Accept must all have the same length.

    • Box: An array of arrays, where each inner array defines a spectrogram box: [lower_time, lower_frequency, time_extent, frequency_extent].
    • Score: An array of confidence scores (numbers).
    • Type: An array of call labels/types (strings).
    • Accept: An array of booleans determining if the call is used for training.
    {
    "audiodata": {
        "Filename": "..\\Annotated audio files\\example\\audio.wav",
        "CompressionMethod": "NONE",
        "NumChannels": 2,
        "SampleRate": 48000,
        "TotalSamples": 140393039,
        "Duration": 2924.8549791666665,
        "Title": "audio.wav",
        "Comment": "",
        "Artist": "",
        "BitsPerSample": 16
    },
    "Calls": {
        "Box": [
            [1801.644187, 0.0, 0.147222999993993, 60.0],
            [1801.79141, 0.0, 0.20243099999332, 60.0]
        ],
        "Score": [1.0, 1.0],
        "Type": ["Call", "Call"],
        "Accept": [true, true]
    }
}

  4. Structure of DeepSqueak .mat files

    master

    DeepSqueak .mat files contain three variables: Calls, audiodata, and detection_metadata. For manual creation or integration, only Calls and audiodata are strictly necessary.

    Calls Table

    This is a table containing four parallel lists (all of length $N$, where $N$ is the number of calls):

    1. Box: An $N imes 4$ matrix of doubles. Each row is [lower_x, lower_y, extension_x, extension_y], where $x$ is time and $y$ is frequency.
    2. Score: A double representing the network confidence. For manual labels, set this to 1.0.
    3. Type: A categorical list containing the call labels.
    4. Accept: A logical (boolean) list determining if the call is accepted (e.g., for training).

    audiodata Struct

    Used to locate and read the original .wav files. Members include:

    • Filename (char): Full path to the .wav file.
    • CompressionMethod (char): Compression method string.
    • NumChannels (double): Number of channels.
    • SampleRate (double): Samples per second.
    • TotalSamples (double): Total samples.
    • Duration (double): Duration in seconds.
    • Title (double/string): Audio file title.
    • Comment (double/string): File comment.
    • Artist (double/string): Producer/Rodent name.
    • BitsPerSample (double): Bits per sample.