DOCSEARCHbyAlgolia
Submit Docs

audioFlux Documentation

repository·master·Indexed 25 days ago

https://github.com/libaudioflux/audioflux

A high-performance deep learning tool library for audio and music analysis. It provides modules for time-frequency transformations (transform), feature extraction (feature), and Music Information Retrieval (mir), including support for BFT, NSGT, CWT, PWT, and various frequency scales. The library includes utilities for visualization via audioflux.display and supports installation via pip, conda, and builds for iOS and Android.

Tokens
13.6K
Snippets
28
Records
70
Agent score
83%

What's inside audioflux

  1. Overview of audioFlux functionality

    master

    audioFlux is a library designed for systematic, comprehensive, and multi-dimensional audio feature extraction and combination. It is optimized for tasks such as Audio Classification, Speech Enhancement, Audio/Music Separation, Music Information Retrieval (MIR), and Automatic Speech Recognition (ASR).

    The library provides common features like mel spectrogram and mfcc for both traditional machine learning and deep learning workflows.

    Key architectural features:

    • Data Flow Design: Built on a data flow architecture that decouples algorithm modules structurally.
    • Efficiency: Designed to be convenient, fast, and efficient for extracting features from large batches of audio data.

  2. Overview of audioFlux modules

    master

    audioFlux is a deep learning tool library for audio and music analysis. It is organized into three main functional modules:

    1. transform: Provides time-frequency representation algorithms. Includes multi-scale transforms like BFT (Based Fourier Transform), NSGT (Non-Stationary Gabor Transform), CWT (Continuous Wavelet Transform), and PWT (Pseudo Wavelet Transform). It also supports sharpening techniques like reassign, synsq, and wsst.

    2. feature: Contains algorithms for feature extraction, such as spectral (spectrum features), xxcc (Cepstrum coefficients), deconv (Deconvolution), and chroma (Chroma features).

    3. mir: Music Information Retrieval module containing algorithms for pitch (e.g., YIN, STFT), onset (e.g., spectrum flux, novelty), and hpss (Harmonic Percussive Source Separation via median filtering or NMF).

  4. Visualize audio data with audioflux.display

    master

    The audioflux.display module provides utilities for visualizing audio signals and spectral data. It includes functions for filling specific components of a plot, filling waveforms, and creating plot objects.

    Key components include:

    • fill_spec: Likely used for filling spectral representations.
    • fill_wave: Used for filling waveform representations.
    • fill_plot: Used for filling general plot components.
    • Plot: A class or interface for managing plot objects.

  5. Explore MIR algorithms in audioFlux

    master

    The mir (Music Information Retrieval) module in audioFlux provides several core algorithms for audio analysis and manipulation. Available algorithms include:

    • Pitch Estimation: Uses algorithms like YIN and STFT to estimate pitch.
    • Onset Detection: Uses algorithms like flux and novelty to detect onsets.
    • HPSS (Harmonic and Percussive Source Separation): Uses median filtering or NMF (Non-negative Matrix Factorization) to separate harmonic and percussive components.
    • Harmonics: Extraction of harmonic components.
    • Pitch Shifting: Algorithms for changing the pitch of an audio signal.
    • Time Stretching: Algorithms for changing the duration of an audio signal without affecting pitch.

  6. Available feature extraction algorithms in audioFlux

    master

    The feature module in audioFlux provides a variety of algorithms for audio feature extraction. These algorithms support multiple time-frequency representations, including:

    • BFT (Binary Fourier Transform)
    • NSGT (Non-Stationary Gabor Transform)
    • CWT (Continuous Wavelet Transform)
    • PWT (Paul Wavelet Transform)
    • CQT (Constant-Q Transform)
    • VQT (Variable-Q Transform)
    • ST (Short-Time Fourier Transform)
    • FST (Fractional Short-Time Fourier Transform)
    • DWT (Discrete Wavelet Transform)
    • WPT (Wavelet Packet Transform)
    • SWT (Stationary Wavelet Transform)

    Supported algorithm modules include spectral, xxcc, deconv, and chroma (which specifically supports CQT).

  9. Build audioflux for iOS

    master

    To compile for iOS on a Mac, you must have the full Xcode package and Xcode Command Line Tools installed. You can install the tools using xcode-select --install. Once ready, navigate to the audioFlux project scripts directory and run the build script.

    $ xcode-select --install

# Navigate to scripts directory and run:
$ ./build_iOS.sh

  11. Use FeatureExtractor for batch feature extraction

    master

    The FeatureExtractor class allows for batch feature extraction from audio signals. You can initialize it with specific transforms (such as 'bft', 'cwt', or 'cqt'), the sampling rate, and scale configurations. Once initialized, you can extract spectrograms and subsequently derive spectral features, cross-correlation (xxcc), or deconvolution results from that spectrogram.

    import audioflux as af
from audioflux.type import SpectralFilterBankScaleType

# Load an audio file
sample_path = af.utils.sample_path('880')
audio_arr, sr = af.read(sample_path)

# Create FeatureExtractor object
fa_obj = af.FeatureExtractor(
    transforms=['bft', 'cwt', 'cqt'], 
    samplate=sr, 
    radix2_exp=12,
    scale_type=SpectralFilterBankScaleType.OCTAVE
)

# Extract spectrogram
spec_result = fa_obj.spectrogram(audio_arr, is_continue=True)

# Extract spectral features (e.g., flux)
spectral_result = fa_obj.spectral(spec_result, spectral='flux', spectral_kw={'is_positive': True})

# Extract xxcc (cross-correlation)
xxcc_result = fa_obj.xxcc(spec_result, cc_num=13)

# Extract deconvolution
deconv_result = fa_obj.deconv(spec_result)

  12. Build audioflux for Android

    master

    Android builds require Android NDK (version >= 16). You must set the NDK_ROOT environment variable to your NDK installation path and update your PATH. Additionally, the build uses the fftw library; you must compile the single-floating point version for Android and copy it to the audioFlux project scripts/android/fftw3 directory before building. Run the build script from the scripts directory.

    # Set NDK environment variables
$ export NDK_ROOT=~/Android/android-ndk-r16b
$ export PATH=$NDK_ROOT:$PATH

# Run build from the scripts directory
$ ./build_android.sh