DOCSEARCHbyAlgolia
Submit Docs

torchaudio

repository·main·Indexed 25 days ago

https://github.com/pytorch/audio

An audio library for PyTorch providing tools for audio and speech processing, optimized for machine learning workflows with GPU acceleration and autograd support. The library includes examples for Automatic Speech Recognition (ASR) using Emformer RNN-T, Conformer RNN-T (including TCPGen biasing), and CTC beam search decoders for wav2vec 2.0 and Zipformer models.

Tokens
29.5K
Snippets
69
Records
174
Agent score
81%

What's inside torchaudio

  1. Overview of torchaudio

    main

    torchaudio is a machine learning library designed to apply PyTorch to the audio domain. It focuses on processing audio data for ML rather than being a general signal processing library. Key features include:

    • GPU Acceleration: Leverages PyTorch for strong GPU support.
    • Autograd Integration: Focuses on trainable features through the PyTorch autograd system.
    • Dataloaders: Provides loaders for common audio datasets.
    • Processing Functions: Includes functions like forced_align.
    • Transforms: Common audio transforms such as Spectrogram, AmplitudeToDB, MelScale, MelSpectrogram, MFCC, MuLawEncoding, MuLawDecoding, and Resample.
    • Compliance Interfaces: Provides implementations that align with other libraries, such as Kaldi (e.g., spectrogram, fbank, mfcc).

    Note: torchaudio has transitioned into a maintenance phase. Some user-facing features were deprecated in version 2.8 and removed in 2.9 to reduce redundancy and focus on its core strength: processing audio data for ML.

  2. Verify Autograd and TorchScript support

    main

    TorchAudio uses specific badges in API references to indicate verified feature support. If an API lacks these badges, it may be incompatible or simply untested:

    • Autograd: APIs with this support can correctly backpropagate gradients. APIs without this badge might throw errors during backpropagation or silently return incorrect gradients.
    • TorchScript: APIs with this support can be serialized and executed in non-Python environments.

  3. Understand the TorchAudio maintenance phase status

    main

    TorchAudio has transitioned into a maintenance phase. The project's focus has shifted to reducing redundancies with the PyTorch ecosystem and tightening its scope to audio data processing for ML.

    Important considerations for developers:

    • The examples located in the examples/ directory are provided for reference only and are no longer maintained. They may not work as expected.
    • For up-to-date, hands-on examples of supported features, use the scripts found in the tutorials/ subdirectory instead.
    • For more context, refer to the community message.

  4. Use torchaudio.functional for audio utility operations

    main

    The torchaudio.functional module provides a wide range of utility functions for common audio processing tasks, including unit conversions, masking, resampling, and signal manipulation.

    Key utility categories include:

    • Unit Conversions: amplitude_to_DB, DB_to_amplitude.
    • Spectral/Feature Utilities: melscale_fbanks, linear_fbanks, create_dct.
    • Masking: mask_along_axis, mask_along_axis_iid.
    • Encoding: mu_law_encoding, mu_law_decoding.
    • Signal Processing: resample, loudness, convolve, fftconvolve, add_noise, preemphasis, deemphasis, speed, frechet_distance.

  5. Important notice on Torchaudio maintenance and TorchCodec migration

    main

    Starting with version 2.8, Torchaudio has transitioned into a maintenance phase. Key changes include:

    • Some APIs were deprecated in version 2.8 and removed in version 2.9.
    • Audio and video decoding/encoding capabilities in PyTorch have been consolidated into TorchCodec.

    Users should check for deprecation warnings and consider migrating decoding/encoding tasks to TorchCodec.

  6. Use torchaudio.models for audio task architectures

    main

    The torchaudio.models subpackage provides definitions for various model architectures designed to address common audio tasks. These models are responsible for constructing computation graphs and executing them.

    Note: If you are looking for models with pre-trained parameters (weights), you should use the torchaudio.pipelines module instead of torchaudio.models.

  7. Understand the DNN MVDR Beamforming Example architecture

    main

    The DNNBeamformer model is a composite architecture designed for speech enhancement using Time-Frequency (T-F) masks. It utilizes the following components:

    • torchaudio.transforms.Spectrogram: Applies Short-time Fourier Transform (STFT) to the input waveform.
    • ConvTasNet (without encoder/decoder): Predicts T-F masks for speech and noise.
    • torchaudio.transforms.PSD: Computes covariance matrices for speech and noise.
    • torchaudio.transforms.SoudenMVDR: Estimates the complex-valued STFT for the enhanced speech.
    • torchaudio.transforms.InverseSpectrogram: Applies inverse STFT (iSTFT) to generate the enhanced waveform.

  8. Modularized Self-supervised Learning Recipe Overview

    main
    The self_supervised_learning directory provides a modularized training recipe designed for audio and speech self-supervised learning. The architecture allows users to inject new components—such as models, data_modules, or loss functions—into the existing recipe to support different tasks like Wav2Vec 2.0 or HuBERT.

  9. Understand the torchaudio.pipelines Bundle concept

    main

    The torchaudio.pipelines module uses the Bundle class to package pre-trained models with their required support functions (feature extraction, transforms, etc.) and metadata.

    Instead of manually managing sampling rates, FFT bins, or post-processing steps, you instantiate a Bundle instance. Different model implementations can share the same Bundle interface, allowing you to swap models (e.g., switching from ConvTasNet to HDemucs in a SourceSeparationBundle) without changing your client code.

  10. Understand torchaudio feature release classifications

    main

    Features in torchaudio are categorized by their stability and support level. When choosing APIs for production or research, consider these classifications:

    • Stable: Maintained long-term with minimal performance limitations and complete documentation. Backwards compatibility is generally maintained, though breaking changes may occur with one release's notice.
    • Beta: Features that may undergo API changes based on feedback, performance improvements, or operator coverage. While torchaudio commits to moving Beta features to Stable, backwards compatibility is not guaranteed.
    • Prototype: Early-stage features for testing and feedback. These are typically not included in standard binary distributions (like PyPI or Conda) and may require specific run-time flags to access.

  11. Merge LRS3 labels

    main

    Once preprocessing is complete, use merge.py to consolidate all labels.

    Run the following command:

    python merge.py \
    --root-dir=[root_dir] \
    --dataset=[dataset] \
    --subset=[subset] \
    --seg-duration=[seg_duration] \
    --groups=[n]

    Arguments:

    • --root-dir: Path to the root directory where all preprocessed files are stored.
    • --dataset: Name of the dataset. Valid values: lrs2, lrs3.
    • --subset: The subset name.
      • For lrs2: train, val, test.
      • For lrs3: train, test.
    • --seg-duration: Length of the maximal segment in seconds. Default: 16.
    • --groups: Number of groups to split the dataset into.
    python merge.py \
    --root-dir=[root_dir] \
    --dataset=[dataset] \
    --subset=[subset] \
    --seg-duration=[seg_duration] \
    --groups=[n]