DOCSEARCHbyAlgolia
Submit Docs

audio-separator

repository·main·Indexed 23 days ago

https://github.com/nomadkaraoke/python-audio-separator

A Python package for audio stem separation (e.g., vocals vs. instrumental) using models such as MDX-Net, VR Arch, Demucs, and MDXC. It provides a Command Line Interface (CLI), a Python API, and support for remote separation via an API server deployed on Modal.com. The library supports hardware acceleration for NVIDIA GPUs (CUDA), Apple Silicon (CoreML), and Windows AMD/Intel GPUs (DirectML), and includes specialized tools for loading and validating Roformer models.

Tokens
102K
Snippets
69
Records
615
Agent score
79%

What's inside audio-separator

  1. Overview of Gsep Audio Separation

    main

    Gsep (available at studio.gaudiolab.io) is an audio separation service capable of isolating 2, 4, 5, or 6 stems (including karaoke modes).

    Key Characteristics:

    • Strengths: Excellent piano extraction, high-quality 'other' stem (often better than Demucs or Audioshake), and robust handling of loud synths/orchestra elements. It is particularly good at removing vocals with fewer 'dips' in the instrumental compared to UVR.
    • Weaknesses: Can occasionally cause 'missing sounds' (sounds lost in the cracks between stems), struggles with heavily processed vocals in choruses, and the electric guitar model can be inconsistent. Bass detection for high notes is limited.
    • Output Formats:
      • Free Trial: MP3 output only, 10-minute input limit.
      • Paid Users: WAV and MP3 output, 20-minute input limit, faster queue, and long-term storage.
    • Input Requirements: For maximum quality (MP3 320kbps output), it is ideal to upload 44kHz files with a minimum 320kbps bitrate. Avoid 48kHz files if possible, as Gsep may match the bitrate of the input file if it is lower than 320kbps.

  2. Use the Remote API for audio separation

    main

    The audio-separator package includes a remote API client. This allows you to perform audio separation by connecting to a deployed API service (such as one hosted on modal.com) instead of running heavy AI models on your local machine. The API uses asynchronous processing with job polling.

    For deployment instructions on Modal, refer to the audio_separator/remote/README.md file in the repository.

  3. Using ZLUDA for CUDA applications on AMD GPUs

    main

    ZLUDA is a translation layer that allows CUDA-written applications to run on AMD (and formerly Intel) GPUs without modification.

    Key Considerations

    • Performance: On high-end cards like the 7900 XT, ZLUDA can significantly speed up processes compared to DirectML. However, weaker GPUs may show performance degradation.
    • Training Warning: Users have reported that torch.stft may not always return correct results when using ZLUDA, making it potentially unreliable for model training. It is better suited for inference.
    • Compatibility:
      • Works on Windows.
      • Supports older AMD GPUs (e.g., RX 500 series) via specific forks (e.g., lshqqytiger/ZLUDA).
      • For Linux/ROCm, compatibility varies by GPU architecture (e.g., RX 5700 XT may require specific HIP/ROCm versions).

  5. Optimize Roformer chunk_size and VRAM usage

    main

    For Roformer models, chunk_size significantly impacts SDR and VRAM consumption. Increasing chunk_size beyond the training value can improve SDR up to a certain peak, after which quality degrades.

    VRAM Constraints (AMD/Intel GPUs):

    • 4GB VRAM: Max chunk_size is approximately 112455 (2.55s) or 132300 (3s).
    • 8GB VRAM: Max chunk_size is approximately 352800 (8.00s) or 485100 (11.00s).

    Conversion Formula: chunk_size = (dim_t - 1) * hop_length

    Common dim_t to chunk_size mappings:

    • dim_t = 801 $\rightarrow$ chunk_size = 352800 (8.00s)
    • dim_t = 1101 $\rightarrow$ chunk_size = 485100 (11.00s)
    • dim_t = 256 $\rightarrow$ chunk_size = 112455 (2.55s)
    • dim_t = 1333 $\rightarrow$ chunk_size = 587412 (13.32s)

  7. Configure BigShifts and Overlap for separation quality

    main

    Two key parameters influence the balance between speed and separation quality:

    • BigShifts: Based on the Demucs 'shift trick', this shifts the audio by 1 second for each pass. For example, bigshifts=3 performs one pass with 0s shift, one with 1s shift, and one with 2s shift. Increasing this can reduce muddiness but significantly increases processing time. Default is 3; values like 5 or 7 are often optimal.
    • Overlap: Controls audio chunk processing. Higher overlap values (e.g., 0.99) generally improve quality and reduce vocal residues but increase processing time and can make results muddier if set too high.

  8. Understand SDR (Signal-to-Distortion Ratio) limitations in audio separation

    main

    SDR is a common metric used to evaluate audio separation models, but it has significant limitations when predicting real-world performance:

    • Averaging Effect: SDR provides an average score based on specific datasets (like the MVSEP synth dataset or the Demixing Challenge 2021). A high SDR score does not guarantee a good result for a specific song, as performance varies by genre and production style.
    • Fullness vs. Bleed: SDR conflates two different issues: how "full" a stem is (presence of all necessary frequencies) and how "bleedless" it is (absence of other instruments). A model might have a high SDR because it is very clean (low bleed) but sounds muffled (low fullness), or vice versa.
    • Human Perception: SDR is a mathematical calculation and does not always align with human hearing. For example, a model with a lower SDR might actually sound better to a human if the bleeding is occasional rather than permanent.
    • Dataset Bias: The MVSEP synth dataset is sample-based and may not reflect actual music characteristics, potentially leading to inflated scores compared to real-world music datasets.

    Recommendation: Use SDR as a starting point for experimentation, but supplement it with human listening or testing on "AI killing tracks" (songs known to challenge separation models) to ensure quality.

  9. Compare Time-Domain vs. Time-Frequency Domain metrics

    main

    When evaluating audio separation models, metrics are categorized into two domains:

    Time-Domain Metrics

    These assess how accurately the waveform matches the original sound. They are often biased toward low frequencies and are less affected by high frequencies.

    • SDR (Signal-to-Distortion Ratio)
    • SiSDR (Scale-Invariant Signal-to-Distortion Ratio)
    • log_wmse (Log Weighted Mean Squared Error): A relatively new metric that is not overly sensitive to low frequencies and allows weighting tailored to human hearing.

    Time-Frequency Domain Metrics

    These compare how closely the spectrogram of the separated stem resembles the original spectrogram. They can evaluate low and high frequencies more equally.

    • l1_freq: A mix between fullness and bleedless that aims to avoid noise issues.
    • aura_stft: Compares spectrograms of a single resolution.
    • aura_mrstft: Compares Mel spectrograms of multiple resolutions (often considered more perceptually relevant).

    Key Limitation: Most time-frequency metrics (including aura_stft and aura_mrstft) are based on amplitude or power spectrograms and therefore ignore phase information.

  10. Optimize Roformer separation parameters in Colab

    main

    When using Roformer models in certain Colab environments, avoid high overlap settings to prevent issues caused by broken overlap implementations.

    To achieve the best Signal-to-Distortion Ratio (SDR), use the following settings:

    • For 1101 segments: Set dim_t=1101 and overlap=2. Do not set overlap higher than 10.
    • For 801 segments: Do not set overlap higher than 8.

  11. Evaluate separation quality using SDR, Fullness, and Bleedless metrics

    main

    When choosing a model, rely on these metrics rather than just SDR (Signal-to-Distortion Ratio):

    • SDR (Signal-to-Distortion Ratio): Measures the ratio of the target signal to the distortion introduced.
    • Fullness: Measures how much of the original instrument's character and body is preserved. High fullness is critical for instruments like piano or guitar to avoid phase distortion.
    • Bleedless: Measures how effectively the model removes the unwanted stem (e.g., how much vocal remains in an instrumental).