DOCSEARCHbyAlgolia
Submit Docs

omnizart

repository·main·Indexed 23 days ago

https://github.com/music-and-culture-technology-lab/omnizart

A Python library for automatic music transcription capable of transcribing pitched instruments, vocal melodies, chords, drum events, and beats from polyphonic music. Version 0.6.3 includes a CLI for various transcription modes (e.g., music-piano, chord, drum, vocal) and tools for NNLS Chroma analysis, Chordino chord transcription, and pitch estimation. It supports installation via Docker, Conda, and Google Colab, though it is incompatible with ARM-based MacOS.

Tokens
24.9K
Snippets
36
Records
210
Agent score
83%

What's inside omnizart

  1. Overview of Omnizart capabilities

    main

    Omnizart is a Python library designed for Automatic Music Transcription (AMT). It provides a streamlined solution for transcribing various music contents and performing related Music Information Retrieval (MIR) tasks.

    Core Transcription Capabilities:

    • Piano Solo Transcription: Frame-level and note-level transcription for solo piano.
    • Multi-instrument Polyphonic Transcription: Supports 11 output classes (piano, violin, viola, cello, flute, horn, bassoon, clarinet, harpsichord, contrabass, and oboe).
    • Drum Transcription: Transcription of drum events in polyphonic music.
    • Vocal Transcription: Pitch extraction and note segmentation for singing voices in polyphonic music.

    Related MIR Tasks:

    • Chord Recognition: Recognizes 25 chord types (including major, minor, and absence of chord) from audio input.
    • Beat/Downbeat Tracking: Targets symbolic data (MIDI) to predict beat and downbeat positions with 10 ms resolution.

  2. Extract audio features with omnizart.feature

    main

    The omnizart.feature module provides tools for extracting various musical and rhythmic features from audio files. Supported feature extraction modules include:

    • CFP: Constant Frequency Perceptual features.
    • HCFP: High-resolution Constant Frequency Perceptual features.
    • CQT: Constant-Q Transform features.
    • Beat Tracking: Rhythmic feature extraction, specifically via the beat_for_drum module.

  5. Configure feature extraction settings

    main
    Feature extraction settings in Omnizart are managed via the omnizart.constants.feature module. These settings record the parameters used during the extraction process. While the module provides programmatic access to these constants, you can also adjust the default settings by modifying the YAML files located in the defaults/*.yaml directory of the project.

  6. Understand the module feature generation process flow

    main

    When implementing a module, the overall process flow follows these steps:

    1. Determine Dataset Type: Identify the dataset type from the provided dataset path.
    2. Select Dataset Structure: Choose the corresponding dataset structure class.
    3. Parse Files: Parse the audio and ground-truth file pairs.
    4. Verify Output Path: Ensure the feature output path exists.
    5. Parallel Generation: Generate feature and label representations in parallel.
    6. Save Metadata: Write the module settings to the output path as a *.success.yaml file.

  7. How chord recognition works in Omnizart

    main

    Omnizart implements chord recognition using the Harmony Transformer (HT) architecture.

    Key details for users:

    • Input Type: Currently supports audio inputs only (does not support symbolic/MIDI inputs for this specific task).
    • Pre-processing: Audio is pre-processed using the Chordino VAMP plugin to generate a non-negative-least-squares chromagram.
    • Output: The model outputs 25 chord types (covering 12 major and minor chords plus a class for the absence of a chord) with a time resolution of 230 ms.

  8. Configure music model settings via MusicSettings

    main

    Music model configurations are managed by the omnizart.setting_loaders.MusicSettings class. These settings are loaded from a YAML configuration (found at omnizart/defaults/music.yaml).

    Attribute Mapping Rules:

    • Attributes are converted from PascalCase to snake_case (e.g., HopSize becomes hop_size).
    • YAML paths are transformed into object attributes by removing the /Settings level. For example, a YAML path General/Training/Settings/BatchSize is accessed via MusicSettings.training.batch_size.

  9. How beat and downbeat tracking works in Omnizart

    main

    Unlike many MIR tools that focus on audio, Omnizart's beat and downbeat tracking model targets symbolic data (MIDI).

    Key details for users:

    • Input Type: MIDI files.
    • Input Representation: The model combines piano-roll, spectral flux, and inter-onset interval extracted from the MIDI data.
    • Output: Predicts beat and downbeat positions with a time resolution of 10 ms using a two-layer BLSTM network with an attention mechanism.

  10. How the omnizart CLI workflow works

    main

    The omnizart command-line interface follows a consistent pipeline pattern for its core sub-commands:

    omnizart <application> <action> <arguments>

    Core Components

    1. Applications: The specific domain of music processing. Supported applications include:

      • music: Pitched instruments (MIDI output).
      • drum: Percussive instruments (MIDI output).
      • chord: Chord progressions (MIDI or CSV output).
      • vocal: Note-level vocal melody (MIDI output).
      • vocal-contour: Frame-level vocal melody (F0 in text).
      • beat: Beat position (expects MIDI input).

    2. Actions: The operation to perform on an application. All applications share these three actions:

      • transcribe: Converts audio (WAV) or MIDI to musical notation/events.
      • generate-feature: Processes datasets into features required for training/testing.
      • train-model: Trains a new model from scratch using generated features.

    3. Arguments: Specific flags and parameters required for the chosen action.

  11. Install Omnizart from source using Conda

    main

    To install Omnizart from the source repository using Conda, follow these steps:

    1. Clone the repository.
    2. Create a new environment from the provided environment.yml.
    3. Activate the environment.
    4. Install the package locally using pip install ..
    5. Download the checkpoints.
    git clone https://github.com/Music-and-Culture-Technology-Lab/omnizart
cd omnizart
# Create a new conda environment
conda env create -f environment.yml
conda activate omnizart
# Install omnizart
pip install .
omnizart download-checkpoints

  12. Install Omnizart for development

    main

    If you want to install Omnizart from source for development purposes, clone the repository and use make install. This will create a virtual environment under the omnizart/ directory by default and automatically download the necessary checkpoints. To install development-specific dependencies, use poetry install.

    # Clone the omnizart repository from GitHub
git clone https://github.com/Music-and-Culture-Technology-Lab/omnizart.git

# Install dependencies, with checkpoints automatically downloaded
cd omnizart
make install

# Install Dev dependencies
poetry install