DOCSEARCHbyAlgolia
Submit Docs

MidiTok

repository·main·Indexed 21 days ago

https://github.com/natooz/miditok

A Python package for tokenizing MIDI and ABC music files into sequences optimized for Deep Learning models, such as Transformers. It supports multiple tokenization schemes including REMI, REMI+, MIDI-Like, TSD, Structured, CPWord, Octuple, MuMIDI, MMM, and PerTok. The library provides tools for training tokenizers using BPE, Unigram, or WordPiece, as well as utilities for PyTorch dataset preparation, data augmentation, and integration with Hugging Face.

Tokens
23.6K
Snippets
54
Records
93
Agent score
74%

What's inside miditok

  1. Overview of MidiTok

    main

    MidiTok is a Python package designed for MIDI file tokenization. It converts symbolic music files (such as MIDI and ABC) into sequences of tokens suitable for machine learning models like Transformers. This is useful for tasks involving music generation, transcription, or Music Information Retrieval (MIR).

    Key features include:

    • Support for various MIDI tokenization methods.
    • Ability to train tokenizers using BPE (Byte Pair Encoding), Unigram, or WordPiece.
    • Integration with the Hugging Face Hub for pushing and pulling trained tokenizers.

  2. Benchmark tokenization performance

    main

    MidiTok provides benchmarks for tokenization times across different MIDI datasets including Maestro, Lakh (MMD), and POP909. The benchmarks compare various tokenization strategies such as REMI, TSD, MIDILike, and Structured.

    Performance results are measured in milliseconds (ms) per file. Based on the provided benchmarks, REMI is generally the fastest tokenization method across the tested datasets.

    |            | Maestro        | MMD            | POP909        |
|:-----------|:---------------|:--------------|:--------------|
| REMI       | 38.97±32.92 ms | 24.55±52.25 ms | 11.00±7.73 ms |
| TSD        | 52.62±41.59 ms | 31.70±73.93 ms | 13.35±7.66 ms |
| MIDILike   | 61.75±48.27 ms | 36.28±76.87 ms | 17.77±8.91 ms |
| Structured | 60.38±46.78 ms | 35.85±88.48 ms | 16.56±8.62 ms |

  3. Evaluate the impact of WordPiece `max_input_chars_per_word`

    main

    The max_input_chars_per_word parameter in WordPiece models affects training and encoding performance:

    1. Training Time: This parameter has almost no impact on the overall training time of the tokenizer.
    2. Encoding Time: Increasing max_input_chars_per_word has a significant negative impact on the encoding time of token IDs.
    3. Data Integrity (Unknown Tokens): If you do not split tokens per bars or beats, you face a dilemma:
      • Low max_input_chars_per_word values lead to a high proportion of unknown tokens (losing data integrity).
      • High max_input_chars_per_word values lead to very high encoding times.

    Recommendation: Use bar or beat splitting to maintain high data integrity without requiring extremely high max_input_chars_per_word values.

  4. How to create a custom MusicTokenizer

    main

    You can implement a custom tokenization scheme by creating a class that inherits from miditok.MusicTokenizer. To build a functional tokenizer, you must override the following core methods:

    • _add_time_events: Creates time events from global and track events.
    • _tokens_to_score: Decodes tokens back into a Score object.
    • _create_vocabulary: Defines the tokenizer's vocabulary.
    • _create_token_types_graph: Defines the possible token type successions (primarily used for evaluation).

    If you need more granular control, you can also override:

    • _score_to_tokens: The main method that orchestrates specific tokenization methods.
    • _create_track_events: To include special track-level events.
    • _create_global_events: To include special global-level events.
    from miditok import MusicTokenizer

class MyCustomTokenizer(MusicTokenizer):
    def _add_time_events(self, ...):
        # Implementation
        pass

    def _tokens_to_score(self, ...):
        # Implementation
        pass

    def _create_vocabulary(self, ...):
        # Implementation
        pass

    def _create_token_types_graph(self, ...):
        # Implementation
        pass

  5. How MidiTok interoperates with the Hugging Face Hub

    main

    MidiTok implements the huggingface_hub.ModelHubMixin component, allowing it to behave similarly to the Hugging Face Transformers library. This enables you to upload, share, and download tokenizers seamlessly.

    Key behaviors to note:

    • miditok.MusicTokenizer.save_pretrained is equivalent to calling save_params.
    • miditok.MusicTokenizer.from_pretrained can load tokenizers from either the Hugging Face hub or a local filesystem.
    • When using save_pretrained or push_to_hub, you can ignore the config argument as it is intended for models, not tokenizers.
    • You can specify a custom configuration filename using the filename keyword argument in both save_pretrained and from_pretrained. If not provided, the default filename used is tokenizer.json.

  6. How Attribute Controls work for music generation

    main

    Attribute Controls are special tokens used to condition music generation during inference. They function by being placed at the beginning of tracks or bars in a token sequence, allowing a causal model to predict subsequent tokens based on these attributes.

    Key behaviors:

    • Scope: They can operate at either the track-level or the bar-level.
    • Inference: During generation, you can strategically place these tokens at the start of new tracks or bars to control the musical output.
    • Compatibility Warning: Attribute controls are not compatible with "multi-vocabulary" tokenizers (e.g., Octuple) or multitrack "one token stream" tokenizers.

  7. Why you should train a MidiTok tokenizer

    main

    While a freshly created tokenizer can immediately serialize MIDI or ABC files, training it on a specific corpus is highly recommended for two reasons:

    1. Meaningful Embeddings: Training allows the tokenizer to learn new tokens that represent successions of basic attributes (e.g., a whole note or a specific melody pattern). This helps models learn better semantic representations of melody and harmony.
    2. Reduced Sequence Lengths: Using only basic tokens (Pitch, Velocity, Duration, etc.) results in very long sequences (at least 3x the number of notes). Training 'compresses' these sequences by creating single tokens for common successions, which drastically improves Transformer efficiency by reducing the quadratic computational cost of long sequences.

    Note: All tokenizers can be trained except those using embedding pooling (is_multi_voc=True).

  8. Understand the different vocabulary forms in MidiTok

    main

    MidiTok uses the Hugging Face tokenizers library. A token has three forms: a text description (e.g., Pitch_58), an integer ID, and a byte form (a character or succession of characters).

    You can access different vocabulary mappings to translate between these forms:

    • vocab: The base vocabulary (mapping token descriptions to IDs).
    • vocab_model: The vocabulary including learned tokens (mapping byte forms to integer IDs).
    • _vocab_base_byte_to_token: Maps base token byte forms to their string forms.
    • _vocab_base_id_to_byte: Maps base token IDs to their byte forms.
    • _vocab_bpe_bytes_to_tokens: Maps the byte forms of the complete vocabulary to their string forms as a list of strings.

  9. Understanding Sequential Models for Music Generation

    main

    Sequential models (often called language models) are designed to process sequences of discrete elements. In the context of MidiTok, these models take sequences of integer tokens (representing musical attributes like pitch, velocity, or duration) and learn to predict subsequent elements.

    Common architectures include:

    • RNN / LSTM: Recurrent architectures that process sequences.
    • Transformers: Attention-based architectures that can be configured in three ways:
      • seq2seq: Composed of a bi-directional encoder (processes input into hidden states) and a causal decoder (generates output autoregressively). Useful for translation-style tasks.
      • Encoder-only: (e.g., BERT) Best for non-generative tasks like classification.
      • Decoder-only: Designed for content generation. These models are typically trained with teacher forcing to predict the next element and generate music autoregressively (one element at a time by feeding the generated element back into the input).

  10. Tokenizer model: WordPiece

    main

    WordPiece is a subword-based algorithm similar to BPE. It attempts to tokenize data into the fewest tokens possible while ensuring that when a sequence must be split, it is split into tokens with maximum frequency in the training data.

    Usage Warning: WordPiece includes a max_input_chars_per_word attribute. If a succession of base tokens exceeds this length, it is replaced by an unk_token (MidiTok uses the padding token by default). Because music files can have tens of thousands of base tokens, WordPiece should exclusively be used when splitting token IDs per bars or beats to ensure succession lengths stay below this limit.

  11. How MidiTok handles time resolution

    main

    MidiTok manages time by resampling the MIDI file's time division to a new resolution defined by the beat_res attribute in TokenizerConfig. This attribute determines the vocabulary of Duration and TimeShift tokens.

    Time tokens are represented as tuples in the format: (num_beats, num_samples, resolution).

    • Example: (2, 3, 8) represents 2 beats and 3/8 of a beat.
    • Example: (2, 2, 4) represents 2 beats and 1/2 of a beat (2.5).

    For position-based tokenizers, the number of Position tokens in the vocabulary is equal to the maximum resolution found in beat_res.

  12. Understand the difference between symbolic and audio music

    main

    MidiTok is designed for symbolic music, which represents music as a sequence of discrete musical elements (notes, tempos, time signatures) rather than a continuous sound signal (audio).

    Symbolic Music

    • Representation: Successions of notes, often visualized via sheet music or a piano roll.
    • Usage with AI: Because symbolic music can be represented as sequences of tokens, it is most commonly used with discrete sequential models (like Transformers). MidiTok's primary purpose is to convert these symbolic representations into token sequences.

    Audio Music

    • Representation: The physical sound signal, typically represented as waveforms (time domain) or spectrograms (frequency domain).
    • Usage with AI: Audio is a continuous modality. While raw waveforms are often too high-resolution to model directly, audio is frequently processed as spectrograms using CNNs, or compressed into discrete tokens using neural audio codecs (e.g., EnCodec) for use with Transformers.