repository·main·Indexed 18 days ago
https://github.com/yujia-yan/transkunA piano transcription system using Neural Semi-CRFs to convert expressive piano performance audio into MIDI files. It supports event-based transcription of intervals, velocity, and onset/offset positions via a Python library and CLI. The project includes modules for model training, Maestro dataset metadata generation, and evaluation tools such as computeMetrics.py and plotDeviation.py for analyzing transcription accuracy.
Before training, you must generate a configuration template file for the model using the moduleconf.generate module.
mkdir checkpoint
python3 -m moduleconf.generate Model:transkun.ModelTransformer > checkpoint/conf.jsonOnce the metadata files are generated and the configuration template is created, you can initiate training using the transkun.train module.
python3 -m transkun.train -hYou can install the transkun package using pip to access the command-line transcription tool.
pip3 install transkunOnce installed, you can use the transkun command to transcribe piano performance audio into a MIDI file. By default, it uses the CPU, but you can specify a CUDA device for faster processing.
Note: The default checkpoint is trained without pedal extension of notes and with data augmentation. This is intended to be closer to real performances, but be aware that many previous works use a convention that extends notes by sustain pedal durations.
# Basic transcription
transkun input.mp3 output.mid
# Transcription using CUDA
transkun input.mp3 output.mid --device cudaThe transkun project includes a specialized NeuralSemiCRF module optimized for event-based piano transcription. This module takes a score tensor (representing interval scores) and a noise score (representing non-interval scores) to decode event intervals using dynamic programming (Viterbi).
Input Dimensions:
score: [TEnd, TBegin, NBatch] (only the lower triangular part is used)noiseScore: [TBegin, TBegin+1] (representing the score for not being an interval)intervals: A list of lists containing non-overlapping tuples (TBegin, TEnd) for each batch.import CRF
import torch
T = 200
NBatch = 4
# representing the score for the interval [TBegin, TEnd]
# dimensions: [TEnd, TBegin, NBatch]
score = ((torch.randn(T, T, NBatch))).cuda()
# representing the score for being not an interval, dimensions [TBegin, TBegin+1]
noiseScore= ((torch.randn(T-1, NBatch))).cuda()
# a list of list of non-overlapping intervals
intervals = [
[(0,2), (4,6),(6,6), (7,8)],
[(1,2), (3,5), (19,19)],
[(0,0),(4,7)],
[],
]
crf = CRF.NeuralSemiCRFInterval(score, noiseScore)
## log probability
logP = crf.logProb(intervals)
## decoding
decoded = crf.decode()
## decoding starting from a given position, useful for segment based processing
decoded = crf.decode(forcedStartPos = [4]*NBatch)Arguments and options for the computeMetrics.py evaluation tool.
positional arguments:
estDIR
groundTruthDIR
options:
-h, --help show this help message and exit
--outputJSON OUTPUTJSON
path to save the output file for detailed metrics per audio file
--noPedalExtension Do not perform pedal extension according to the sustain pedal for the ground truth
--applyPedalExtensionOnEstimated
perform pedal extension for the estimated midi
--nProcess [NPROCESS]
number of workers for multiprocessing
--alignOnset whether or not to realign the onset.
--dither DITHER amount of noise added to the prediction.
--pedalOffset PEDALOFFSET
offset added to the groundTruth sustain pedal when extending notes
--onsetTolerance ONSETTOLERANCE
onset tolerance, default: 0.05 (50ms)Arguments and options for the plotDeviation.py visualization tool.
positional arguments:
evalJsons a seqeunce of the output json files from the computeMetrics script, the deviation output should be enabled
options:
-h, --help show this help message and exit
--labels [LABELS ...]
specify labels to show on the legend
--offset plot the offset deviation curve. If not specified, onset deviation curve will be plotted
--T T time limit(ms), default: 50ms
--output [OUTPUT] filename to save
--noDisplay Do not show the figure.
--cumulative plot the empirical cumulative density.
--absolute use absolute deviation.
--targetPitch TARGETPITCH
only plot specific number.Use the computeMetrics.py script (or the transkunEval command if the pip package is installed) to compare estimated MIDI files against ground truth MIDI files.
Requirements:
estDIR (estimated MIDI directory) must have the same folder structure as groundTruthDIR.Output: Metrics are outputted in the order: precision, recall, f1, and overlap.
python computeMetrics.py [--outputJSON OUTPUTJSON] [--noPedalExtension] [--applyPedalExtensionOnEstimated] [--nProcess [NPROCESS]] [--alignOnset] [--dither DITHER] [--pedalOffset PEDALOFFSET] [--onsetTolerance ONSETTOLERANCE] estDIR groundTruthDIRTo prepare the Maestro dataset for training, use transkun.createDatasetMaestro to combine groundtruth MIDI and metadata into .pt files (train.pt, val.pt, test.pt).
Note: All audio files must be converted to a 44100Hz sampling rate before running this command.
python3 -m transkun.createDatasetMaestro -h
usage: createDatasetMaestro.py [-h] [--noPedalExtension] datasetPath metadataCSVPath outputPath
positional arguments:
datasetPath folder path to the maestro dataset
metadataCSVPath path to the metadata file of the maestro dataset (csv)
outputPath path to the output folder
optional arguments:
-h, --help show this help message and exit
--noPedalExtension Do not perform pedal extension according to the sustain pedalUse plotDeviation.py to plot the Empirical Cumulative Distribution Function (ECDF) curve for onset or offset deviations. This script requires the JSON output files generated by the computeMetrics.py script (ensure deviation output was enabled during metric computation).
python plotDeviation.py [--labels [LABELS ...]] [--offset] [--T T] [--output [OUTPUT]] [--noDisplay] [--cumulative] [--absolute] [--targetPitch TARGETPITCH] evalJsons [evalJsons ...]The transkun.transcribe module (accessible via the transkun CLI) allows for fine-grained control over the transcription process, including weight selection, device management, and segment-based processing.
Arguments:
audioPath: Path to the input audio file.outPath: Path to the output MIDI file.Options:
--weight WEIGHT: Path to a pretrained weight file.--conf CONF: Path to the model configuration file.--device [DEVICE]: The device for computation (e.g., cpu, cuda). Defaults to cpu.--segmentHopSize SEGMENTHOPSIZE: The segment hopsize for processing (seconds). Defaults to the value in the model config.--segmentSize SEGMENTSIZE: The segment size for processing (seconds). Defaults to the value in the model config.python3 -m transkun.transcribe -h
usage: transcribe.py [-h] [--weight WEIGHT] [--conf CONF] [--device [DEVICE]] [--segmentHopSize SEGMENTHOPSIZE] [--segmentSize SEGMENTSIZE] audioPath outPath
positional arguments:
audioPath path to the input audio file
outPath path to the output MIDI file
options:
-h, --help show this help message and exit
--weight WEIGHT path to the pretrained weight
--conf CONF path to the model conf
--device [DEVICE] The device used to perform the most computations (optional), DEFAULT: cpu
--segmentHopSize SEGMENTHOPSIZE
The segment hopsize for processing the entire audio file (s), DEFAULT: the value defined in model conf
--segmentSize SEGMENTSIZE
The segment size for processing the entire audio file (s), DEFAULT: the value defined in model conf