AudioCraft

AudioCraft provides inference and training code for several generative audio models:

• MusicGen: Controllable text-to-music model.
• AudioGen: Text-to-sound model.
• EnCodec: High-fidelity neural audio codec.
• Multi Band Diffusion: An EnCodec compatible decoder using diffusion.
• MAGNeT: Non-autoregressive model for text-to-music and text-to-sound.
• AudioSeal: Audio watermarking.
• MusicGen Style: Text-and-style-to-music model.
• JASCO: High-quality text-to-music model conditioned on chords, melodies, and drum tracks.

AudioSeal is a method for speech localized watermarking designed for proactive detection of voice cloning. It features a joint training approach:

1. A Generator that embeds a watermark into audio.
2. A Detector that identifies watermarked fragments within longer audio sequences, maintaining robustness even when the audio has been edited.

AudioGen is a textually guided audio generation model developed by Meta AI's FAIR team. It consists of two main components:

1. EnCodec model: Used for audio tokenization.
2. Auto-regressive language model: A transformer-based architecture used for audio modeling.

Key Specifications (v2):

• Parameters: 1.5B
• Generation Length: Trained on 10-second clips (compared to 5 seconds in v1).
• Tokenization: Uses a discrete representation extracted via a retrained EnCodec model on environmental sound data.
• License: The code is released under MIT, but the model weights are released under CC-BY-NC 4.0 (Non-Commercial).

When configuring training, note that FSDP (Fully Sharded Data Parallel) and autocast are mutually exclusive because FSDP handles autocast internally.

• Use autocast: If you are working with models up to 1.5B (medium) and have sufficient GPU RAM.
• Use FSDP: If you need to free up memory for activations by sharding the optimizer state, though it increases configuration complexity.

For all audio generation metrics (FAD, KLD, Text Consistency, etc.), you can choose to compute the metric on the audio reconstructed by EnCodec rather than the raw generated sample. This is controlled by the flag <metric>.use_gt=true.

# Example for FAD
dora run <...> evaluate.metrics.fad=true metrics.fad.use_gt=true

To create a new conditioning module, you should implement a class that follows the BaseConditioner pattern. All conditioners must implement two specific methods to ensure efficient execution and avoid synchronization points:

• tokenize(...): Performs preprocessing (like BPE tokenization or moving data to GPU). The output of this method is passed to the forward method.
• forward(...): Performs the core computation to produce the conditioning embedding and a mask indicating valid indices (e.g., for padding).

EnCodec training is implemented via the CompressionSolver class. It trains an encoder-decoder with a quantization bottleneck (specifically a SEANet encoder-decoder with Residual Vector Quantization) using a combination of objective and perceptual losses (discriminators).

Configuration files for compression are located in config/solver/compression, and example training grids can be found in audiocraft/grids/compression.

# Example: Run training for base causal encodec on monophonic 24khz audio
dora grid compression.encodec_base_24khz

# Example: Run training for encodec model used for MusicGen on monophonic 32khz audio
dora grid compression.encodec_musicgen_32khz

Both MusicDataset and SoundDataset extend the base AudioDataset by requiring additional metadata.

MusicDataset

Expects additional metadata to be stored in a .json file located at the same path as the corresponding audio file (e.g., audio_file.wav requires audio_file.json).

SoundDataset

Expects metadata in a .json file at the same path as the audio file. Additionally, SoundDataset supports an external_metadata_source parameter, which allows you to point to a separate folder containing all the JSON metadata files (provided they share the same filename as the audio files).

When using JASCO models that support chord control, you can pass a list of tuples to the chords parameter in model.generate_music(). Each tuple should contain a chord name (as a string) and a start time (as a float).

Example format: [('C', 0.0), ('D', 2.0)]

# Example chord progression: C at 0s, D at 2s, F at 4s, etc.
chords = [('C', 0.0), ('D', 2.0), ('F', 4.0), ('Ab', 6.0), ('Bb', 7.0), ('C', 8.0)]