repository·main·Indexed 25 days ago
https://github.com/haoheliu/audioldmA latent diffusion model for high-quality audio generation. AudioLDM supports text-to-audio, audio-to-audio, and text-guided style transfer. It provides a command-line interface for generation and style transfer, as well as programmatic access via the Hugging Face diffusers library (v0.15.0+). The tool includes multiple pre-trained checkpoints such as audioldm-s-full, audioldm-m-full, and audioldm-l-full.
To use AudioLDM via the command line, set up a Python 3.8 environment and install the package directly from the GitHub repository.
conda create -n audioldm python=3.8; conda activate audioldm
pip3 install git+https://github.com/haoheliu/AudioLDM.gitGenerate audio from a text prompt using the default generation mode. Results are saved to ./output/generation by default.
audioldm -t "A hammer is hitting a wooden surface" Generate new audio that contains similar audio events to an input file. Use the --file_path flag to specify the source audio.
audioldm --file_path trumpet.wavTransfer the style of an existing audio file into a new style described by text.
--mode "transfer": Sets the mode to style transfer.--file_path: The original audio file to be transformed.-t: The text description of the target style.--transfer_strength: A value between 0 and 1. 0 keeps the original audio unchanged; 1 completely transfers to the text-described style. Default is 0.5.audioldm --mode "transfer" --file_path trumpet.wav -t "Children Singing" --transfer_strength 0.25AudioLDM is integrated into the Hugging Face diffusers library (v0.15.0+). This is the preferred method for programmatic access in Python.
pip install --upgrade diffusers transformersAudioLDMPipeline to generate audio from a prompt.from diffusers import AudioLDMPipeline
import torch
repo_id = "cvssp/audioldm-s-full-v2"
pipe = AudioLDMPipeline.from_pretrained(repo_id, torch_dtype=torch.float16)
pipe = pipe.to("cuda")
prompt = "Techno music with a strong, upbeat tempo and high melodic riffs"
audio = pipe(prompt, num_inference_steps=10, audio_length_in_s=5.0).audios[0]The tool uses a cache directory for model weights. You can override the default location by setting the AUDIOLDM_CACHE_DIR environment variable.
Default location: ~/.cache/audioldm
The audioldm command-line tool supports the following arguments:
| Flag | Long Flag | Description |
|---|---|---|
-h | --help | Show this help message and exit |
| N/A | --mode {generation,transfer} | generation: text-to-audio; transfer: style transfer |
-t | --text TEXT | Text prompt for audio generation (Default: "") |
-f | --file_path FILE_PATH | (transfer mode): Original audio for transfer; (generation mode): guidance audio (Default: None) |
| N/A | --transfer_strength STRENGTH | Value 0 to 1 (0: original, 1: complete transfer) (Default: 0.5) |
-s | --save_path SAVE_PATH | Path to save output (Default: "./output") |
| N/A | --model_name NAME | Checkpoint to use (Default: "audioldm-s-full") |
| N/A | --batchsize BATCHSIZE | Number of samples to generate at once (Default: 1) |
| N/A | --ddim_steps STEPS | Sampling steps for DDIM (Default: 200) |
-gs | --guidance_scale SCALE | Large = better quality/relevancy; Small = better diversity (Default: 2.5) |
-dur | --duration DURATION | Duration of samples (Default: 10) |
-n | --n_candidate_gen_per_text N | Number of candidates for automatic quality control (Default: 3) |
--seed | --seed SEED | Integer seed for reproducibility (Default: 42) |
usage: audioldm [-h] [--mode {generation,transfer}] [-t TEXT] [-f FILE_PATH] [--transfer_strength TRANSFER_STRENGTH] [-s SAVE_PATH] [--model_name {audioldm-s-full,audioldm-l-full,audioldm-s-full-v2}] [-ckpt CKPT_PATH]
[-b BATCHSIZE] [--ddim_steps DDIM_STEPS] [-gs GUIDANCE_SCALE] [-dur DURATION] [-n N_CANDIDATE_GEN_PER_TEXT] [--seed SEED]
optional arguments:
-h, --help show this help message and exit
--mode {generation,transfer}
generation: text-to-audio generation; transfer: style transfer
-t TEXT, --text TEXT Text prompt to the model for audio generation, DEFAULT ""
-f FILE_PATH, --file_path FILE_PATH
(--mode transfer): Original audio file for style transfer; Or (--mode generation): the guidance audio file for generating simialr audio, DEFAULT None
--transfer_strength TRANSFER_STRENGTH
A value between 0 and 1. 0 means original audio without transfer, 1 means completely transfer to the audio indicated by text, DEFAULT 0.5
-s SAVE_PATH, --save_path SAVE_PATH
The path to save model output, DEFAULT "./output"
--model_name {audioldm-s-full,audioldm-l-full,audioldm-s-full-v2}
The checkpoint you gonna use, DEFAULT "audioldm-s-full"
-ckpt CKPT_PATH, --ckpt_path CKPT_PATH
(deprecated) The path to the pretrained .ckpt model, DEFAULT None
-b BATCHSIZE, --batchsize BATCHSIZE
Generate how many samples at the same time, DEFAULT 1
--ddim_steps DDIM_STEPS
The sampling step for DDIM, DEFAULT 200
-gs GUIDANCE_SCALE, --guidance_scale GUIDANCE_SCALE
Guidance scale (Large => better quality and relavancy to text; Small => better diversity), DEFAULT 2.5
-dur DURATION, --duration DURATION
The duration of the samples, DEFAULT 10
-n N_CANDIDATE_GEN_PER_TEXT, --n_candidate_gen_per_text N_CANDIDATE_GEN_PER_TEXT
Automatic quality control. This number control the number of candidates (e.g., generate three audios and choose the best to show you). A Larger value usually lead to better quality with heavier computation, DEFAULT 3
--seed SEED
Change this value (any integer number) will lead to a different generation result. DEFAULT 42Use the --model_name parameter to choose between different pre-trained checkpoints.
Available options:
audioldm-m-full: Medium model, trained with audio embeddings (Recommended).audioldm-s-full: Original open-sourced version (Recommended).audioldm-s-full-v2: Original version with more training steps (Recommended).audioldm-s-text-ft: Small model finetuned with AudioCaps and MusicCaps.audioldm-m-text-ft: Medium model finetuned with AudioCaps and MusicCaps.audioldm-l-full: Larger model compared to audioldm-s-full.Example usage:
audioldm --model_name audioldm-s-fullbuild_model function to initialize an AudioLDM model instance. You must provide a model_name string to specify which model architecture and weights to load.text_to_audio function performs the core text-to-audio generation task. It requires a loaded latent diffusion model and several parameters to control the output quality and characteristics.Use the following flags to control the quality, duration, and behavior of the audio generation:
| Flag | Short | Default | Description |
|---|---|---|---|
--mode | N/A | generation | generation for text-to-audio; transfer for style transfer |
--text | -t | "" | Text prompt for audio generation |
--file_path | -f | None | Original audio for transfer OR guidance audio for generation |
--transfer_strength | N/A | 0.5 | Value [0, 1]. 0 is original audio; 1 is complete transfer to text style |
--save_path | -s | ./output | Directory to save model output |
--model_name | N/A | audioldm-m-full | Checkpoint to use. Choices: audioldm-s-full, audioldm-l-full, audioldm-s-full-v2, audioldm-m-text-ft, audioldm-s-text-ft, audioldm-m-full |
--ckpt_path | -ckpt | None | Path to pretrained .ckpt model (Note: has no effect after v0.0.20) |
--batchsize | -b | 1 | Number of samples to generate simultaneously |
--ddim_steps | N/A | 200 | Sampling steps for DDIM |
--guidance_scale | -gs | 2.5 | Higher = better quality/relevancy; Lower = better diversity |
--duration | -dur | 10.0 | Duration of samples. Must be a multiple of 2.5 |
--n_candidate_gen_per_text | -n | 3 | Number of candidates to generate for automatic quality control |
--seed | N/A | 42 | Integer seed for reproducible results |
When calling text_to_audio, use the following parameters to control generation:
latent_diffusion: The model instance returned by build_model.text: The descriptive text prompt for the audio.seed: Integer used for reproducibility.duration: The length of the generated audio in seconds.guidance_scale: Controls the trade-off between quality/relevancy (higher) and diversity (lower).n_candidate_gen_per_text: The number of candidate audio clips to generate (useful for quality control).