DOCSEARCHbyAlgolia
Submit Docs

ACE-Step 1.5

repository·main·Indexed 11 days ago

https://github.com/ace-step/ace-step-1.5

An efficient open-source music foundation model for high-quality music generation on consumer hardware. It utilizes a hybrid architecture combining a Language Model (LM) planner with a Diffusion Transformer (DiT) to generate audio from 10 seconds to 10 minutes. Supports multiple interaction methods including a Gradio Web UI, Python API, REST API, CLI, and VST3 plugin. Includes tools for LoRA fine-tuning and inference profiling.

Tokens
83.3K
Snippets
180
Records
355
Agent score
95%

What's inside ACE-Step

  1. Overview of ACE-Step 1.5 Core Concepts

    main

    The ACE-Step 1.5 tutorial provides a foundation for mastering the platform through several key pillars:

    • Mental Models: Adopting a human-centered generation design philosophy.
    • Model Architecture: Understanding the synergy between Language Models (LM) and Diffusion Transformers (DiT).
    • Input Control: Managing generation via text (Caption, Lyrics, metadata) and audio (reference audio, source audio).
    • Inference Hyperparameters: Tuning parameters that directly affect the generation process.
    • Random Factors: Leveraging randomness to explore creative spaces, specifically using the combination of Large Batch + AutoGen + Automatic Scoring to improve efficiency.

  2. Overview of ACE-Step 1.5

    main

    ACE-Step 1.5 is an open-source music generation system designed for commercial-grade quality on consumer hardware. It is optimized for high performance and low resource usage, capable of generating audio ranging from 10 seconds to 10 minutes.

    Key Capabilities

    • Ultra-Fast Generation: Generates full songs in under 2s on an A100 and under 10s on an RTX 3090.
    • Commercial-Grade Quality: Supports 1000+ instruments and styles across 50+ languages, with fine-grained control over timbre and metadata.
    • Versatile Editing: Includes features for cover generation, repainting, track separation, vocal-to-BGM, and multi-track generation.
    • Lightweight & Customizable: Runs locally with less than 4GB of VRAM and supports LoRA training to capture specific musical styles from a few sample songs.

  3. Overview of Side-Step documentation

    main

    Side-Step is a tool integrated into ACE-Step-1.5 that handles music generation workflows, including training, preprocessing, and model management. The documentation covers several key areas:

    • Setup & Tutorials: Installation, first-run setup, and end-to-end walkthroughs from raw audio to music.
    • Data & Training: Dataset preparation (JSON schemas, audio formats), training methodologies (LoRA vs LoKR, corrected vs vanilla), and CLI usage.
    • Model & Preset Management: Managing checkpoints, fine-tunes, built-in presets, and the 'never rename' rule for model files.
    • Optimization & Advanced Concepts: VRAM optimization, gradient sensitivity analysis (Estimation Guide), and technical details on Shift and Timestep Sampling.
    • Platform Specifics: Windows-specific workarounds for num_workers and pathing issues.

    Note: This documentation is for the version of Side-Step integrated into ACE-Step-1.5. For the standalone version with more frequent updates, visit the original Side-Step repository.

  4. Use Side-Step for advanced LoRA/LoKr training

    main

    For advanced users requiring more control, the community-developed Side-Step toolkit is available. It provides features such as:

    • Corrected timestep sampling
    • CLI-based workflows
    • VRAM optimization strategies
    • Gradient sensitivity analysis
    • LoKr adapter support

    Detailed documentation for Side-Step is located in the docs/sidestep/ directory of this repository.

  5. Available ACE-Step 1.5 Usage Methods

    main

    ACE-Step 1.5 provides several ways to interact with the music generation engine depending on your needs:

    • Gradio Web UI: An interactive web interface for manual music generation.
    • Python API: Programmatic access for integrating the engine into your own Python applications.
    • REST API: An HTTP-based asynchronous API suitable for service-oriented architectures.
    • CLI: An interactive wizard and configuration tool for command-line usage.
    • VST3 Plugin: A standalone plugin (C++/GGML) for integration directly into Digital Audio Workstations (DAWs).
    • UI Support Baseline: A checklist for UI developers to ensure parity with supported boundaries.

  6. Navigate the ACE-Step 1.5 Web Interface

    main

    The ACE-Step 1.5 web interface is organized into two primary tabs:

    • Generation Tab: The main workspace used for creating music. It contains two modes:
      • Simple Mode: For quick results using natural language descriptions. Click "Create Sample" to have the AI fill in details, then "Generate Music".
      • Custom Mode: For precise control over captions, lyrics (with structure tags), tempo, key, duration, and advanced settings.
    • LoRA Training Tab: Used for teaching the AI your personal musical style. It includes a Dataset Builder sub-tab for preparing training data.

  7. What is LoKr and how does it differ from LoRA?

    main

    LoKr (Low-rank Kronecker product) is an alternative adapter method that uses Kronecker decomposition instead of standard low-rank matrix factorization used in LoRA.

    Key Advantages:

    • Speed: Significantly faster training times (e.g., reducing an hour of training to ~5 minutes).
    • Efficiency: Well-suited for consumer-grade GPUs.
    • Features: Supports DoRA (Weight-Decomposed Low-Rank Adaptation) by default via the lokr_weight_decompose parameter.

    Comparison Summary:

    AspectLoRALoKr
    DecompositionLow-rank matrix factorizationKronecker product decomposition
    Training speedBaselineUp to 10x faster
    Default learning rate1e-40.03
    Default epochs10500
    DoRA supportNoYes (via weight_decompose)

  8. What is Differential Correction in Wavelet Domain (DCW)?

    main

    Differential Correction in Wavelet Domain (DCW) is a training-free, sampler-side correction technique designed to mitigate SNR-t bias in diffusion models (including ACE-Step's flow-matching formulation).

    During inference, the actual Signal-to-Noise Ratio (SNR) of a sample can drift away from what the model was trained on at a specific timestep. Because diffusion models reconstruct low-frequency content before high-frequency content, DCW applies corrections to different frequency bands (via Discrete Wavelet Transform) using specific schedules to improve generation quality without adding significant compute.

    Key characteristics:

    • Training-free: No fine-tuning required.
    • Low overhead: Adds negligible compute.
    • Frequency-aware: Applies corrections in the wavelet domain rather than directly on the latent.
    • Compatible: Works across all ACE-Step sampler paths (PyTorch and MLX).

  9. What is Side-Step?

    main

    Side-Step is a standalone training toolkit designed for ACE-Step 1.5 models. It provides corrected LoRA and LoKR fine-tuning implementations that address fundamental bugs in the original ACE-Step trainer.

    Key improvements include:

    • Logit-Normal continuous sampling: Replaces the discrete 8-step schedule to ensure the model learns the full denoising range.
    • 15% CFG Dropout: Implements Classifier-Free Guidance dropout to improve inference quality.
    • Low-VRAM Support: Optimized for local GPUs with features like gradient checkpointing enabled by default.
    • Standalone Operation: Most features (corrected training loop, preprocessing, and wizard) work without a base ACE-Step installation; you only need the model checkpoints.

  10. Overview of the ACE-Step 1.5 Profiling & Benchmarking Tool

    main

    profile_inference.py is a tool used to measure the performance of ACE-Step 1.5 inference. It provides detailed timing breakdowns for various stages of the generation process, including end-to-end wall time, LLM planning time, DiT diffusion time, and VAE decoding time. It supports different devices (CUDA, MPS, CPU) and LLM backends (vLLM, PyTorch, MLX).

    python profile_inference.py

  11. Select the appropriate model based on GPU VRAM

    main

    ACE-Step 1.5 offers different model combinations (DiT and LM) depending on your available VRAM. The Gradio UI automatically selects the best configuration, but you can use this guide to understand the tiers:

    Your GPU VRAMRecommended DiTRecommended LM ModelBackendNotes
    ≤6GB2B turboNone (DiT only)—LM disabled; INT8 quantization + CPU offload
    6-8GB2B turboacestep-5Hz-lm-0.6BptLightweight LM with PyTorch
    8-16GB2B turbo/sftacestep-5Hz-lm-0.6B / 1.7Bvllm0.6B for 8-12GB, 1.7B for 12-16GB
    16-20GB2B sft or XL turboacestep-5Hz-lm-1.7BvllmXL requires CPU offload below 20GB
    20-24GBXL turbo/sftacestep-5Hz-lm-1.7BvllmXL fits without offload
    ≥24GBXL sft (or xl-base)acestep-5Hz-lm-4BvllmBest quality; all models fit

    XL (4B) Models (acestep-v15-xl-*): These provide higher audio quality. They require $\ge$12GB VRAM with offload/quantization, or $\ge$20GB without offload. All LM models are compatible with XL.

  12. How ACE-Step discovers models

    main

    When starting training or preprocessing, Side-Step automatically scans the checkpoints/ directory to classify models:

    1. Official models: Folders starting with acestep-v15- (e.g., acestep-v15-turbo). These are auto-detected with their correct timestep parameters.
    2. Custom models / fine-tunes: Any other folder containing a config.json.

    If a custom model is detected, Side-Step will prompt you to select which official base model it descends from to ensure correct timestep conditioning. The discovery process provides a numbered list for selection and supports fuzzy searching by typing names.