acestep.cpp

The project manages GPU memory through a ModelStore using two distinct modes controlled by a single flag. This determines how modules (LM, DiT, VAE, etc.) are loaded and evicted from VRAM.

VRAM Modes

• Default Mode (Optimizes VRAM): Uses a STRICT eviction policy. At most one GPU module is resident at a time. When a new module is requested, the previous one is evicted. This allows the full stack to run on consumer-grade hardware by swapping weights (e.g., DiT loads, then evicts; VAE loads, then evicts).
• --keep-loaded Mode (Optimizes Latency): Uses a NEVER eviction policy. Everything stays resident in VRAM across requests. This is intended for workstations with high VRAM capacity to eliminate reload overhead.

ModelStore Mechanics

• Module Keys: Modules are keyed to prevent redundant loading.
  • LM: Keyed by (path, max_seq, n_kv_sets). This ensures ace-lm and ace-understand share the same single LM instance.
  • DiT: Keyed by (path, adapter_path, adapter_scale).
  • Others: Keyed by (path).
• RAII Management: The system uses ModelHandle (an RAII handle) to ensure that when a pipeline finishes using a module, it is correctly released back to the store, triggering the appropriate eviction policy.

The generation process depends on the content of your input JSON. The LLM fills missing fields and generates audio codes, while existing fields are treated as fixed. All modes output numbered files (request0.json .. requestN-1.json) and never modify the original input JSON.

Core Generation Modes

• Caption only (lyrics=""): Performs two LLM passes. Phase 1 expands the caption into enriched metadata (bpm, keyscale, etc.) via Chain of Thought (CoT). Phase 2 generates audio codes. CFG is forced to 1.0 in Phase 1.
• Caption + lyrics (+ optional metadata): A single LLM pass. Missing metadata is filled via CoT, and the caption is enriched. User-provided metadata is preserved.
• Everything provided: If caption, lyrics, bpm, duration, keyscale, and timesignature are all present, the LLM skips CoT and generates audio codes directly.
• Instrumental (lyrics="[Instrumental]"): Uses the single-pass path. The DiT is specifically trained to handle this string as a no-vocal condition.
• Passthrough (audio_codes present): The LLM is skipped entirely. Use ace-synth to decode existing codes.

Audio Transformation Modes

• Cover ("task_type": "cover" + --src-audio): Creates a free reinterpretation. It uses an FSQ roundtrip (5:1 temporal compression) which destroys micro-timings and transients, causing the DiT to diverge from the source. Control divergence with audio_cover_strength.
• Cover-nofsq ("task_type": "cover-nofsq" + --src-audio): Creates a faithful remix. It skips the FSQ roundtrip, providing clean VAE latents to the DiT. For best results, pass --ref-audio pointing to the same file as --src-audio.
• Repaint ("task_type": "repaint" + --src-audio): Regenerates a specific time region. Use repainting_start and repainting_end (in seconds).
  • Inpainting: repainting_start and repainting_end are within the source duration.
  • Outpainting: Use negative values for repainting_start to generate audio before the source, or an end value beyond the source duration to generate after.
• Lego ("task_type": "lego" + --src-audio): Layers a new instrument track over an existing backing track. Requires acestep-v15-base DiT (Turbo and SFT are not supported). Requires a track field in the JSON.

Track Names for Lego, Extract, and Complete

vocals, backing_vocals, drums, bass, guitar, keyboard, percussion, strings, synth, fx, brass, woodwinds.

The ACE-Step server supports three different trainer formats for adapters. The server automatically detects the format by inspecting the safetensors payload.

Supported formats include:

• PEFT: Provided as a folder. It is identified by the presence of lora_alpha in the adapter_config.json file.
• ComfyUI: Provided as a single .safetensors file. It is identified by a per-tensor .alpha scalar.
• LyCORIS: Provided as a single .safetensors file. It is identified by a per-module .alpha scalar. LyCORIS LoKr supports both factorized weights (lokr_w2_a + lokr_w2_b) and monolithic weights (lokr_w2), with optional DoRA support via the dora_scale parameter.

| Trainer | Layout                | Alpha source                          | Example                                |
|---------|-----------------------|---------------------------------------|----------------------------------------|
| PEFT    | folder                | `lora_alpha` in `adapter_config.json` | `ACE-Step-v1.5-chinese-new-year-LoRA/` |
| ComfyUI | single `.safetensors` | per tensor `.alpha` scalar            | `turbo_v9_1850_comfyui.safetensors`    |
| LyCORIS | single `.safetensors` | per module `.alpha` scalar            | `acestep-qinglong-lokr.safetensors`    |

Control how the model interprets your prompt and handles vocals using caption and lyrics:

• caption (string, required): A natural language description of style, mood, and instruments. This is fed to both the LLM and the DiT text encoder.
• lyrics (string): The single source of truth for vocal content. Use one of these three states:
  • "" (empty): The LLM generates lyrics based on the caption.
  • "[Instrumental]": No vocals. The LLM skips lyrics generation and the DiT receives this instruction.
  • "Any other string": Use your own lyrics. The LLM will only fill in missing metadata.

Metadata (Auto-filled by LLM if unset):

• bpm: Beats per minute (defaults to 0 for auto-generation).
• duration: Target length in seconds. 0 lets the LLM decide (constrained to [10, 600]s).
• keyscale: Musical key/scale (e.g., "C major").
• timesignature: Numerator as a string (e.g., "4" for 4/4).
• vocal_language: BCP-47 code (e.g., "en"). If "", the LLM detects it via CoT. If "unknown", it signals the DiT to expect no specific language.

The ace-server provides an HTTP interface for ace-lm, ace-synth, ace-understand, and ace-vae pipelines.

Workflow:

1. Submit Job: Send a POST request to an endpoint (e.g., /lm, /synth). The server returns a job id immediately.
2. Poll Status: Use GET /job?id=N to check if the status is running, done, failed, or cancelled.
3. Fetch Result: Once status is done, use GET /job?id=N&result=1 to retrieve the data.
4. Cancel: Use POST /job?id=N&cancel=1 to stop a running job.

Concurrency Model: All requests are pushed to a single FIFO queue processed by one worker thread. This ensures serial execution without GPU mutex contention. Completed jobs are cached in memory (up to 32 entries) before being evicted FIFO.

# Example: Start server with custom port and batch limit
./ace-server --models /path/to/models --host 0.0.0.0 --port 8085 --max-batch 2

Music generation is a two-step process involving ace-lm (which generates lyrics and audio codes) and ace-synth (which synthesizes the audio).

Step 1: Generate Metadata and Codes with ace-lm

Input a JSON request file. The output is a new JSON file (e.g., request0.json) enriched with metadata, lyrics, and codes.

./ace-lm --models models --request /tmp/request.json

Step 2: Synthesize Audio with ace-synth

Input the enriched JSON file from the previous step to produce the audio file (e.g., request00.mp3).

./ace-synth --models models --request /tmp/request0.json

Advanced Usage

• Using Adapters (LoRA/PEFT): Set the adapter and adapter_scale in your input JSON, then point the CLI to the directory containing the adapter using --adapters.
• Batch Generation: Set lm_batch_size in the input JSON. ace-lm will generate multiple request files (e.g., request0.json, request1.json), and ace-synth can process them all in a single GPU batch by passing them as multiple arguments.
• Audio Covers: To transform an existing song, use the --src-audio flag with ace-synth and set task_type to "cover" in your JSON.

# 1. Generate metadata
./ace-lm --models models --request /tmp/request.json

# 2. Synthesize audio
./ace-synth --models models --request /tmp/request0.json