DOCSEARCHbyAlgolia
Submit Docs

DiffSynth-Studio

repository·main·Indexed 11 days ago

https://github.com/modelscope/diffsynth-studio

An open-source Diffusion model engine ecosystem for universal inference and training. It supports academic research and industrial deployment via DiffSynth-Engine, featuring advanced VRAM management, CPU offload training, and a plugin framework called Diffusion Templates for controllable generation. The framework provides pipelines for Stable Diffusion, SDXL, FLUX.2, and Anima, and supports training for base models, LoRAs, and ControlNets using techniques like FP8 precision and Split Training.

Tokens
138.6K
Snippets
214
Records
363
Agent score
96%

What's inside DiffSynth-Studio

  1. Overview of Diffusion Templates

    main
    Diffusion Templates is a controllable generation plugin framework for Diffusion models within DiffSynth-Studio. It provides additional controllable generation capabilities for base models, allowing users to perform specific tasks like structure control, brightness adjustment, and image editing by using specialized template models. The framework supports both standard inference and low-VRAM inference modes, as well as full training and validation workflows.

  2. Overview of DiffSynth-Studio and DiffSynth-Engine

    main

    DiffSynth-Studio is an open-source Diffusion model engine developed by the ModelScope team. The ecosystem is split into two distinct projects based on user needs:

    1. DiffSynth-Studio: Focused on aggressive technical exploration and academic research. It provides support for cutting-edge model capabilities and experimental features.
    2. DiffSynth-Engine: Focused on stable model deployment for industrial applications. It prioritizes high computational performance and stable functionality.

    Both projects serve as core engines for the ModelScope AIGC zones (ModelScope AIGC for Chinese users and ModelScope Civision for global users).

  3. Overview of ACE-Step Model Variants and Tasks

    main

    The ACE-Step model family supports various tasks including text-to-music, audio cover, and repainting. Different model IDs are optimized for specific workflows such as full training, LoRA training, or low-VRAM inference.

    Key model variants include:

    • Base/SFT models: Standard text-to-music and fine-tuned versions.
    • Turbo models: Optimized for faster inference (e.g., acestep-v15-turbo-shift1, acestep-v15-turbo-shift3).
    • XL models: Larger parameter versions for higher quality.
    • Task-specific variants: Models like acestep-v15-base: CoverTask or acestep-v15-base: RepaintTask are specialized for specific audio manipulation tasks.

    Refer to the model table in the documentation for specific code examples for inference, low-VRAM inference, and various training modes (Full vs LoRA).

  4. Overview of LTX-2 Model Capabilities and Inference Modes

    main

    The LTX-2 model family in DiffSynth-Studio supports various tasks including Text-to-Video (T2AV), Image-to-Video (I2AV), Audio-to-Video (A2V), and Retake capabilities.

    Inference Modes

    • Standard Inference: High-performance inference using standard pipelines.
    • Low VRAM Inference: Optimized inference paths for hardware with limited video memory.

    Task Types and Required Parameters

    • I2AV (Image-to-Video): Requires input_images.
    • A2V (Audio-to-Video): Requires retake_audio, audio_sample_rate, and retake_audio_regions.
    • Retake: Requires retake_video, retake_video_regions, retake_audio, audio_sample_rate, and retake_audio_regions.
    • In-Context (IC) LoRA Control: Requires in_context_videos and in_context_downsample_factor.

    Training Support

    Certain LTX-2 models support Full Training, LoRA Training, and specific validation scripts for both training modes.

  5. Overview of FLUX models in DiffSynth-Studio

    main

    DiffSynth-Studio supports a wide range of models within the FLUX.1 series, including base models, ControlNet variants, IP-Adapters, and specialized editing models. The platform provides dedicated support for standard inference, low-VRAM inference, full model training, and LoRA training for these models.

    Key model categories include:

    • Base Models: black-forest-labs/FLUX.1-dev, black-forest-labs/FLUX.1-Krea-dev, and black-forest-labs/FLUX.1-Kontext-dev.
    • ControlNet Models: Variants for Inpainting, Union, and Upscaling.
    • Adapter Models: Such as InstantX/FLUX.1-dev-IP-Adapter.
    • Specialized Editing/Generation: Models like DiffSynth-Studio/Eligen, stepfun-ai/Step1X-Edit, and DiffSynth-Studio/Nexus-GenV2.

    Each model may require specific extra parameters during inference (e.g., controlnet_inputs, ipadapter_images, or nexus_gen_reference_image).

  6. Overview of Qwen-Image model variants and capabilities

    main

    The Qwen-Image series includes various specialized models for image editing, distillation, control, and layered generation. DiffSynth-Studio provides support for inference (including low VRAM modes), full parameter training, LoRA training, and validation for most models in this lineage.

    Key model categories include:

    • Base & Edit Models: Qwen/Qwen-Image, Qwen/Qwen-Image-Edit, and their updated versions (e.g., -2512, -2509).
    • EliGen Series: Specialized for high-quality generation (e.g., Qwen-Image-EliGen, Qwen-Image-EliGen-Poster).
    • Distill Series: Optimized models like Qwen-Image-Distill-Full and Qwen-Image-Distill-LoRA.
    • ControlNet Series: Models for structural control such as Blockwise-ControlNet-Canny, Depth, and Inpaint, as well as In-Context-Control-Union.
    • Layered Models: Qwen-Image-Layered and Qwen-Image-Layered-Control for handling layered image data.

  8. FLUX.2 Model Overview and Capabilities

    main

    The FLUX.2 model family in DiffSynth-Studio supports various tasks including standard inference, low VRAM inference, full model training, LoRA training, and specialized template-based workflows.

    Available model variants include:

    • Black Forest Labs models: FLUX.2-dev, FLUX.2-klein-4B, FLUX.2-klein-9B, FLUX.2-klein-base-4B, and FLUX.2-klein-base-9B.
    • DiffSynth-Studio Templates: Specialized models for tasks like Aesthetic, Brightness, Age, ControlNet, Edit, Inpaint, PandaMeme, Sharpness, SoftRGB, Upscaler, ContentRef, and KleinBase4B-i2L-v2.

    Each model variant has specific implementation examples for inference (standard and low VRAM), training (full or LoRA), and validation.

  9. Capabilities of Diffusion Templates

    main

    Diffusion Templates in DiffSynth-Studio allow for complex image processing pipelines by combining multiple diffusion-based tasks. Key capabilities include:

    • Super-Resolution & Sharpness Enhancement: Upscaling low-resolution inputs to ultra-high-resolution outputs.
    • Full ControlNet Pipelines: Combining structure control (e.g., Depth) with aesthetic alignment and sharpness enhancement.
    • Artistic Style Creation: Combining structure control with image editing and color adjustment to transfer styles from a reference image to a structured input.
    • Local Redrawing & Brightness Control: Using masks for local inpainting/redrawing combined with brightness control and image editing to integrate cross-dimensional elements.

  10. What is Split Training and how does it work?

    main

    Split training is an experimental feature designed to reduce VRAM usage and accelerate training speed by dividing the training process into two distinct stages.

    In most text-to-image models, significant computation occurs during 'preprocessing' (e.g., VAE encoding, text encoding) which is unrelated to the denoising model itself. When these model parameters are fixed, these computations are repetitive across epochs. Split training automatically analyzes the computational graph and splits these Pipeline Units into a first stage that computes and stores results to disk, and a second stage that reads those results to perform the actual training. This avoids re-computing the same encodings every epoch.

  11. What is Direct Distillation in DiffSynth-Studio

    main

    Direct Distillation is an end-to-end distillation accelerated training scheme designed for generality, stability, and simplicity. Unlike complex methods like Adversarial Diffusion Distillation (ADD) or Hyper-SD, Direct Distillation aims to align the generation effects of a small number of steps with those of a large number of steps using a simple MSE loss between a high-step inference result and a low-step inference result.

    The core logic follows this pattern:

    1. Generate a high-quality reference image (image_1) using a large number of steps (e.g., 50 steps) with torch.no_grad().
    2. Generate a target image (image_2) using a small number of steps (e.g., 4 steps).
    3. Calculate the loss as the Mean Squared Error (MSE) between image_1 and image_2.
    seed = xxx
with torch.no_grad():
    image_1 = pipe(prompt, steps=50, seed=seed, cfg=4)
image_2 = pipe(prompt, steps=4, seed=seed, cfg=1)
loss = torch.nn.functional.mse_loss(image_1, image_2)

  12. Use Diffusion Templates for controllable generation

    main
    Diffusion Templates provide a plugin framework for controllable generation. This architecture allows you to use Template models for both inference and training to achieve specific generation constraints. The framework includes detailed architecture documentation and specific guides for template model inference and training.