RealRestorer Framework

The Text2VideoZeroPipeline enables zero-shot video generation by leveraging existing text-to-image synthesis methods (like Stable Diffusion) without requiring additional training or optimization. It achieves temporal consistency by enriching latent codes with motion dynamics and using cross-frame attention (where each frame attends to the first frame) to preserve context, appearance, and identity.

Supported generation modes:

1. Textual prompt: Standard text-to-video.
2. Prompt with guidance: Combining text with poses or edges.
3. Video Instruct-Pix2Pix: Instruction-guided video editing.

WARNING

This pipeline is deprecated. It is no longer tested or updated. If you encounter issues, consider reverting to a previous version of diffusers that supported this model.

Marigold is a diffusion-based collection of pipelines designed for dense computer vision tasks. It processes input RGB images to generate high-quality predictions for several modalities. The available pipelines are:

• MarigoldDepthPipeline: Predicts Depth and Disparity. Recommended checkpoint: prs-eth/marigold-depth-v1-1.
• MarigoldNormalsPipeline: Predicts Surface normals. Recommended checkpoint: prs-eth/marigold-normals-v1-1.
• MarigoldIntrinsicsPipeline: Predicts Intrinsic image decomposition (Albedo, Materials, Lighting). Recommended checkpoints: prs-eth/marigold-iid-appearance-v1-1 or prs-eth/marigold-iid-lighting-v1-1.

MusicLDM is a text-to-music latent diffusion model (LDM) that generates music samples from text prompts. It learns continuous audio representations from CLAP latents and was trained using beat-synchronous mixup strategies to ensure generated music is diverse yet faithful to the input style.

Note: This pipeline is currently deprecated. While it can still be used, it is no longer tested or updated. If you encounter issues, it is recommended to reinstall the last Diffusers version that supported this model.

Kandinsky 2.1 is a diffusion model that uses CLIP as a text and image encoder, combined with a diffusion image prior to map between latent spaces. This architecture enables high-quality text-to-image generation, image-to-image manipulation, and inpainting.

When using Kandinsky in diffusers, you can choose from several specialized pipeline classes depending on your task:

• Text-to-Image: Use KandinskyPipeline or KandinskyCombinedPipeline.
• Image-to-Image: Use KandinskyImg2ImgPipeline or KandinskyImg2ImgCombinedPipeline.
• Inpainting: Use KandinskyInpaintPipeline or KandinskyInpaintCombinedPipeline.
• Latent Space Manipulation: Use KandinskyPriorPipeline for interpolating between latents.

For official model checkpoints, refer to the Kandinsky Community organization on the Hugging Face Hub.

Diffusers supports various techniques to control the semantics, quality, and style of generated images. These techniques are categorized into two main types:

1. Inference-only methods: These work with existing pre-trained models without requiring additional training or fine-tuning. Examples include InstructPix2Pix, Attend and Excite, Semantic Guidance (SEGA), Self-attention Guidance (SAG), Depth2Image, MultiDiffusion Panorama, ControlNet, and Prompt Weighting.
2. Training/Fine-tuning methods: These require training the model on specific data to learn new subjects or concepts. Examples include DreamBooth, Textual Inversion, and Custom Diffusion.

Many techniques can be combined (e.g., using Textual Inversion with SEGA) to achieve more complex control.

Diffusers provides a collection of single-purpose, self-contained training scripts located in the diffusers/examples directory. These scripts are designed to be beginner-friendly and easy to modify, with data preprocessing and training loops fully exposed for customization.

Key supported training tasks include:

• Text-to-image: Supports SDXL and LoRA.
• DreamBooth: Supports SDXL and LoRA.
• ControlNet: Supports SDXL.
• InstructPix2Pix: Supports SDXL.
• T2I-Adapters: Supports SDXL.
• Textual Inversion
• Unconditional image generation
• Custom Diffusion
• Kandinsky 2.2 (LoRA support)
• Wuerstchen (LoRA support)

The diffusers library provides a variety of customized normalization layers designed to support different model architectures. These layers are primarily located in models.normalization (or models.transformers for specific implementations like Mochi).

Commonly used layers include:

• AdaLayerNorm variants: AdaLayerNorm, AdaLayerNormZero, AdaLayerNormSingle, AdaLayerNormContinuous, AdaLayerNormZeroSingle.
• RMSNorm variants: RMSNorm, LuminaRMSNormZero, MochiRMSNorm, MochiRMSNormZero.
• Other specialized layers: AdaGroupNorm, GlobalResponseNorm, LuminaLayerNormContinuous, SD35AdaLayerNormZeroX, LpNorm, CogView3PlusAdaLayerNormZeroTextImage, and CogVideoXLayerNormZero.

LEDITS++ (Limitless Image Editing using Text-to-Image Models) is an efficient and precise textual image manipulation technique that requires no fine-tuning or optimization. It supports multiple simultaneous edits and uses an implicit masking technique to limit changes to relevant image regions.

In the diffusers implementation, there are two primary pipelines available depending on your base model:

1. LEditsPPPipelineStableDiffusion for standard Stable Diffusion models.
2. LEditsPPPipelineStableDiffusionXL for Stable Diffusion XL (SDXL) models.

Note on Inversion Quality: Due to backward compatibility issues with ~schedulers.DPMSolverMultistepScheduler, this implementation cannot guarantee perfect inversion. While unlikely to affect most use cases, if perfect inversion is required, refer to the dedicated GitHub repo.

The StableDiffusionUpscalePipeline is used to enhance the resolution of input images by a factor of 4. It utilizes a diffusion model developed by CompVis, Stability AI, and LAION to perform super-resolution tasks.

# Note: This is a conceptual usage based on the pipeline name
# as the specific __call__ signature is referenced via autodoc
from diffusers import StableDiffusionUpscalePipeline
import torch

pipeline = StableDiffusionUpscalePipeline.from_pretrained("stabilityai/stable-diffusion-x4-upscaler", torch_dtype=torch.float16)
pipeline.to("cuda")

# The pipeline enhances input images by a factor of 4
image = pipeline(prompt="your prompt", image=input_image).images[0]