DOCSEARCHbyAlgolia
Submit Docs

Diffusers

repository·main·Indexed 12 days ago

https://github.com/huggingface/diffusers

A modular library for state-of-the-art pretrained diffusion models used to generate images, audio, and 3D molecular structures. It provides high-level Diffusion Pipelines for inference, as well as modular models and schedulers for custom training and implementation. Supported tasks include text-to-image, image-to-image, inpainting, super resolution, and unconditional image generation.

Tokens
396.4K
Snippets
847
Records
1.2K
Agent score
98%

What's inside Diffusers

  1. Overview of Stable Diffusion pipelines

    main

    Stable Diffusion is a text-to-image latent diffusion model that operates in a lower-dimensional latent space to reduce memory and compute requirements. The diffusers library provides various pipelines to support different tasks using Stable Diffusion models.

    Available pipelines include:

    • StableDiffusion: text-to-image
    • StableDiffusionImg2Img: image-to-image
    • StableDiffusionInpaint: inpainting
    • StableDiffusionDepth2Img: depth-to-image
    • StableDiffusionImageVariation: image variation
    • StableDiffusionPipelineSafe: filtered text-to-image
    • StableDiffusion2: text-to-image, inpainting, depth-to-image, super-resolution
    • StableDiffusionXL: text-to-image, image-to-image
    • StableDiffusionLatentUpscale: super-resolution
    • StableDiffusionUpscale: super-resolution
    • StableDiffusionLDM3D: text-to-rgb, text-to-depth, text-to-pano
    • StableDiffusionUpscaleLDM3D: ldm3d super-resolution

  3. Overview of NVIDIA Cosmos 3

    main

    NVIDIA Cosmos 3 is a unified world foundation model (WFM) for Physical AI. It uses a Mixture-of-Transformers (MoT) architecture via the Cosmos3OmniTransformer to combine world generation, physical reasoning, and action generation into a single omni-model.

    Key capabilities include:

    • Generating physically plausible video from text, images, or actions.
    • Reasoning about motion, causality, and spatial relationships.
    • Predicting future video and action sequences.

    Available checkpoints on the Hugging Face Hub:

    • nvidia/Cosmos3-Nano: Smaller and faster.
    • nvidia/Cosmos3-Super: Larger and higher quality.

  4. Overview of Controlled Generation Techniques

    main

    Diffusers supports various techniques to control the output of diffusion models, ranging from inference-only methods to those requiring model fine-tuning. These techniques allow for semantic preservation, style control, and improved image quality.

    Inference-only methods

    These methods work with existing pre-trained models without requiring additional training:

    • InstructPix2Pix: Edit images using instruction-like prompts.
    • Pix2Pix Zero: Image editing via optimization.
    • Attend and Excite: Ensures specific subjects in a prompt are faithfully represented.
    • Semantic Guidance (SEGA): Adds or removes concepts (e.g., increasing a smile) by interacting with the diffusion process.
    • Self-attention Guidance (SAG): Improves general image quality using UNet self-attention maps.
    • Depth2Image: Preserves semantics for text-guided image variation using monocular depth estimates.
    • MultiDiffusion Panorama: Generates high-quality images at arbitrary aspect ratios (e.g., panoramas) using spatial guiding signals.
    • ControlNet: Adds extra conditioning (e.g., edges, depth, segmentation) via an auxiliary network.
    • Prompt Weighting: Adjusts the attention weight of specific parts of a text prompt.
    • Model Editing / DiffEdit / T2I-Adapter / FABRIC: Other inference-based control methods.

    Training/Fine-tuning methods

    These methods require training on user-provided data to learn new subjects or concepts:

    • DreamBooth: Teaches the model a new specific subject (e.g., a specific person).
    • Textual Inversion: Teaches the model a new concept or style (e.g., a specific art style).
    • Custom Diffusion: Fine-tuning for custom control.

  5. Explore popular Diffusers tasks and pipelines

    main

    Diffusers provides specialized pipelines for various generative AI tasks. Common tasks include:

    • Text-to-Image: Use Stable Diffusion Text-to-Image, unCLIP, DeepFloyd IF, or Kandinsky.
    • Text-guided Image-to-Image: Use ControlNet, InstructPix2Pix, or Stable Diffusion Image-to-Image.
    • Text-guided Image Inpainting: Use Stable Diffusion Inpainting.
    • Image Variation: Use Stable Diffusion Image Variation.
    • Super Resolution: Use Stable Diffusion Upscale or Stable Diffusion Latent Upscale.
    • Unconditional Image Generation: Use DDPM.

    You can find pre-trained models for these pipelines on the 🤗 Hub.

  6. Text-to-Video Generation with AnimateDiff

    main
    AnimateDiff is a framework that enables animating personalized text-to-image models (like Stable Diffusion) by injecting a motion modeling module into a frozen text-to-image model. This allows for generating temporally smooth animation clips while preserving the domain and diversity of the original model's outputs without requiring model-specific tuning.

  8. Available Stable Diffusion Pipelines

    main

    The Stable Diffusion implementation in diffusers provides several specialized pipelines for different generative tasks:

    • StableDiffusionPipeline: Used for standard Text-to-Image Generation.
    • StableDiffusionImg2ImgPipeline: Used for Image-to-Image Text-Guided Generation.
    • StableDiffusionInpaintPipeline: Used for Text-Guided Image Inpainting.
    • CycleDiffusionPipeline: Used for cycle-consistent image generation (e.g., transforming an image from one prompt to another and back).

  9. Use EasyAnimate for video generation

    main

    EasyAnimate is a transformer-based pipeline by Alibaba PAI designed for generating AI images and videos. It supports Text-to-Video (T2V), Image-to-Video (I2V), and Control-to-Video tasks.

    Key Capabilities:

    • Resolutions: Supports multiple resolutions with width and height ranging from 256 to 1024.
    • Frame Counts: Supports generation of 1 to 49 frames (works best at 49 frames).
    • Frame Rate: Exporting at 8 FPS is recommended.

    Recommended Checkpoints:

    • Text-to-Video (T2V): alibaba-pai/EasyAnimateV5.1-12b-zh (use torch.float16).
    • Image-to-Video (I2V) & Video-to-Video: alibaba-pai/EasyAnimateV5.1-12b-zh-InP (use torch.float16).
    • Control-to-Video: alibaba-pai/EasyAnimateV5.1-12b-zh-Control or alibaba-pai/EasyAnimateV5.1-12b-zh-Control-Camera (use torch.float16).

  10. Explore Community Projects built with Diffusers

    main

    The diffusers library powers a wide variety of community-driven applications, ranging from specialized training GUIs to real-time generation pipelines. Below is a list of notable projects that leverage diffusers for specific tasks:

    Specialized Generation & Manipulation

    • HiDiffusion: Increases resolution and speed by adding a single line of code.
    • IC-Light: Manipulates the illumination of images.
    • InstantID: Zero-shot identity-preserving generation.
    • MagicAnimate: Temporally consistent human image animation.
    • OOTDiffusion: Controllable virtual try-on using latent diffusion.
    • StoryDiffusion: Generates consistent images and videos for storytelling.
    • stable-dreamfusion: Text-to-3D, Image-to-3D, and mesh exportation using NeRF + Diffusion.

    Tools & Interfaces

    • dream-textures: Stable Diffusion integrated into Blender.
    • IOPaint: Image inpainting tool for removing or replacing objects.
    • Kohya (kohya_ss): Gradio GUI for Stable Diffusion trainers.
    • SD.Next: Advanced implementation of Stable Diffusion and other generative models.
    • StreamDiffusion: Pipeline-level solution for real-time interactive generation.
    • Stable Diffusion Server: A server configured for Inpainting, Generation, and img2img.
    • auto_diffusers: Search models on Civitai and Hugging Face.
    • Skrample: Fully modular scheduler functions with first-class diffusers integration.

  11. Available training tasks and pipelines

    main

    Diffusers provides a collection of single-purpose, self-contained training scripts located in the diffusers/examples directory. These scripts are designed to be beginner-friendly and easy to tweak by exposing data preprocessing and training loops.

    Supported training tasks include:

    Training TaskSDXL SupportLoRA Support
    unconditional image generation
    text-to-imageYesYes
    textual inversion
    DreamBoothYesYes
    ControlNetYes
    InstructPix2PixYes
    Custom Diffusion
    T2I-AdaptersYes
    Kandinsky 2.2Yes
    WuerstchenYes

  12. Use the diffusers-cli to manage pipelines

    main

    diffusers-cli is a command line client designed for running, inspecting, and packaging Diffusers pipelines. It provides several subcommands to interact with models and pipelines directly from the terminal.

    Available Commands:

    • env: Print environment information for bug reports.
    • schema: Inspect a pipeline's __call__ signature without downloading weights.
    • run: Run a pipeline locally or in a Hugging Face Sandbox.
    • custom_blocks: Package a local ModularPipelineBlocks subclass for the Hub.
    • fp16_safetensors: Convert a checkpoint to fp16 .safetensors.
    • skills: Install pre-authored skill bundles into your AI coding agent.
    # Example of accessing help for a specific subcommand
diffusers-cli run --help