DOCSEARCHbyAlgolia
Submit Docs

BrushNet Documentation

repository·main·Indexed 23 days ago

https://github.com/tencentarc/brushnet

BrushNet is a plug-and-play, text-guided image inpainting model based on decomposed dual-branch diffusion. It integrates into pre-trained diffusion models such as Stable Diffusion v1.5 and SDXL to enable high-quality inpainting.

Tokens
312.2K
Snippets
696
Records
1.1K
Agent score
82%

What's inside BrushNet

  1. Overview of Diffusers training examples

    main

    The examples directory contains self-contained scripts for training or fine-tuning diffusion models. These scripts are designed to be easy to tweak, often exposing data preprocessing and the training loop directly.

    Supported training tasks include:

    • Unconditional Image Generation: Training models without text conditioning.
    • Text-to-Image fine-tuning: Adapting models to specific text-image pairs.
    • Textual Inversion: Learning new concepts via embeddings.
    • Dreambooth: Fine-tuning models on specific subjects.
    • ControlNet: Training models with additional conditioning (e.g., edges, depth).
    • InstructPix2Pix: Image editing via instructions.
    • Reinforcement Learning for Control: Training for locomotion tasks.

  3. Overview of Text2Video-Zero

    main

    Text2Video-Zero is a zero-shot video generation method that leverages existing text-to-image diffusion models (like Stable Diffusion) to create temporally consistent videos without additional training or optimization. It achieves temporal consistency by:

    1. Enriching latent codes with motion dynamics to maintain global scene and background consistency.
    2. Reprogramming frame-level self-attention using a cross-frame attention mechanism where each frame attends to the first frame, preserving the context, appearance, and identity of foreground objects.

    It supports three main modes of operation:

    • Text-to-Video: Generating video from a textual prompt.
    • Guided Generation: Combining a prompt with guidance from poses or edges.
    • Video Instruct-Pix2Pix: Instruction-guided video editing.

  4. Overview of ControlNet-XS

    main

    ControlNet-XS is a lightweight architecture designed to condition and control Stable Diffusion generation using additional control images (e.g., depth maps). It is based on the principle that control models can be significantly smaller than the original ControlNet while maintaining high quality.

    Key Benefits:

    • Speed: 20-25% faster than regular ControlNet (benchmarked with StableDiffusion-XL).
    • Memory Efficiency: Uses approximately 45% less memory.
    • Parameter Efficiency: Can achieve state-of-the-art results with as little as 1% of the base model's parameters.

  5. Overview of ControlNet-XS with Stable Diffusion XL

    main

    ControlNet-XS is a lightweight architecture designed to control Stable Diffusion generation using additional control images (e.g., depth maps) to preserve spatial information. It is based on the principle that control models can be significantly smaller than the original ControlNet while maintaining comparable quality.

    Key Benefits:

    • Speed: 20-25% faster than regular ControlNet.
    • Memory Efficiency: Uses approximately 45% less memory.
    • Parameter Efficiency: Can achieve state-of-the-art results with as little as 1% of the base model's parameters.

    This implementation supports controlling Stable Diffusion XL (Model B, 48M Parameters) and Stable Diffusion 2.1 (Model B, 14M Parameters) under the OpenRAIL license.

  6. Overview of Controlled Generation Techniques

    main

    Controlled generation in diffusion models involves influencing how semantics are preserved or changed in outputs, as well as controlling image quality, style, or realism. Techniques can be categorized by whether they require model fine-tuning or are inference-only.

    Inference-only Methods

    • InstructPix2Pix: Edit images using instruction-like prompts.
    • Pix2Pix Zero: Zero-shot image editing by translating concepts while preserving semantics.
    • Attend and Excite: Ensures all subjects in a prompt are faithfully represented.
    • Semantic Guidance (SEGA): Adds or removes concepts (e.g., increasing a smile) via conceptual prompts.
    • Self-attention Guidance (SAG): Influences attention maps during denoising.
    • Depth2Image: Uses depth information to guide generation.
    • MultiDiffusion Panorama: Enables high-resolution panoramic generation.
    • ControlNet: Uses conditioning (e.g., edges, poses) to control structure.
    • Prompt Weighting: Adjusts the influence of specific prompt tokens.
    • Model Editing: Modifies model behavior without full fine-tuning.
    • DiffEdit: Performs image editing via mask and prompt guidance.
    • T2I-Adapter: Uses adapters for structural conditioning.
    • FABRIC: Advanced control mechanism.

    Fine-tuning Required Methods

    • DreamBooth: Fine-tunes a model on a specific subject.
    • Textual Inversion: Learns a new concept as a token.
    • Custom Diffusion: Fine-tunes the model for specific concepts.

  7. Explore Community Stable Diffusion Extensions

    main

    The examples/community directory contains various specialized Stable Diffusion pipelines and extensions developed by the community. These include advanced prompting techniques, multilingual support, inpainting methods, and model manipulation tools.

    Key community extensions include:

    • Prompting & Composition:
      • Composable Stable Diffusion: Supports | in prompts for AND conditions and weight adjustments.
      • K-Diffusion: Integrates K-Diffusion samplers into the Stable Diffusion pipeline.
    • Image Manipulation:
      • Seed Resizing: Resizes images while retaining concepts from the original 512x512 generation.
      • Imagic: Uses text prompts to edit existing images.
      • Inpainting: Includes Image to Image Inpainting (overlaying two images) and Text Based Inpainting (generating masks via text prompts).
      • MagicMix: Semantic mixing of images and text prompts.
    • Multilingual Support:
      • Multilingual Stable Diffusion: Supports prompts in 50 different languages.
      • GlueGen Stable Diffusion: Uses the GlueGen adapter for multilingual prompting.
    • Model & Data Operations:
      • Checkpoint Merger: Merges saved model checkpoints.
      • Stable UnCLIP: Combines a prior model (e.g., kakaobrain/karlo-v1-alpha) with a decoder pipeline (e.g., lambdalabs/sd-image-variations-diffusers).
      • Bit Diffusion: Diffusion on discrete data.

  8. Use Stable Diffusion pipelines

    main

    Stable Diffusion provides several specialized pipelines for different generative tasks. The available pipelines include:

    • Text-to-Image Generation: Using StableDiffusionPipeline for creating images from text prompts.
    • Image-to-Image Text-Guided Generation: Using StableDiffusionImg2ImgPipeline for generating images based on an initial image and a text prompt.
    • Text-Guided Image Inpainting: Using StableDiffusionInpaintPipeline for filling in or modifying parts of an existing image guided by text.
    • Cycle Diffusion: Using CycleDiffusionPipeline for tasks involving source and target prompts to transform images (e.g., changing an astronaut riding a horse to an astronaut riding an elephant).

  9. Explore Community Pipelines and Examples

    main

    The examples/community directory contains a collection of specialized pipelines and implementations contributed by the community. These include advanced workflows for image-to-image, video generation, upscaling, and specific model architectures like Latent Consistency Models (LCM), AnimateDiff, and Stable Diffusion XL.

    Available community implementations include:

    • Latent Consistency Models: Latent Consistency Img2img Pipeline and Latent Consistency Interpolation Pipeline.
    • Video & Motion: AnimateDiff ControlNet Pipeline, AnimateDiff Image-To-Video Pipeline, and Rerender A Video Pipeline.
    • Editing & Control: SDE Drag Pipeline, Regional Prompting Pipeline, Null-Text Inversion Pipeline, and StyleAligned Pipeline.
    • Specialized Generation: DemoFusion Pipeline, Instaflow Pipeline, InstantID Pipeline, and IP Adapter FaceID.
    • Upscaling & Optimization: LDM3D-sr (LDM3D upscaler) and Stable Diffusion XL IPEX Pipeline (for Intel Xeon CPUs).

  10. Available Kandinsky 2.2 Pipelines

    main

    The BrushNet repository supports several specialized pipelines for the Kandinsky 2.2 model, which utilizes the CLIP-ViT-G image encoder and ControlNet support. Depending on your task (text-to-image, image-to-image, inpainting, or control-guided generation), you should select the appropriate pipeline class:

    • Text-to-Image / General Generation:

      • KandinskyV22Pipeline: Standard pipeline for generating images.
      • KandinskyV22CombinedPipeline: A combined version of the pipeline.
      • KandinskyV22PriorPipeline: Focused on the prior process.
      • KandinskyV22PriorEmb2EmbPipeline: Prior pipeline with embedding-to-embedding capabilities.

    • Image-to-Image:

      • KandinskyV22Img2ImgPipeline: Standard image-to-image generation.
      • KandinskyV22Img2ImgCombinedPipeline: Combined image-to-image pipeline.
      • KandinskyV22ControlnetImg2ImgPipeline: Image-to-image generation with ControlNet guidance.

    • Inpainting:

      • KandinskyV22InpaintPipeline: Standard inpainting pipeline.
      • KandinskyV22InpaintCombinedPipeline: Combined inpainting pipeline.

    • ControlNet Guided:

      • KandinskyV22ControlnetPipeline: Pipeline for generating images using ControlNet mechanisms.

  11. Text-to-Image Generation with T2I-Adapter Conditioning

    main

    T2I-Adapter allows for granular control (such as color or structure) over Stable Diffusion text-to-image generation by providing a control image (e.g., a depth map or sketch) alongside a text prompt. The process involves two main steps:

    1. Converting input images into the appropriate control image format.
    2. Passing both the control image and the prompt to the specialized adapter pipeline.

    Available pipelines include:

    • StableDiffusionAdapterPipeline: For Stable Diffusion 1.4/1.5 base models.
    • StableDiffusionXLAdapterPipeline: For Stable Diffusion-XL base models.

  12. Fine-tune Stable Diffusion XL (SDXL)

    main

    The repository supports fine-tuning Stable Diffusion XL (SDXL) via two methods:

    1. Full UNet Fine-tuning: Use the train_text_to_image_sdxl.py script.
    2. LoRA Fine-tuning: Use the train_text_to_image_lora_sdxl.py script to fine-tune both the UNet and the Text Encoder using LoRA.

    Refer to the dedicated SDXL documentation for specific details.