DOCSEARCHbyAlgolia
Submit Docs

DemoFusion Documentation

repository·main·Indexed 24 days ago

https://github.com/pris-cv/demofusion

A framework for high-resolution image generation that extends Latent Diffusion Models (LDMs) using Progressive Upscaling, Skip Residual, and Dilated Sampling. Includes the DemoFusionSDXLPipeline for text-to-image generation and support for low-VRAM setups.

Tokens
1.5K
Snippets
5
Records
7
Agent score
35%

What's inside DemoFusion

  1. Install DemoFusion via Conda

    main

    To set up the standard environment for DemoFusion, use Conda to create a Python 3.9 environment and install the required dependencies from requirements.txt.

    conda create -n demofusion python=3.9
conda activate demofusion
pip install -r requirements.txt

  2. Install DemoFusion on Windows with 8 GB VRAM

    main

    For low-VRAM setups on Windows, follow these specific installation steps to use xformers and compatible versions of diffusers and transformers.

    git clone "https://github.com/PRIS-CV/DemoFusion"
cd DemoFusion
python -m venv venv
venv\Scripts\activate
pip install -U "xformers==0.0.22.post7+cu118" --index-url https://download.pytorch.org/whl/cu118
pip install "diffusers==0.21.4" "matplotlib==3.8.2" "transformers==4.35.2" "accelerate==0.25.0"

  4. Use DemoFusion for Low VRAM (8 GB) Text2Image

    main

    When working with limited VRAM (e.g., 8 GB), use a specialized VAE (madebyollin/sdxl-vae-fp16-fix) and set the lowvram=True flag in the pipeline call. This is suitable for resolutions like 2048x2048.

    from pipeline_demofusion_sdxl import DemoFusionSDXLPipeline
import torch
from diffusers.models import AutoencoderKL

vae = AutoencoderKL.from_pretrained("madebyollin/sdxl-vae-fp16-fix", torch_dtype=torch.float16)
model_ckpt = "stabilityai/stable-diffusion-xl-base-1.0"
pipe = DemoFusionSDXLPipeline.from_pretrained(model_ckpt, torch_dtype=torch.float16, vae=vae)
pipe = pipe.to("cuda")

prompt = "Envision a portrait of an elderly woman..."
negative_prompt = "blurry, ugly, duplicate, poorly drawn, deformed, mosaic"

images = pipe(prompt, negative_prompt=negative_prompt,
              height=2048, width=2048, view_batch_size=4, stride=64,
              num_inference_steps=40, guidance_scale=7.5,
              cosine_scale_1=3, cosine_scale_2=1, cosine_scale_3=1, sigma=0.8,
              multi_decoder=True, show_image=False, lowvram=True
             )

for i, image in enumerate(images):
    image.save('image_' + str(i) + '.png')

  5. Use DemoFusionSDXLPipeline for Text2Image

    main

    The DemoFusionSDXLPipeline allows for high-resolution text-to-image generation. For standard high-VRAM setups (approx. 17 GB VRAM required for 3072x3072), use the following pattern. Note that multi_decoder=True is recommended for resolutions exceeding 3072x3072 on an RTX 3090.

    from pipeline_demofusion_sdxl import DemoFusionSDXLPipeline
import torch

model_ckpt = "stabilityai/stable-diffusion-xl-base-1.0"
pipe = DemoFusionSDXLPipeline.from_pretrained(model_ckpt, torch_dtype=torch.float16)
pipe = pipe.to("cuda")

prompt = "Envision a portrait of an elderly woman..."
negative_prompt = "blurry, ugly, duplicate, poorly drawn, deformed, mosaic"

images = pipe(prompt, negative_prompt=negative_prompt,
              height=3072, width=3072, view_batch_size=16, stride=64,
              num_inference_steps=50, guidance_scale=7.5,
              cosine_scale_1=3, cosine_scale_2=1, cosine_scale_3=1, sigma=0.8,
              multi_decoder=True, show_image=True
             )

for i, image in enumerate(images):
    image.save('image_' + str(i) + '.png')

  6. Cite DemoFusion in research

    main

    If you use DemoFusion in your research, please cite the CVPR 2024 paper using the BibTeX entry below.

    @inproceedings{du2024demofusion,
  title={DemoFusion: Democratising High-Resolution Image Generation With No $\$\$\$,},
  author={Du, Ruoyi and Chang, Dongliang and Hospedales, Timothy and Song, Yi-Zhe and Ma, Zhanyu},
  booktitle={CVPR},
  year={2024}
}