repository·master·Indexed 21 days ago
https://github.com/horseee/deepcacheA training-free, almost lossless paradigm designed to accelerate diffusion models by reusing high-level features in the U-Net architecture while updating low-level features cheaply. It includes modifications for Stable Diffusion pipelines, such as non-uniform sampling and feature caching loops, and provides support for Latent Diffusion Models (LDM), Retrieval Augmented Diffusion Models (RDM), and DDPM experiments.
DeepCache is compatible with various diffusion model architectures and sampling algorithms.
DeepCache accelerates diffusion models by modifying the core pipeline components to incorporate feature caching. The key architectural changes include:
pipeline_stable_diffusion.py file includes a sample_from_quad_center function to allow selecting a specific number of steps from the total inference steps.__call__ method of StableDiffusionPipeline is modified to incorporate feature caching within the outermost inference loop.pipeline_utils.py is updated to load model classes specifically designed for DeepCache compatibility.CrossAttnDownBlock2D, CrossAttnUpBlock2D (unet_2d_block.py), and unet_2d_condition.py are altered to enable the reuse or caching of features during the forward pass.To run experiments using the DDPM pipeline, first install the required dependencies. Then, navigate to the ddpm/ directory and use accelerate launch to sample images.
Available Datasets: cifar10, bedroom, church.
Key Flags:
--config: Path to the dataset configuration YAML file.--exp deepcache: Specifies the DeepCache experiment.--cache: Enables caching.--cache_interval N: Corresponds to the $1:N$ skip ratio specified in the paper.--branch B: The selected branch for the caching strategy (starting from 0). For example, branch=2 targets the 3rd skip path.--skip_type quad: Use this flag specifically for the CIFAR10 dataset to use quadratic skip type.After sampling, you can evaluate the results using FID.
# 1. Install requirements
pip install accelerate lmdb scipy diffusers pytorch_fid
# 2. Sample Images
cd ddpm/
accelerate launch ddim.py --config configs/{DATASET_NAME}.yml --exp deepcache --fid --timesteps 100 --eta 0 --ni --use_pretrained --cache --cache_interval 5 --branch 2
# 3. Testing FID
python fid.py --path runtime_log/{YOUR_PATH_FOR_IMAGES}/images npz/cifar10_fid.npzInstall the DeepCache package using pip to enable training-free acceleration for diffusion models.
pip install DeepCacheTo download and extract all available pretrained autoencoding (first stage) models, run the provided shell script. Once downloaded, models are located in models/first_stage_models/<model_spec>.
bash scripts/download_first_stages.shTo run experiments with Stable Diffusion, install the specific versions of diffusers and other dependencies. The process involves two steps: generating images using either DeepCache or a baseline, and then evaluating them using CLIP score.
DeepCache Generation Flags:
--dataset: The dataset to use (e.g., coco2017).--layer: The layer index.--block: The block index.--update_interval: The interval for updates.--uniform: Enables uniform settings.Baseline Generation Flags:
--original: Runs the original pipeline.--bk base: Runs the BK-SDM baseline.# 1. Install requirements
pip install diffusers==0.24.0 transformers open_clip_torch
# 2. Generate Images (DeepCache)
python generate.py --dataset coco2017 --layer 0 --block 0 --update_interval 2 --uniform --steps 50 --batch_size 16
# 2. Generate Images (Baselines)
python generate.py --dataset coco2017 --original --steps 50 --batch_size 16
python generate.py --dataset coco2017 --bk base --steps 50 --batch_size 16
# 3. Evaluate
python clip_score.py PATH_TO_SAVED_IMAGESGenerate images from text prompts using the txt2img.py script.
Required Setup:
Download the pre-trained weights (5.7GB) to models/ldm/text2img-large/model.ckpt before running.
Usage:
python scripts/txt2img.py --prompt "a virus monster is playing guitar, oil on canvas" --ddim_eta 0.0 --n_samples 4 --n_iter 4 --scale 5.0 --ddim_steps 50Key Arguments for Quality and Speed:
--scale: Higher values produce better samples but reduce diversity.--ddim_steps: Increasing this improves quality (diminishing returns > 250). Use low values for fast sampling.--ddim_eta: Set to 0.0 for faster sampling while retaining quality.--plms: Use for even faster sampling with good quality.--H <height> and --W <width>: Allows generating at resolutions other than 256x256 (e.g., --H 384 --W 1024).Outputs are saved to outputs/txt2img-samples by default.
To download and extract the collection of pretrained Latent Diffusion Models (LDMs), run the provided shell script. Once downloaded, models are located in models/ldm/<model_spec>.
bash scripts/download_models.shTo reproduce the results presented in the DeepCache paper using the experimental scripts, you must install specific versions of the diffusers and transformers libraries. Note that these scripts are implemented for specific model structures and may require these exact versions to function correctly due to potential updates in the diffusers library.
pip install diffusers==0.24.0 transformersUse sample_diffusion.py to sample from unconditional models (e.g., LSUN, FFHQ).
CUDA_VISIBLE_DEVICES=<GPU_ID> python scripts/sample_diffusion.py -r models/ldm/<model_spec>/model.ckpt -l <logdir> -n <#samples> --batch_size <batch_size> -c <#ddim steps> -e <#eta>CUDA_VISIBLE_DEVICES=0 python scripts/sample_diffusion.py -r models/ldm/lsun/model.ckpt -l logs/lsun -n 10 --batch_size 4 -c 50 -e 0.0Perform image inpainting using the inpaint.py script.
Required Setup:
Download weights to models/ldm/inpainting_big/last.ckpt.
Usage:
python scripts/inpaint.py --indir data/inpainting_examples/ --outdir outputs/inpainting_resultsInput Format:
--indir must contain images named *.png and corresponding masks named <image_fname>_mask.png.
RDMs support two inference modes.
Runs without explicit retrieval by using CLIP text embeddings directly.
python scripts/knn2img.py --prompt "a happy bear reading a newspaper, oil on canvas"Requires downloading retrieval databases (Openimages or ArtBench) and potentially a ScaNN search index.
To run with visual nearest neighbors:
python scripts/knn2img.py --prompt "a happy pineapple" --use_neighbors --knn <number_of_neighbors>Arguments:
--use_neighbors: Enables retrieval-based sampling.--knn <number>: Number of neighbors to retrieve (max 20).--database <name>: Specifies the database. Options include: openimages, artbench-art_nouveau, artbench-baroque, artbench-expressionism, artbench-impressionism, artbench-post_impressionism, artbench-realism, artbench-renaissance, artbench-romanticism, artbench-surrealism, artbench-ukiyo_e.Note: For --database openimages, you must first run python scripts/train_searcher.py to create the index.
python scripts/knn2img.py --prompt "a happy pineapple" --use_neighbors --knn 10