InvSR Documentation

The diffusers.experimental module contains code for novel applications and usages of the Diffusers library that are not yet part of the stable core.

Currently, the experimental features include:

• Reinforcement Learning: An implementation of the Diffuser model.

Pipelines provide a unified, high-level API to run complex diffusion models in inference. Diffusion systems typically consist of multiple independently trained components (e.g., Autoencoders, Unet, Text Encoders, Schedulers, and Safety Checkers) that must work together. Pipelines encapsulate these components, handling the pre-processing, model forwarding, and post-processing required for an end-to-end workflow.

Key Characteristics:

• Fidelity: They load officially published weights to yield outputs consistent with original research papers.
• Simplicity: They offer a simple user interface for inference.
• Inference Only: Pipelines are designed for inference and have PyTorch's autograd disabled via torch.no_grad. They are not intended for training functionality.

To avoid requiring a Hugging Face authentication token during runtime, you can download the model weights locally using Git LFS and pass the local directory path to the from_pretrained method instead of a Hub model ID.

1. Install Git LFS and clone the repository:

git lfs install
git clone https://huggingface.co/runwayml/stable-diffusion-v1-5

2. Load the pipeline from the local path:

from diffusers import StableDiffusionPipeline

pipe = StableDiffusionPipeline.from_pretrained("./stable-diffusion-v1-5")

To set up the InvSR environment, create a conda environment named invsr and install the required dependencies including Python 3.10, PyTorch 2.4.0, and xformers 0.0.27.post2. Ensure you use the correct index URL for CUDA 12.1 support as specified in the installation steps.

conda create -n invsr python=3.10
conda activate invsr
pip install torch==2.4.0 torchvision==0.19.0 torchaudio==2.4.0 --index-url https://download.pytorch.org/whl/cu121
pip install -U xformers==0.0.27.post2 --index-url https://download.pytorch.org/whl/cu121
pip install -e ".[torch]"
pip install -r requirements.txt

To reproduce the quantitative results for Imagenet-Test and RealSRV3, use the following datasets and configuration:

• Datasets:
  • Synthetic ImageNet-Test: [Google Drive link provided in repo]
  • RealSRV3: [External link]
  • RealSet80: Located in testdata/RealSet80
• Color Fixing: When reproducing results, add the --color_fix wavelet option to the inference command.

Before starting training, complete these steps:

1. Download Weights: Download the finetuned LPIPS model (vgg16_sdturbo_lpips.pth) and place it in the weights folder.
2. Configure configs/sd-turbo-sr-ldis.yaml:
   • configs.sd_pipe.params.cache_dir: Path to SD-Turbo.
   • data.train.params.data_source: Path to training data.
   • data.val.params.dir_path: Path to low-quality validation images.
   • data.val.params.extra_dir_path: Path to high-quality validation images.
   • configs.train.batch and configs.train.microbatch: Set these to control total batch size (Total = microbatch * #GPUS * num_grad_accumulation).

The encode method returns an OobleckDiagonalGaussianDistribution. You can interact with this distribution to either get the most likely latent (the mode) or sample from it stochastically.

Methods:

• mode(): Returns the mean of the distribution (the most likely latent).
• sample(generator=None): Returns a sample drawn from the distribution using the reparameterization trick.

To reduce VRAM usage when using Kandinsky combined pipelines, you can use the following methods:

1. enable_model_cpu_offload(): Moves one whole model at a time to the GPU when its forward method is called. This has a lower impact on performance than sequential offloading.
2. enable_sequential_cpu_offload(gpu_id=None, device="cuda"): Offloads all models to CPU using accelerate. It saves state dicts to CPU and loads submodules to GPU only when needed. This provides higher memory savings but lower performance.
3. enable_xformers_memory_efficient_attention(attention_op=None): Enables xformers memory-efficient attention in the decoder pipe.

The main.py script serves as the entrypoint for the INVSR training pipeline. It uses a combination of a YAML configuration file (via OmegaConf) and command-line arguments to initialize a trainer. Command-line arguments can be used to override specific loss coefficients and training parameters defined in the YAML config.

To run training, execute main.py with the desired configuration path and optional overrides.