DOCSEARCHbyAlgolia
Submit Docs

StoryDiffusion

repository·main·Indexed 27 days ago

https://github.com/hvision-nku/storydiffusion

A framework for consistent self-attention in long-range image and video generation, enabling character consistency across sequences for comic generation and long-form video creation. It features custom attention processors like SpatialAttnProcessor2_0 for PyTorch 2.0+ and a process_generation engine that supports textual descriptions, reference images via PhotoMaker, and specific comic typesetting styles.

Tokens
870
Snippets
2
Records
8
Agent score
42%

What's inside StoryDiffusion

  1. Install StoryDiffusion via Conda and Pip

    main

    To set up the StoryDiffusion environment, use Python >= 3.8 (Python 3.10 is recommended) and PyTorch >= 2.0.0. It is recommended to use Anaconda or Miniconda.

    Follow these steps to create the environment and install dependencies:

    1. Create a new conda environment named storydiffusion with Python 3.10.
    2. Activate the environment.
    3. Upgrade pip.
    4. Install the project requirements using requirements.txt.
    ```bash
conda create --name storydiffusion python=3.10
conda activate storydiffusion
pip install -U pip

# Install requirements
pip install -r requirements.txt
```埋

  3. Run a local Gradio demo for comic generation

    main

    You can launch a local web interface using Gradio to generate comics. A low GPU memory version is provided, which has been tested on a machine with 24GB GPU memory (Tesla A10) and 30GB RAM. It is expected to work well on systems with >20GB of GPU memory.

    Run the following command to start the demo:

    python gradio_app_sdxl_specific_id_low_vram.py

  5. Configure Paired Self-Attention with set_attention_processor

    main

    To enable consistent character generation across multiple frames, you must configure the UNet's attention processors using set_attention_processor. This function replaces standard attention processors with SpatialAttnProcessor2_0 (for up-blocks) or AttnProcessor to facilitate long-range consistency.

    Parameters:

    • unet: The UNet model from the Stable Diffusion pipeline.
    • id_length (int): The number of initial 'identity' images used to establish character consistency.
    • is_ipadapter (bool): Set to True if using an IP-Adapter-based pipeline (like PhotoMaker).

  7. Generate consistent images with process_generation

    main

    The process_generation function is the core engine for generating a sequence of consistent images or a comic. It supports both textual descriptions and reference images (via PhotoMaker).

    Key Logic for Prompting:

    • Character Customization: If using reference images, you must include a trigger word (e.g., man img or woman img) in the general_prompt.
    • No Character Flag: Prepend [NC] to a prompt line to indicate a scene with no characters (e.g., [NC] The leaves are falling.).
    • Captions: Use # at the end of a prompt line to specify a custom caption for that frame (e.g., walking in the park #She enjoys nature).
    • Comic Typesetting: Supports Four Pannel and Classic Comic Style via the _comic_type parameter.

  8. Use SpatialAttnProcessor2_0 for PyTorch 2.0+

    main

    The SpatialAttnProcessor2_0 is a custom torch.nn.Module designed for PyTorch 2.0+ that implements specialized attention mechanisms for character consistency. It utilizes F.scaled_dot_product_attention for efficiency.

    Initialization Arguments:

    • hidden_size (int): The hidden size of the attention layer.
    • cross_attention_dim (int): Number of channels in encoder_hidden_states.
    • id_length (int, default: 4): Number of identity tokens.
    • device (str, default: "cuda"): Target device.
    • dtype (torch.dtype, default: torch.float16): Data type.