repository·main·Indexed 27 days ago
https://github.com/cumulo-autumn/streamdiffusionA pipeline-level solution for real-time interactive image generation. It provides performance enhancements over standard diffusion pipelines using techniques such as Stream Batch, Residual CFG (RCFG), and TensorRT acceleration. The library supports real-time text-to-image and image-to-image workflows, including a stochastic similarity filter to reduce GPU load during video input.
To run the real-time text-to-image generation demo, you must install the Python dependencies, build the frontend using pnpm, and then execute the main Python script. This example allows for real-time generation of 4x4 image grids based on changing text prompts.
pip install -r requirements.txt
cd frontend
pnpm i
pnpm run build
cd ..
python main.pyYou can run the Img2Img example using Docker with TensorRT support. Use the following commands to build and run the container. Setting ENGINE_DIR and HF_HOME to a local directory allows for faster container restarts by caching models and engines locally.
docker build -t img2img .
docker run -ti -e ENGINE_DIR=/data -e HF_HOME=/data -v ~/.cache/huggingface:/data -p 7860:7860 --gpus all img2imgUse Docker for a TensorRT-ready environment. This builds an image and runs it with GPU support.
git clone https://github.com/cumulo-autumn/StreamDiffusion.git
cd StreamDiffusion
docker build -t stream-diffusion:latest -f Dockerfile .
docker run --gpus all -it -v $(pwd):/home/ubuntu/streamdiffusion stream-diffusion:latestFollow these steps to set up StreamDiffusion in your environment.
git clone https://github.com/cumulo-autumn/StreamDiffusion.gitUsing Conda:
conda create -n streamdiffusion python=3.10
conda activate streamdiffusionOr using venv:
python -m venv .venv
# Windows
.\.venv\Scripts\activate
# Linux
source .venv/bin/activateInstall the version matching your CUDA setup:
CUDA 11.8:
pip3 install torch==2.1.0 torchvision==0.16.0 xformers --index-url https://download.pytorch.org/whl/cu118CUDA 12.1:
pip3 install torch==2.1.0 torchvision==0.16.0 xformers --index-url https://download.pytorch.org/whl/cu121For Users (Recommended): To install the latest version with TensorRT support:
pip install git+https://github.com/cumulo-autumn/StreamDiffusion.git@main#egg=streamdiffusion[tensorrt]To install the stable version:
pip install streamdiffusion[tensorrt]After installation, install the TensorRT extension:
python -m streamdiffusion.tools.install-tensorrtNote: On Windows, if you installed the stable version, you may need to install pywin32 manually:
pip install --force-reinstall pywin32For Developers:
python setup.py develop easy_install streamdiffusion[tensorrt]
python -m streamdiffusion.tools.install-tensorrtgit clone https://github.com/cumulo-autumn/StreamDiffusion.git
conda create -n streamdiffusion python=3.10
conda activate streamdiffusion
pip3 install torch==2.1.0 torchvision==0.16.0 xformers --index-url https://download.pytorch.org/whl/cu121
pip install git+https://github.com/cumulo-autumn/StreamDiffusion.git@main#egg=streamdiffusion[tensorrt]
python -m streamdiffusion.tools.install-tensorrtAfter creating the project and installing dependencies (using npm install, pnpm install, or yarn), start the development server using npm run dev. To automatically open the application in a new browser tab, use the --open flag.
npm run dev
# or start the server and open the app in a new browser tab
npm run dev -- --openUse npm create svelte@latest to initialize a new Svelte project. You can either create it in the current directory or specify a directory name like my-app.
# create a new project in the current directory
npm create svelte@latest
# create a new project in my-app
npm create svelte@latest my-appnpm run build. This command bundles React in production mode and optimizes the build for performance, outputting the minified files with hashed filenames into the build folder.npm run buildTo use StreamDiffusion with TensorRT support via Docker, run the following commands:
git clone https://github.com/cumulo-autumn/StreamDiffusion.git
cd StreamDiffusion
docker build -t stream-diffusion:latest -f Dockerfile .
docker run --gpus all -it -v $(pwd):/home/ubuntu/streamdiffusion stream-diffusion:latestnpm run build. You can then preview the resulting production build locally using npm run preview. Note that deploying to specific environments may require installing a SvelteKit adapter.npm run build
npm run previewnpm test.npm testThis demo provides a GUI for real-time text-to-image generation using StreamDiffusion. When you change the input prompt, it generates a 4x4 grid of images in real-time.
To run the demo, follow these steps in order:
pnpm.pip install -r requirements.txt
cd frontend
pnpm i
pnpm run build
cd ..
python main.pyThe Img2Img demo allows for real-time image-to-image generation using a webcam or screen capture. By changing the prompt, the effects are reflected in the generated image in real-time.
Prerequisites:
Execution Steps:
http://0.0.0.0:7860 (or http://localhost:7860).cd frontend
npm i
npm run build
cd ..
pip install -r requirements.txt
python main.py --acceleration tensorrt