Insert Anything

repository·main·Indexed 20 days ago

https://github.com/song-wensong/insert-anything

An image insertion tool using in-context editing in Diffusion Transformers (DiT) to insert objects into images via text or mask prompts. It supports various VRAM configurations, including a quantized version for 10GB VRAM via Nunchaku, and provides a Gradio web UI, ComfyUI workflow support, and training scripts for mask-prompt datasets.

Tokens: 8.5K
Snippets: 32
Records: 39
Agent score: 69%

What's inside Insert Anything

Download Checkpoints for 26GB or 40GB VRAM
main
For higher VRAM configurations, download the following checkpoints and update the paths in inference.py and app.py:
1. Insert Anything Model: Download from HuggingFace (WensongSong/Insert-Anything). Replace /path/to/lora.
2. FLUX.1-Fill-dev Model: Download from HuggingFace (black-forest-labs/FLUX.1-Fill-dev). Replace /path/to/black-forest-labs-FLUX.1-Fill-dev.
3. FLUX.1-Redux-dev Model: Download from HuggingFace (black-forest-labs/FLUX.1-Redux-dev). Replace /path/to/black-forest-labs-FLUX.1-Redux-dev.
https://github.com/song-wensong/insert-anything/blob/main/README.md
Launch Gradio Web UI
main
To run the interactive Gradio interface, use the app.py script:
```
python app.py
```
https://github.com/song-wensong/insert-anything/blob/main/README.md
Train with Mask-Prompt
main
To perform mask-prompt training, follow these steps:
1. Configure Flux Paths: In experiments/config/insertanything.yaml, replace the paths for black-forest-labs-FLUX.1-Fill-dev and black-forest-labs-FLUX.1-Redux-dev.
2. Download Dataset: Get the AnyInsertion mask-prompt dataset from HuggingFace.
3. Convert Data: Use parquet_to_image.py to convert Parquet files into images.
4. Optional Testing: To enable testing during training, modify the test path in src/train/callbacks.py at line 350.
5. Execute Training: Run the training script:
```
bash scripts/train.sh
```
https://github.com/song-wensong/insert-anything/blob/main/README.md

Install Insert Anything

main

To install the project, clone the repository and set up a Conda environment with Python 3.10. Then, install the required dependencies using pip.

git clone https://github.com/song-wensong/insert-anything
cd insert-anything

# Using Conda for Linux
conda create -n insertanything python==3.10
conda activate insertanything
pip install -r requirements.txt

git clone https://github.com/song-wensong/insert-anything
cd insert-anything

conda create -n insertanything python==3.10
conda activate insertanything
pip install -r requirements.txt

Run Inference
main
Depending on your available VRAM, use the corresponding script to run inference.
For 10GB VRAM (Nunchaku version):
```
python inference_for_nunchaku.py
```
For 26GB or 40GB VRAM:
```
python inference.py
```
```
python inference_for_nunchaku.py
# or
python inference.py
```
https://github.com/song-wensong/insert-anything/blob/main/README.md
Generate Masks for Mask-Prompt Examples
main
To run mask-prompt examples, you need a mask that fully covers the area to be edited. You can use Grounded SAM or the provided draw_mask.py script:
```
python draw_mask.py
```
https://github.com/song-wensong/insert-anything/blob/main/README.md
Download Checkpoints for 10GB VRAM (Nunchaku)
main
If you are running on hardware with approximately 10GB VRAM, you must download specific quantized models and update the paths in inference_for_nunchaku.py.
Required models:
1. Insert Anything Model: Download from HuggingFace (aha2023/insert-anything-lora-for-nunchaku). Replace /path/to/lora-for-nunchaku in inference_for_nunchaku.py.
2. FLUX.1-Fill-dev Model: Download from HuggingFace (black-forest-labs/FLUX.1-Fill-dev). Replace /path/to/black-forest-labs-FLUX.1-Fill-dev.
3. FLUX.1-Redux-dev Model: Download from HuggingFace (black-forest-labs/FLUX.1-Redux-dev). Replace /path/to/black-forest-labs-FLUX.1-Redux-dev.
4. Nunchaku-FLUX.1-Fill-dev Model: Download from HuggingFace (mit-han-lab/svdq-int4-flux.1-fill-dev). Replace /path/to/svdq-int4-flux.1-fill-dev.
Note: You must also install the appropriate version of nunchaku from the official nunchaku repository.
https://github.com/song-wensong/insert-anything/blob/main/README.md
Use ComfyUI Workflows
main
Insert Anything supports ComfyUI. A dedicated repository for the official workflow can be found here: InsertAnything-ComfyUI-official.
https://github.com/song-wensong/insert-anything/blob/main/README.md
Process images and masks with data_utils
main
The src/data/data_utils.py module provides a collection of utility functions for image manipulation, mask processing, and data augmentation, primarily used for preparing datasets for model training or inference. Key capabilities include:
- Mask Scoring & Analysis: Evaluate mask connectivity and extract bounding boxes.
- Geometric Transformations: Resize images with padding, expand bounding boxes, and convert boxes to squares.
- Noise & Augmentation: Add diffusion-style noise, create mosaic effects, and perturb masks to simulate imperfect segmentation.
- Edge Detection: Extract high-frequency maps or target boundaries using Sobel filters.
Note: Many functions assume standard NumPy/OpenCV formats (e.g., HWC for images, HW for masks).
https://github.com/song-wensong/insert-anything/blob/main/src/data/data_utils.py
Use the Insert-Anything Gradio interface
main
The project provides a Gradio-based web UI for interactive object insertion.
Workflow:
1. Background Image: Upload a background image or use the ImageEditor to draw a mask directly on the image.
2. Background Mask: Either upload a separate mask file or select "Draw Mask" to use the brush tool on the background image.
3. Reference Image: Upload the object you want to insert.
4. Reference Mask: Either upload a separate mask file or select "Draw Mask" to use the brush tool on the reference image.
5. Advanced Options: Adjust the Seed to vary the generation results.
6. Run: Click the Run button to start the inference process.
Input Options: For both Background and Reference, you must choose between:
- "Draw Mask": Uses the layers/brush from the ImageEditor component.
- "Upload with Mask": Uses the uploaded file from the dedicated ImageEditor mask component.
https://github.com/song-wensong/insert-anything/blob/main/app.py
Configure FluxFill and FluxPriorRedux pipelines
main
The application relies on two primary pipelines from the diffusers library. To use the service, these must be initialized with the correct local paths and device settings.
Required Components:
- FluxFillPipeline: Used for the actual image inpainting/insertion.
- FluxPriorReduxPipeline: Used to extract priors from the reference image.
- LoRA weights: The pipeline requires loading specific LoRA weights for optimal performance.
Setup Example:
```
import torch
from diffusers import FluxFillPipeline, FluxPriorReduxPipeline

dtype = torch.bfloat16

# Initialize Fill Pipeline
pipe = FluxFillPipeline.from_pretrained(
    "/path/to/black-forest-labs-FLUX.1-Fill-dev",
    torch_dtype=dtype
).to("cuda")
pipe.load_lora_weights("/path/to/lora")

# Initialize Redux Pipeline
redux = FluxPriorReduxPipeline.from_pretrained(
    "/path/to/black-forest-labs-FLUX.1-Redux-dev"
).to(dtype=dtype).to("cuda")
```
https://github.com/song-wensong/insert-anything/blob/main/app.py
Configure training via XFL_CONFIG environment variable
main
The training script is driven by a YAML configuration file. To run the training process, you must set the XFL_CONFIG environment variable to the path of your configuration file. The script uses this file to define model paths, training hyperparameters, and dataset locations.
```
export XFL_CONFIG="/path/to/your/config.yaml"
python src/train/train.py
```
https://github.com/song-wensong/insert-anything/blob/main/src/train/train.py