DOCSEARCHbyAlgolia
Submit Docs

InfiniteYou (InfU) Documentation

repository·main·Indexed 25 days ago

https://github.com/bytedance/infiniteyou

A framework for flexible photo recrafting that preserves identity using Diffusion Transformers (DiTs) like FLUX. It utilizes InfuseNet to inject identity features via residual connections to ensure high identity similarity, text-image alignment, and aesthetic quality. The framework supports local inference via test.py, a Gradio demo, and ComfyUI integration. It is compatible with FLUX.1-dev, FLUX.1-schnell, ControlNets, and LoRAs.

Tokens
1.8K
Snippets
5
Records
13
Agent score
33%

What's inside InfiniteYou

  1. Configure memory requirements for InfiniteYou-FLUX

    main

    InfiniteYou-FLUX v1.0 has varying VRAM requirements depending on the optimization flags used in test.py:

    • Full-performance: Requires ~43GB peak VRAM (default bf16 inference).
    • Fast CPU offloading: Use --cpu_offload to reduce peak VRAM to ~30GB with no performance degradation.
    • 8-bit quantization: Use --quantize_8bit to reduce peak VRAM to ~24GB with similar performance.
    • Combined optimization: Use both --cpu_offload and --quantize_8bit to reduce peak VRAM to ~16GB.

  2. Install InfiniteYou dependencies

    main

    To install the necessary dependencies for InfiniteYou, run the following command in your terminal. It is recommended to use a python3 virtual environment before installation.

    pip install -r requirements.txt

  4. Integrate InfiniteYou with existing FLUX and ControlNet methods

    main

    InfiniteYou features a plug-and-play design that allows for integration with various popular generative AI approaches:

    • Base Model Replacement: Compatible with any variants of FLUX.1-dev, including FLUX.1-schnell for more efficient generation (e.g., in 4 steps).
    • Controllability: Supports ControlNets and LoRAs for customized tasks.
    • Multi-concept Personalization: Compatible with OminiControl for interacting identity (ID) and object personalized generation.
    • Stylization: Compatible with IP-Adapter (IPA) to inject style references from personalized images.

  5. Download InfiniteYou and FLUX.1-dev models

    main

    Use download_models() to download the necessary model weights from Hugging Face. It downloads the ByteDance/InfiniteYou repository to ./models/InfiniteYou and the black-forest-labs/FLUX.1-dev model to ./models/FLUX.1-dev.

    Note: Access to black-forest-labs/FLUX.1-dev requires accepting their agreement on Hugging Face and authenticating via huggingface-cli login.

    download_models()

  6. Run local inference with test.py

    main

    Use the test.py script to perform identity-preserved image generation locally. You must provide an identity image and a text prompt.

    python test.py --id_image ./assets/examples/man.jpg --prompt "A man, portrait, cinematic" --out_results_dir ./results

  8. Review InfiniteYou licenses and usage restrictions

    main

    Usage of InfiniteYou is subject to several licenses:

  9. Arguments for test.py inference script

    main

    The test.py script accepts the following arguments:

    Input and Output

    • --id_image (str): Path to the input identity (ID) image. Default: ./assets/examples/man.jpg.
    • --prompt (str): Text prompt for generation. Default: A man, portrait, cinematic.
    • --out_results_dir (str): Directory to save results. Default: ./results.
    • --control_image (str or None): Path to a control image to extract five facial keypoints. Default: None.
    • --base_model_path (str): HuggingFace or local path to the base model. Default: black-forest-labs/FLUX.1-dev.
    • --model_dir (str): Path to the InfiniteYou model directory. Default: ByteDance/InfiniteYou.

    Version Control

    • --infu_flux_version (str): InfiniteYou-FLUX version (currently v1.0).
    • --model_version (str): Model variant: aes_stage2 (default, better aesthetics/alignment) or sim_stage1 (higher identity similarity).

    General Inference

    • --cuda_device (int): CUDA device ID. Default: 0.
    • --seed (int): Seed for reproducibility. Default: 0.
    • --guideance_scale (float): Diffusion guidance scale. Default: 3.5.
    • --num_steps (int): Number of inference steps. Default: 30.

    InfiniteYou-specific

    • --infusenet_conditioning_scale (float): Scale for InfuseNet conditioning. Default: 1.0.
    • --infusenet_guidance_start (float): Start point for InfuseNet guidance injection. Default: 0.0.
    • --infusenet_guidance_end (float): End point for InfuseNet guidance injection. Default: 1.0.

    Optional LoRAs

    • --enable_realism_lora (store_true): Enables the Realism LoRA.
    • --enable_anti_blur_lora (store_true): Enables the Anti-blur LoRA.

    Memory Reduction

    • --quantize_8bit (store_true): Enables 8-bit quantization.
    • --cpu_offload (store_true): Enables fast CPU offloading.

  10. Generate images with generate_image

    main

    The generate_image function is the primary entry point for image generation. It prepares the pipeline and executes the inference.

    Parameters:

    • input_image: The identity (ID) image containing a human face (PIL Image).
    • control_image: An optional image for facial keypoint control (PIL Image).
    • prompt: Text description of the target image.
    • seed: Integer seed (use 0 for random).
    • width, height: Output dimensions.
    • guidance_scale: Guidance scale for the model.
    • num_steps: Number of inference steps.
    • infusenet_conditioning_scale: Scale for InfuNet conditioning.
    • infusenet_guidance_start: Start time for InfuNet guidance.
    • infusenet_guidance_end: End time for InfuNet guidance.
    • enable_realism: Boolean to enable realism LoRA.
    • enable_anti_blur: Boolean to enable anti-blur LoRA.
    • model_version: ModelVersion.STAGE_1 or ModelVersion.STAGE_2.

  11. Prepare the InfUFluxPipeline

    main

    The prepare_pipeline function initializes or retrieves a cached InfUFluxPipeline based on the requested configuration. It handles model switching, memory management (via gc.collect() and torch.cuda.empty_cache()), and loading optional LoRAs for realism or anti-blur.

    Parameters:

    • model_version: Use ModelVersion.STAGE_1 (sim_stage1) for higher identity similarity, or ModelVersion.STAGE_2 (aes_stage2) for better text-image alignment and aesthetics.
    • enable_realism: Boolean to load the realism LoRA.
    • enable_anti_blur: Boolean to load the anti-blur LoRA.