EasyPhoto

EasyPhoto generates realistic AI portraits by combining a pre-prepared template (the generation scene) with a user's digital twin (a fine-tuned LoRA model). Unlike traditional face-swapping methods (like roop) that can introduce unnatural lighting, EasyPhoto uses Stable Diffusion's image-to-image capabilities to ensure the face is seamlessly integrated into the scene.

Training Process

1. Face Detection & Cropping: Detects the face in user images and crops them at a specific ratio.
2. Preprocessing: Uses saliency detection and skin retouching models to create clean, face-focused training images.
3. Labeling: Applies a fixed tag to each image (no external tagger required).
4. Fine-tuning: Fine-tunes a Stable Diffusion model to create the user's digital twin LoRA.
5. Validation: Uses template images for real-time validation during training, calculating the face ID gap between validation and user images to ensure a perfect digital twin.

Inference Process

1. First Diffusion:
   • Detects the face in the template image to create a mask.
   • Performs face fusion between the template and the best user image (creating fusion_image).
   • Uses affine transformation to paste the best face_id image onto the template (creating replace_image).
   • Applies ControlNet (Canny on fusion_image and OpenPose on replace_image) to maintain similarity and stability.
   • Generates the image using Stable Diffusion and the user's LoRA.
2. Second Diffusion:
   • Performs face fusion between the first result and the best user image.
   • Runs a second Stable Diffusion pass at a higher resolution for enhanced quality.

Double-blind testing allows you to compare the generation results of two different code versions (e.g., version1 and version2) using preset templates and specific user LoRAs.

Workflow

1. Setup Models: Prepare the user_id models required for the test.
2. Prepare Assets:
   • Place preset templates in a templates directory (e.g., 1.jpg, 2.jpg).
   • Place reference images for the specific user IDs in a ref_image directory (e.g., id1.jpg, id2.jpg).
3. Run Batch Inference: Run the inference script for both versions separately to generate test data.

   python3 post_infer.py --template_dir templates --output_path test_data/version1 --user_ids your_id

   (Repeat for version2 with a different --output_path)
4. Format Data: Convert the generated inference results into a JSON format for comparison.

   python3 ./double_blind/format_data2json.py --ref_images ref_image --version1_dir test_data/version1 --version2_dir test_data/version2 --output_json test_v1_v2.json

5. Launch Double-Blind UI: Start the Gradio application to view and conduct the blind test. Note: Use gradio==3.48.0 to avoid known bugs with BarPlot and other UI issues.

   python3 ./double_blind/app.py --data-path test_v1_v2.json --result-path ./result.jsonl

After running the UI, you can determine the WinningRate for version1 and version2 to document the performance for Pull Requests (PRs).

Inference is a two-stage diffusion process designed to maximize realism and resolution:

Stage 1: First Diffusion

1. Masking: Face detection is performed on the template image to create an inpainting mask.
2. Face Fusion: The template image undergoes face fusion with the optimal user image.
3. Inpainting: The fusion_image is created by inpainting the template using the fused face.
4. Affine Transformation: The face_id image (obtained during training) is affixed to the template via affine transformation to create a replaced_image.
5. ControlNet Application:
   • Canny (with color) is applied to fusion_image to extract features.
   • OpenPose is applied to replaced_image to ensure stability.
6. Generation: Stable Diffusion combines these elements with the user's digital doppelgänger (LoRA) to generate the initial result.

Stage 2: Second Diffusion

1. Refinement: The result from the First Diffusion is fused again with the optimal user image.
2. High-Resolution Generation: Stable Diffusion is run a second time with the digital doppelgänger at a higher resolution to produce the final portrait.