DOCSEARCHbyAlgolia
Submit Docs

EchoMimicV2

repository·main·Indexed 26 days ago

https://github.com/antgroup/echomimic_v2

A framework for simplified, semi-body human animation that generates lifelike video from audio and reference images. It includes a pipeline for synchronized video production, support for int8 weight quantization to reduce VRAM usage, and an accelerated inference mode for faster generation. The system provides tools for reference image alignment, pose sequence processing, and audio feature extraction via Wav2Vec2.

Tokens
4K
Snippets
9
Records
29
Agent score
84%

What's inside EchoMimicV2

  1. Manual Installation of EchoMimicV2

    main

    Follow these steps for a manual setup in a controlled environment (e.g., Conda).

    System Requirements:

    • OS: Centos 7.2 / Ubuntu 22.04
    • CUDA: >= 11.7
    • GPUs: A100(80G), RTX4090D(24G), or V100(16G)
    • Python: 3.8, 3.10, or 3.11

    Steps:

    1. Clone the repository:
    git clone https://github.com/antgroup/echomimic_v2
cd echomimic_v2
    1. Create and activate a Conda environment:
    conda create -n echomimic python=3.10
conda activate echomimic
    1. Install dependencies:
    pip install pip -U
pip install torch==2.5.1 torchvision==0.20.1 torchaudio==2.5.1 xformers==0.0.28.post3 --index-url https://download.pytorch.org/whl/cu124
pip install torchao --index-url https://download.pytorch.org/whl/nightly/cu124
pip install -r requirements.txt
pip install --no-deps facenet_pytorch==2.6.0
    1. Configure ffmpeg-static: Download and decompress ffmpeg-static, then set the environment variable:
    export FFMPEG_PATH=/path/to/ffmpeg-4.4-amd64-static
    git clone https://github.com/antgroup/echomimic_v2
cd echomimic_v2
conda create -n echomimic python=3.10
conda activate echomimic
pip install pip -U
pip install torch==2.5.1 torchvision==0.20.1 torchaudio==2.5.1 xformers==0.0.28.post3 --index-url https://download.pytorch.org/whl/cu124
pip install torchao --index-url https://download.pytorch.org/whl/nightly/cu124
pip install -r requirements.txt
pip install --no-deps facenet_pytorch==2.6.0

  2. Download EchoMimicV2 Pretrained Weights

    main

    Download the necessary model checkpoints using Git LFS into a pretrained_weights directory.

    git lfs install
git clone https://huggingface.co/BadToBest/EchoMimicV2 pretrained_weights

    Weight Structure:

    • denoising_unet.pth (Main checkpoint)
    • reference_unet.pth (Main checkpoint)
    • motion_module.pth (Main checkpoint)
    • pose_encoder.pth (Main checkpoint)
    • sd-vae-ft-mse/ (VAE model)
    • audio_processor/tiny.pt (Whisper model)

  3. Automatic Installation of EchoMimicV2

    main

    Use the automatic setup script for a quick installation. This requires CUDA >= 11.7 and Python 3.10.

    1. Clone the repository:
    git clone https://github.com/antgroup/echomimic_v2
cd echomimic_v2
    1. Run the setup script:
    sh linux_setup.sh
    git clone https://github.com/antgroup/echomimic_v2
cd echomimic_v2
sh linux_setup.sh

  4. Manage EMTD Dataset

    main

    The EMTD dataset can be downloaded, sliced, and preprocessed using the provided scripts.

    1. Download: python ./EMTD_dataset/download.py
    2. Slice: bash ./EMTD_dataset/slice.sh
    3. Preprocess: python ./EMTD_dataset/preprocess.py
    python ./EMTD_dataset/download.py
bash ./EMTD_dataset/slice.sh
python ./EMTD_dataset/preprocess.py

  5. Run the EchoMimicV2 Gradio Demo

    main

    To launch the interactive web interface for EchoMimicV2, run the app.py script. This will start a Gradio server that allows you to upload images, audio, and specify pose directories via a browser.

    python app.py

  6. Align reference image for EchoMimicV2

    main

    The reference image must be aligned and padded to a specific size (e.g., MAX_SIZE = 768) to match the model's expected input format. This process involves detecting the pose in the reference image, calculating bounding boxes for the face and body, and resizing/padding the image accordingly.

    Key functions used in the demo:

    • get_img_pose: Detects poses in the reference image.
    • get_pose_params: Calculates scaling and padding parameters based on detected body/face landmarks.
    • save_aligned_img: Crops and pads the original image to the target size.
    detected_poses, height, width, ori_frame = get_img_pose(refimg_path, max_frame=None)
res_params = get_pose_params(detected_poses, MAX_SIZE)
refimg_aligned_path = save_aligned_img(ori_frame, res_params['video_params'], MAX_SIZE)

  7. Prepare inputs for EchoMimicV2

    main

    To run the animation pipeline, you need to provide three primary inputs:

    1. Reference Image (refimg_path): A static image of the person to be animated.
    2. Audio File (audio_path): The audio track that will drive the facial/body movement.
    3. Pose Sequence (pose_path): A directory containing .npy files representing the pose for each frame. If using video driving, these are extracted from a driving video.

    Note: If using_video_driving is set to False, you must provide a pre-processed pose_path directory.

    refimg_path = './assets/halfbody_demo/refimag/test.png'
audio_path ='./assets/halfbody_demo/audio/chinese/echomimicv2_woman.wav'
using_video_driving = False
if not using_video_driving:
  pose_path = './assets/halfbody_demo/pose/good'

  8. Configure FFMPEG environment variable

    main

    The application requires ffmpeg for video processing. You must ensure ffmpeg is available in your system path or explicitly set the FFMPEG_PATH environment variable to the directory containing the ffmpeg executable.

    Example:

    export FFMPEG_PATH=./ffmpeg-4.4-amd64-static

  9. Run EchoMimicV2 Inference

    main

    You can run EchoMimicV2 via a Gradio UI or through command-line inference scripts.

    Gradio UI:

    python app.py

    Standard Inference:

    python infer.py --config='./configs/prompts/infer.yaml'

    Accelerated Inference: Use the accelerated version for significantly faster generation (up to 9x speedup on A100). Ensure you use the specific accelerated configuration file.

    python infer_acc.py --config='./configs/prompts/infer_acc.yaml'

  12. Manage checkpoint directories with delete_additional_ckpt

    main
    Use delete_additional_ckpt(base_path, num_keep) to clean up training checkpoints. It scans base_path for directories starting with checkpoint-, sorts them numerically by the suffix, and deletes the oldest ones, keeping only the most recent num_keep directories.