DOCSEARCHbyAlgolia
Submit Docs

AniPortrait

repository·main·Indexed 26 days ago

https://github.com/zejun-yang/aniportrait

A framework for generating high-quality, photorealistic portrait animations driven by audio or source video (face reenactment). It supports audio-to-video and video-to-video animation, featuring a Gradio Web UI and a two-stage training process. The system utilizes pretrained weights from Stable Diffusion and Wav2Vec2 to perform motion transfer and audio-driven facial animation.

Tokens
1.8K
Snippets
5
Records
16
Agent score
88%

What's inside AniPortrait

  1. Train AniPortrait (Stage 1 and Stage 2)

    main

    Data Preparation

    Extract keypoints from raw videos (VFHQ/CelebV-HQ) and generate a training JSON:

    python -m scripts.preprocess_dataset --input_dir VFHQ_PATH --output_dir SAVE_PATH --training_json JSON_PATH

    Update the data.json_path in your training config.

    Stage 1 Training

    accelerate launch train_stage_1.py --config ./configs/train/stage1.yaml

    Stage 2 Training

    1. Download mm_sd_v15_v2.ckpt and place it in ./pretrained_weights.
    2. In stage2.yaml, specify the Stage 1 checkpoint directory and step:
    stage1_ckpt_dir: './exp_output/stage1'
stage1_ckpt_step: 30000
    1. Run training:
    accelerate launch train_stage_2.py --config ./configs/train/stage2.yaml

  3. Download and organize AniPortrait weights

    main

    All weights must be placed in the ./pretrained_weights directory. The required files include AniPortrait-specific weights and pretrained models from external sources (Stable Diffusion, Wav2Vec2, etc.).

    AniPortrait weights:

    • denoising_unet.pth
    • reference_unet.pth
    • pose_guider.pth
    • motion_module.pth
    • audio2mesh.pt
    • audio2pose.pt
    • film_net_fp16.pt

    External weights structure:

    ./pretrained_weights/
|-- image_encoder
|   |-- config.json
|   `-- pytorch_model.bin
|-- sd-vae-ft-mse
|   |-- config.json
|   |-- diffusion_pytorch_model.bin
|   `-- diffusion_pytorch_model.safetensors
|-- stable-diffusion-v1-5
|   |-- feature_extractor
|   |   `-- preprocessor_config.json
|   |-- model_index.json
|   |-- unet
|   |   |-- config.json
|   |   `-- diffusion_pytorch_model.bin
|   `-- v1-inference.yaml
|-- wav2vec2-base-960h
|   |-- config.json
|   |-- feature_extractor_config.json
|   |-- preprocessor_config.json
|   |-- pytorch_model.bin
|   |-- README.md
|   |-- special_tokens_map.json
|   |-- tokenizer_config.json
|   `-- vocab.json
|-- audio2mesh.pt
|-- audio2pose.pt
|-- denoising_unet.pth
|-- film_net_fp16.pt
|-- motion_module.pth
|-- pose_guider.pth
`-- reference_unet.pth

    Note: If you have existing installations of models like StableDiffusion V1.5, you can specify their paths in your config files (e.g., ./config/prompts/animation.yaml).

  5. Launch the AniPortrait Gradio Web UI

    main

    The project includes a Gradio-based interactive web interface with two main tabs:

    1. Audio2video: Upload audio, a reference image, and optionally a head pose video to generate animation.
    2. Video2video: Upload a reference image and a source video to transfer motion.

    To launch the interface, run the scripts/app.py script in an environment where the necessary weights and dependencies are installed.

  6. Run Self-driven inference (Pose-to-Video)

    main

    Generate animations from a reference image and a pose video.

    1. Convert raw video to pose video (keypoint sequence):
    python -m scripts.vid2pose --video_path pose_video_path.mp4
    1. Run inference: Use the animation.yaml config to specify your reference images or pose videos. Use the -acc flag to enable acceleration if film_net_fp16.pt is present in ./pretrained_weights.
    python -m scripts.pose2vid --config ./configs/prompts/animation.yaml -W 512 -H 512 -acc

  7. Run Audio-driven inference

    main

    Generate animations driven by audio and a reference image. Configure assets in animation_audio.yaml.

    Key Options:

    • Enable audio2pose: Delete the pose_temp entry in ./configs/prompts/animation_audio.yaml to enable the audio2pose model.
    • Head Pose Control: You can generate a pose_temp.npy for head pose control using: python -m scripts.generate_ref_pose --ref_video ./configs/inference/head_pose_temp/pose_ref_video.mp4 --save_path ./configs/inference/head_pose_temp/pose.npy
    python -m scripts.audio2vid --config ./configs/prompts/animation_audio.yaml -W 512 -H 512 -acc

  8. Run Face Reenactment inference

    main

    Perform face reenactment using a source face video and a reference image. Configure these assets in animation_facereenac.yaml.

    python -m scripts.vid2vid --config ./configs/prompts/animation_facereenac.yaml -W 512 -H 512 -acc

  9. Perform Video-to-Video Portrait Animation with `video2video`

    main

    Animate a reference image by transferring the motion from a source video. This is useful for reenacting specific facial expressions and movements.

    Parameters:

    • ref_img: The reference image of the person to animate.
    • source_video: The video containing the motion to be transferred.
    • size: Resolution (width and height) for the output video. Defaults to 512.
    • steps: Number of denoising steps. Defaults to 25.
    • length: Desired length of the video. Set to 0 to automatically calculate. Defaults to 60.
    • seed: Random seed for reproducibility. Defaults to 42.
    • acc_flag: If True, enables acceleration via frame interpolation. Defaults to True.

    Returns:

    • A tuple containing the path to the generated .mp4 video and the processed reference image (PIL format).