repository·main·Indexed 12 days ago
https://github.com/klingairesearch/liveportraitA PyTorch-based framework for efficient portrait animation of humans and animals. It features advanced stitching and retargeting controls, supporting Image-to-Video (I2V) and Video-to-Video (V2V) generation. The framework includes a Gradio interface for manual facial control via sliders and supports motion templates (.pkl files) for privacy-preserving animation.
The LivePortrait ecosystem includes several community-driven implementations and integrations for different workflows:
expression_restorer and face_editor processors.sd-webui-live-portrait extension.ComfyUI-LivePortraitKJ, ComfyUI-AdvancedLivePortrait (supports real-time preview), and comfyui-liveportrait (supports multi-faces and expression interpolation).You can create motion templates to animate portraits without sharing the original driving video. A motion template is a .pkl file containing only the motion data, making it theoretically impossible to reconstruct the original face.
.pkl file with the same name as your driving video (e.g., d0.mp4 produces d0.pkl) and saves it in the output directory..pkl file using the -d or --driving option.-d path/to/motion_template.pklWhen a driving video contains audio, LivePortrait automatically includes that audio in the generated output. The output video will also automatically match the FPS (frames per second) of the driving video.
Windows Requirement: If you are running LivePortrait on Windows, you must have ffprobe and ffmpeg installed and available in your system path to handle audio and video concatenation.
The --flag_relative_motion option determines how motion is applied from the driving input to the source.
--flag_relative_motion): Applies the motion deformation between the driving input and its canonical form. When both inputs are videos, it uses the motion offset of the current driving frame relative to the first driving frame. This is generally more stable.--no_flag_relative_motion): Uses the absolute motion of the driving input. This can amplify expression driving strength but may cause identity leakage or inter-frame jitter. To use absolute driving via CLI, use the --no_flag_relative_motion flag.# To retain only the driving video's motion (absolute driving)
python inference.py --no_flag_relative_motionThe pretrained_weights directory must follow a specific hierarchy to ensure the model components are correctly loaded. It is divided into three main subdirectories: insightface for face detection/analysis, liveportrait for standard portrait animation, and liveportrait_animals for animal-specific animation.
Ensure that the .pth and .onnx files are placed in their respective subfolders as shown in the structure below.
pretrained_weights
├── insightface
│ └── models
│ └── buffalo_l
│ ├── 2d106det.onnx
│ └── det_10g.onnx
├── liveportrait
│ ├── base_models
│ │ ├── appearance_feature_extractor.pth
│ │ ├── motion_extractor.pth
│ │ ├── spade_generator.pth
│ │ └── warping_module.pth
│ ├── landmark.onnx
│ └── retargeting_models
│ └── stitching_retargeting_module.pth
└── liveportrait_animals
├── base_models
│ ├── appearance_feature_extractor.pth
│ ├── motion_extractor.pth
│ ├── spade_generator.pth
│ └── warping_module.pth
├── retargeting_models
│ └── stitching_retargeting_module.pth
└── xpose.pthtarget lip-open ratio. For example, setting this ratio to 0 will force the mouth in the source video to remain closed.LivePortrait includes an 'animals mode' fine-tuned on approximately 230K frames of animals (primarily cats and dogs).
Requirements:
liveportrait_animals subdirectory on HuggingFace or Google Drive before running.Important Constraints:
--no_flag_stitching option.paste-back is not recommended for this mode.python inference_animals.py -s assets/examples/source/s39.jpg -d assets/examples/driving/wink.pkl --no_flag_stitching --driving_multiplier 1.75You need to download the pretrained weights into a ./pretrained_weights directory.
Option 1: Using HuggingFace CLI (Recommended)
huggingface-cli download KlingTeam/LivePortrait --local-dir pretrained_weights --exclude "*.git*" "README.md" "docs"If you are in a region where HuggingFace is inaccessible, use the mirror:
export HF_ENDPOINT=https://hf-mirror.com
huggingface-cli download KlingTeam/LivePortrait --local-dir pretrained_weights --exclude "*.git*" "README.md" "docs"Option 2: Manual Download
Download from Google Drive or Baidu Cloud and extract the contents into the ./pretrained_weights directory, ensuring the internal directory structure matches the repository requirements.
To use LivePortrait, you must first clone the repository and set up a Conda environment. Ensure you have git, conda, and FFmpeg installed on your system.
git clone https://github.com/KlingTeam/LivePortrait
cd LivePortrait
conda create -n LivePortrait python=3.10
conda activate LivePortraittorch version compatible with your CUDA version. Check your version with nvcc -V.pip install torch==1.10.1+cu111 torchvision==0.11.2 torchaudio==0.10.1 -f https://download.pytorch.org/whl/cu111/torch_stable.htmlpip install torch==2.3.0 torchvision==0.18.0 torchaudio==2.3.0 --index-url https://download.pytorch.org/whl/cu118pip install torch==2.3.0 torchvision==0.18.0 torchaudio==2.3.0 --index-url https://download.pytorch.org/whl/cu121
Note: For Windows, CUDA 11.8 is recommended for stability if higher versions cause issues.pip install -r requirements.txtNote that X-Pose is not supported on macOS. Human mode will work, but animal mode will not. Use the specific requirements file:
pip install -r requirements_macOS.txtLivePortrait supports Portrait Video Editing (v2v), allowing you to use a source video to drive animation.
Requirements:
pykalman package is installed (it is included in requirements_base.txt).Key CLI Options for v2v:
-s or --source: Specify the source video file.--driving_smooth_observation_variance: Adjust the temporal smoothness of the motion.--flag_video_editing_head_rotation: Enable head pose motion transfer.--flag_source_video_eye_retargeting: Ensures the eye-open scalar of each source frame matches the first source frame before animation.python -m liveportrait.inference --source path/to/source_video.mp4 --driving_smooth_observation_variance 0.1 --flag_video_editing_head_rotation --flag_source_video_eye_retargeting