DOCSEARCHbyAlgolia
Submit Docs

FasterLivePortrait

repository·master·Indexed 22 days ago

https://github.com/warmshao/fasterliveportrait

A high-performance real-time portrait animation tool that animates still images using driving videos or audio. It optimizes LivePortrait via TensorRT to achieve 30+ FPS on modern GPUs. Features include a Gradio WebUI, support for animal models, and a deployable API. Supports inference via TensorRT and Onnxruntime, with installation options for Windows integrated packages, Docker, and manual Python environments.

Tokens
8.6K
Snippets
34
Records
39
Agent score
78%

What's inside FasterLivePortrait

  1. Install FasterLivePortrait via Docker

    master

    Docker provides a pre-configured environment that eliminates the need to manually install onnxruntime-gpu and TensorRT.

    1. Install Docker on your system.
    2. Pull the image: docker pull shaoguo/faster_liveportrait:v3.
    3. Run the container, replacing $FasterLivePortrait_ROOT with your local directory path:
    docker run -it --gpus=all \
--name faster_liveportrait \
-v $FasterLivePortrait_ROOT:/root/FasterLivePortrait \
--restart=always \
-p 9870:9870 \
shaoguo/faster_liveportrait:v3 \
/bin/bash

  2. Build the FasterLivePortrait API Docker image

    master

    To build the API container image, use the docker build command pointing to the DockerfileAPI. You should specify a tag name for your image using the -t flag.

    Command:

    docker build -t shaoguo/faster_liveportrait_api:v1.0 -f DockerfileAPI .
    docker build -t shaoguo/faster_liveportrait_api:v1.0 -f DockerfileAPI .

  3. Run the FasterLivePortrait API Docker container

    master

    Run the container locally using Docker with GPU support.

    Prerequisites

    • Nvidia GPU drivers installed.
    • CUDA version 12.0 or higher.

    Configuration Options

    • Port Mapping: The default service port is 9871. You can change this by setting the SERVER_PORT environment variable and updating the -p flag in the docker command.
    • Model Path (CHECKPOINT_DIR): To avoid long download and TensorRT (TRT) conversion times, map your existing FasterLivePortrait checkpoints directory into the container using a volume (-v). If no models are found in CHECKPOINT_DIR, the system will attempt to download and convert them automatically (requires internet access).

    Example Command

    Replace the paths and image names as needed for your environment:

    docker run -d --gpus=all \
--name faster_liveportrait_api \
-v E:\my_projects\FasterLivePortrait\checkpoints:/root/FasterLivePortrait/checkpoints \
-e CHECKPOINT_DIR=/root/FasterLivePortrait/checkpoints \
-e SERVER_PORT=9871 \
-p 9871:9871 \
--restart=always \
shaoguo/faster_liveportrait_api:v1.0

    Verifying Startup

    Check the logs (docker logs <container_id>) to ensure the service is running. Successful startup is indicated by:

    INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:9871 (Press CTRL+C to quit)

    Logs are stored internally at /root/FasterLivePortrait/logs/log_run.log.

    docker run -d --gpus=all \
--name faster_liveportrait_api \
-v E:\my_projects\FasterLivePortrait\checkpoints:/root/FasterLivePortrait/checkpoints \
-e CHECKPOINT_DIR=/root/FasterLivePortrait/checkpoints \
-e SERVER_PORT=9871 \
-p 9871:9871 \
--restart=always \
shaoguo/faster_liveportrait_api:v1.0

  4. Install FasterLivePortrait via Python Virtual Environment

    master

    To install manually in a new Python environment:

    1. Install ffmpeg.
    2. Install dependencies: pip install -r requirements.txt.
    3. Follow specific instructions to install either onnxruntime-gpu or TensorRT based on your preferred inference engine.
    pip install -r requirements.txt

  5. Use Animal Models and MediaPipe

    master

    Animal Models

    To use animal models (which use ONNX and can be converted to TensorRT):

    1. Download ONNX files: huggingface-cli download warmshao/FasterLivePortrait --local-dir ./checkpoints.
    2. Convert to TensorRT (if needed).
    3. Run: python run.py --src_image assets/examples/source/s39.jpg --dri_video 0 --cfg configs/trt_infer.yaml --realtime --animal.

    MediaPipe (Alternative to InsightFace)

    To avoid copyright issues or use MediaPipe for face detection:

    • WebUI: python webui.py --mode trt --mp or python webui.py --mode onnx --mp.
    • Local Camera/Video: python run.py --src_image assets/examples/source/s12.jpg --dri_video assets/examples/driving/d0.mp4 --cfg configs/trt_mp_infer.yaml.
    # Animal model example
python run.py --src_image assets/examples/source/s39.jpg --dri_video 0 --cfg configs/trt_infer.yaml --realtime --animal

# MediaPipe WebUI example
python webui.py --mode trt --mp

  6. Launch the Gradio WebUI

    master

    The Gradio WebUI provides a browser-based interface for the model. The default port is 9870 (accessible at http://localhost:9870/).

    • For Onnxruntime: python webui.py --mode onnx
    • For TensorRT: python webui.py --mode trt

    Webcam Mode Hotkeys (when render window is focused):

    • Q: Exit
    • S: Stitching
    • Z: RelativeMotion
    • X: AnimationRegion
    • C: CropDrivingVideo
    • K, L: AdjustSourceScale
    • N, M: AdjustDriverScale
    # Launch TensorRT mode
python webui.py --mode trt

  7. Run Onnxruntime Inference

    master

    Use Onnxruntime for inference if TensorRT is not available. Note that CPU inference is extremely slow and not recommended.

    Setup:

    1. Download ONNX models: huggingface-cli download warmshao/FasterLivePortrait --local-dir ./checkpoints.
    2. (Optional) For GPU support, you may need to compile onnxruntime-gpu from source with a specific branch (liqun/ImageDecoder-cuda) to support grid_sample with CUDA.

    Command:

    python run.py \
--src_image assets/examples/source/s10.jpg \
--dri_video assets/examples/driving/d14.mp4 \
--cfg configs/onnx_infer.yaml

  8. Enable Text-to-Video/Image driving with Kokoro-82M

    master

    To use text to drive video or image animation via the 'Drive Text' tab, follow these steps:

    1. Update Code & Dependencies:
      • Run git pull origin master and pip install requirements.txt.
      • On Windows, you can run update.bat.
    2. Download Model:
      • huggingface-cli download hexgrad/Kokoro-82M --local-dir ./checkpoints/Kokoro-82M
    3. Install System Dependencies:
      • Linux: apt-get -qq -y install espeak-ng > /dev/null 2>&1
      • Windows: Install espeak-ng manually and configure the espeak-ng environment variable. The project reads this from src/pipelines/gradio_live_portrait_pipeline.py:437.
    git pull origin master
pip install requirements.txt
huggingface-cli download hexgrad/Kokoro-82M --local-dir ./checkpoints/Kokoro-82M

  9. Run the FasterLivePortrait API Container Locally

    master

    To run the API on a local machine, ensure you have Nvidia GPU drivers installed with CUDA 12.0 or higher.

    Configuration Steps:

    1. Image Name: Use the name you defined during the build step.
    2. Port Mapping: The default service port is 9871. If you change the SERVER_PORT environment variable, ensure you update the -p flag to map the new port correctly.
    3. Model Checkpoints: Use the CHECKPOINT_DIR environment variable to point to your models. To avoid re-downloading and re-converting ONNX models to TensorRT (TRT), map your local checkpoint directory into the container using a volume (-v).

    Verification:

    After running, check the logs (docker logs <container_id>) to ensure you see:

    INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:9871 (Press CTRL+C to quit)

    Running logs are stored inside the container at /root/FasterLivePortrait/logs/log_run.log.

    docker run -d --gpus=all \
--name faster_liveportrait_api \
-v E:\my_projects\FasterLivePortrait\checkpoints:/root/FasterLivePortrait/checkpoints \
-e CHECKPOINT_DIR=/root/FasterLivePortrait/checkpoints \
-e SERVER_PORT=9871 \
-p 9871:9871 \
--restart=always \
shaoguo/faster_liveportrait_api:v1.0 \
/bin/bash

  10. Enable Audio-to-Video/Image driving with JoyVASA

    master

    To use audio to drive animation, follow these steps:

    1. Update Code.
    2. Download Required Models:
      • huggingface-cli download TencentGameMate/chinese-hubert-base --local-dir ./checkpoints/chinese-hubert-base
      • huggingface-cli download jdh-algo/JoyVASA --local-dir ./checkpoints/JoyVASA
    3. Usage: Launch the WebUI. When using a video as a source, it is recommended to drive only the mouth area.
    huggingface-cli download TencentGameMate/chinese-hubert-base --local-dir ./checkpoints/chinese-hubert-base
huggingface-cli download jdh-algo/JoyVASA --local-dir ./checkpoints/JoyVASA

  11. Install FasterLivePortrait on Windows

    master

    For Windows users, the recommended method is to use the integrated package to avoid manual dependency installation.

    1. Download the integrated package.
    2. Install git.
    3. Double-click update.bat to update the code.
    4. Double-click scripts/all_onnx2trt.bat to convert ONNX models to TensorRT files (this may take time).
    5. Run the application:
      • For the WebUI: Double-click webui.bat.
      • For real-time camera operation: Double-click camera.bat.
    # Windows users should use the provided .bat scripts
update.bat
scripts/all_onnx2trt.bat
webui.bat
camera.bat

  12. Setup on Windows (One-click Package)

    master

    For Windows users, a pre-packaged version is available in the Releases page.

    1. Download and Extract the Windows integrated package.
    2. Convert ONNX to TensorRT: Double-click scripts/all_onnx2trt.bat inside the extracted folder (this may take some time).
    3. Run WebUI: Double-click webui.bat and open http://localhost:9870/ in your browser.
    4. Run Camera: Double-click camera.bat. Press q to stop. To change the source image, run camera.bat <path_to_image> via command line.

    Note: If you already have CUDA/cuDNN installed, you can skip the manual installation steps. If you encounter CUDA errors, ensure you have CUDA 12.2 and the corresponding cuDNN installed.