DOCSEARCHbyAlgolia
Submit Docs

PIRenderer Documentation

repository·main·Indexed 19 days ago

https://github.com/renyurui/pirender

A controllable portrait image generation system based on semantic neural rendering. PIRenderer enables intuitive portrait editing, audio-driven facial reenactment, and motion imitation (same and cross-identity) by controlling disentangled 3DMM parameters. The system supports training via PyTorch 1.7.1 and CUDA 10.2, and provides tools for dataset preparation, 3DMM coefficient extraction, and inference.

Tokens
1.5K
Snippets
7
Records
7
Agent score
19%

What's inside PIRenderer

  1. Download trained weights for inference

    main

    Download the pre-trained weights using the provided script or via Google Drive/Baidu Drive links. Unzip the files and save them to the ./result/face directory.

    ./scripts/download_weights.sh

  2. Download the demo dataset

    main

    The demo dataset contains 514 test videos. You can download it using the provided shell script or via Google Drive/Baidu Drive links. After downloading, unzip the files and save them to the ./dataset directory.

    ./scripts/download_demo_dataset.sh

  3. Perform intuitive portrait control

    main

    Intuitive control allows generating results by providing specific controlling coefficients. This requires the DeepFaceRecon environment to be built first.

    Workflow:

    1. Copy PIRenderer scripts into the Deep3DFaceRecon_pytorch folder.
    2. Extract 3DMM coefficients from demo images using coeff_detector.py.
    3. Run intuitive_control.py to generate the controlled images.
    # 1. Copy the provided scrips to the folder `Deep3DFaceRecon_pytorch`.
cp scripts/face_recon_videos.py ./Deep3DFaceRecon_pytorch
cp scripts/extract_kp_videos.py ./Deep3DFaceRecon_pytorch
cp scripts/coeff_detector.py ./Deep3DFaceRecon_pytorch
cp scripts/inference_options.py ./Deep3DFaceRecon_pytorch/options

cd Deep3DFaceRecon_pytorch

# 2. Extracte the 3dmm coefficients of the demo images.
python coeff_detector.py \
--input_dir ../demo_images \
--keypoint_dir ../demo_images \
--output_dir ../demo_images \
--name=model_name \
--epoch=20 \
--model facerecon   

# 3. control the source image with our model
cd ..
python -m torch.distributed.launch --nproc_per_node=1 --master_port 12345 intuitive_control.py \
--config ./config/face_demo.yaml \
--name face \
--no_resume \
--output_dir ./vox_result/face_intuitive \
--input_name ./demo_images

  4. Install PIRenderer via Conda

    main

    To install PIRenderer, create a Conda virtual environment with Python 3.6 and install PyTorch 1.7.1 with CUDA 10.2. After setting up the environment, install the remaining dependencies using the provided requirements.txt.

    Requirements:

    • Python 3
    • PyTorch 1.7.1
    • CUDA 10.2
    # 1. Create a conda virtual environment.
conda create -n PIRenderer python=3.6
conda activate PIRenderer
conda install -c pytorch pytorch=1.7.1 torchvision cudatoolkit=10.2

# 2. Install other dependencies
pip install -r requirements.txt

  5. Prepare the dataset for training

    main

    To prepare a custom dataset for training, follow these steps:

    1. Preprocess Videos: Follow the First-Order method to download and crop videos.
    2. Extract 3DMM Parameters: Use Deep3DFaceReconstruction to extract coefficients. The directory structure must follow this format:
    ${DATASET_ROOT_FOLDER}
└───path_to_videos
    ├───train
    │   └───xxx.mp4
    └───test
        └───xxx.mp4
└───path_to_3dmm_coeff
    ├───train
    │   └───xxx.mat
    └───test
        └───xxx.mat
    1. Convert to LMDB: Run the prepare_vox_lmdb.py script to save videos and 3DMM parameters into an LMDB file.
    python scripts/prepare_vox_lmdb.py \
--path path_to_videos \
--coeff_3dmm_path path_to_3dmm_coeff \
--out path_to_output_dir

  6. Run face reenactment (Same & Cross Identity)

    main

    Perform facial reenactment using the inference.py script.

    • Same Identity Reenactment: Use the standard inference command.
    • Cross Identity Reenactment: Add the --cross_id flag to the command.

    Results are saved to the specified --output_dir.

    # same identity
python -m torch.distributed.launch --nproc_per_node=1 --master_port 12345 inference.py \
--config ./config/face_demo.yaml \
--name face \
--no_resume \
--output_dir ./vox_result/face_reenactment

# cross identity
python -m torch.distributed.launch --nproc_per_node=1 --master_port 12345 inference.py \
--config ./config/face_demo.yaml \
--name face \
--no_resume \
--output_dir ./vox_result/face_reenactment_cross \
--cross_id

  7. Train the PIRenderer model

    main

    Train the model using the train.py script. You must provide a configuration file (e.g., ./config/face.yaml) and a name for the training run.

    python -m torch.distributed.launch --nproc_per_node=4 --master_port 12345 train.py \
--config ./config/face.yaml \
--name face