DOCSEARCHbyAlgolia
Submit Docs

JoyVASA Documentation

repository·main·Indexed 21 days ago

https://github.com/jdh-algo/joyvasa

A diffusion-based framework for audio-driven facial animation supporting both human portraits and animal faces. JoyVASA decouples facial expressions from static 3D facial representations to enable identity-independent motion generation and longer video sequences. It includes tools for motion generator training, a Gradio-based web interface, and a CLI for inference using pretrained weights from Hugging Face.

Tokens
3.3K
Snippets
12
Records
14
Agent score
74%

What's inside JoyVASA

  1. Prepare JoyVASA model checkpoints

    main

    JoyVASA requires several pretrained weights to be placed in the pretrained_weights directory. Ensure git-lfs is installed before cloning.

    1. JoyVASA Motion Generator

    git lfs install
git clone https://huggingface.co/jdh-algo/JoyVASA

    2. Audio Encoder Checkpoints

    JoyVASA supports hubert-chinese and wav2vec2-base.

    For hubert-chinese:

    git lfs install
git clone https://huggingface.co/TencentGameMate/chinese-hubert-base

    For wav2vec2-base:

    git lfs install
git clone https://huggingface.co/facebook/wav2vec2-base-960h

    Note: The motion generation model with wav2vec2 encoder is planned for future support.

    3. LivePortrait Checkpoints

    huggingface-cli download KwaiVGI/LivePortrait --local-dir pretrained_weights --exclude "*.git*" "README.md" "docs"

    Windows User Note

    On Windows, rename the folder TencentGameMate:chinese-hubert-base to chinese-hubert-base.

  2. Train the JoyVASA motion generator

    main

    The motion generator is trained using human talking face videos. The process involves two main phases: data preparation and training.

    1. Prepare Training and Validation Data

    First, update the root_dir in src/prepare_data/01_extract_motions.py with your dataset path. Then, execute the following pipeline to generate the necessary motion and audio files:

    cd src/prepare_data
python 01_extract_motions.py
python 05_extract_audio.py
python 02_gen_labels.py
python 03_merge_motions.py
python 04_gen_template.py

# Move generated files to the data directory
mv motion_templete.pkl motions.pkl train.json test.json ../../data
cd ../..

    2. Start Training

    Once data is prepared, run the training script:

    python train.py

    Experimental results will be saved in the experiments/ directory.

  3. Install JoyVASA

    main

    Follow these steps to set up the JoyVASA environment on Ubuntu or Windows.

    System Requirements

    • Ubuntu: Tested on Ubuntu 20.04, CUDA 12.1, and A100 GPUs.
    • Windows: Tested on Windows 11, CUDA 12.1, and RTX 4060 Laptop (8GB VRAM).

    Setup Steps

    1. Create Conda Environment:
      conda create -n joyvasa python=3.10 -y
conda activate joyvasa
    2. Install Dependencies:
      pip install -r requirements.txt
sudo apt-get update && sudo apt-get install ffmpeg -y
    3. Optional: Animal Animation Support (Install MultiScaleDeformableAttention):
      cd src/utils/dependencies/XPose/models/UniPose/ops
python setup.py build install
cd - # Return to root
    conda create -n joyvasa python=3.10 -y
conda activate joyvasa
pip install -r requirements.txt

  5. Animation Mode and Driving Options

    main

    JoyVASA supports two distinct animation pipelines based on the subject type and the desired motion characteristics:

    Animation Modes

    • human: Uses GradioPipeline for portrait-based facial dynamics.
    • animal: Uses GradioPipelineAnimal for animal-specific animation.

    Driving Options

    When configuring the animation, you can select a driving_option to prioritize different aspects of the motion:

    • expression-friendly: Prioritizes facial expressions.
    • pose-friendly: Prioritizes head and body pose dynamics.

  6. Use the JoyVASA Gradio Web Interface

    main

    The application provides a Gradio-based web GUI for easy interaction. Once launched, you can perform Audio-to-Video (A2V) animation by providing the following inputs:

    Core Inputs

    • Reference Image: An image of a human or animal to be animated.
    • Input Audio: The audio file that will drive the facial dynamics and head motion.

    Animation Modes

    • human: Optimized for portrait animation of people.
    • animal: Optimized for animal image animation.

    Key Animation Options

    • do crop (image): Whether to apply cropping to the input image.
    • cfg_scale: Classifier-Free Guidance scale (default: 4.0).

    Optional Animation Options

    • driving option: Choose between expression-friendly or pose-friendly modes.
    • driving multiplier: Adjusts the intensity of the motion (default: 1.0).
    • normalize lip: Whether to normalize lip movement.
    • relative motion: Whether to use relative motion.
    • paste-back: (via flag_remap_input) Whether to remap/paste back the input.
    • stitching: (via flag_stitching_input) Whether to apply stitching.

    Image Cropping Options

    • image crop scale: The scale used for cropping (default: 2.3).
    • image crop x: Horizontal crop ratio (vx_ratio).
    • image crop y: Vertical crop ratio (vy_ratio).
    python app.py

  7. Generate training and testing labels for JoyVASA

    main

    The 02_gen_labels.py script prepares dataset metadata by scanning a directory for video files and verifying the existence of corresponding audio and motion files.

    For each video file (e.g., video.mp4), the script expects matching files with the same base name but different extensions:

    • Audio: .wav
    • Motion: .pkl

    If all three files exist, a label object is created. The script then splits the total collection into a training set (80%) and a testing set (20%), saving them as train.json and test.json respectively. Each JSON entry follows this schema:

    {
    "video_name": "path/to/video.mp4",
    "audio_name": "path/to/video.wav",
    "motion_name": "path/to/video.pkl"
}
    # The script expects a root directory containing .mp4, .wav, and .pkl files with matching base names.
# Example directory structure:
# root_dir/
# ├── sample1.mp4
# ├── sample1.wav
# └── sample1.pkl

# Running the script will output:
# 1. The total count of valid items to stdout.
# 2. 'train.json' (80% of data)
# 3. 'test.json' (20% of data)

  8. Run JoyVASA via CLI

    main

    JoyVASA can be launched as a standalone application using the app.py entrypoint. The application uses tyro for command-line argument parsing. When running the script, you can specify various configurations for inference, cropping, and server settings.

    Prerequisites:

    • FFmpeg: Must be installed and accessible in your system PATH (including ffmpeg and ffprobe).
    • Python Environment: Ensure all requirements are installed as per the project setup guide.

    To run the application, execute the script via Python. The CLI will automatically parse arguments defined in ArgumentConfig.

    python app.py

  9. Run JoyVASA inference via CLI

    main

    Use inference.py to animate images using audio. You must specify the --animation_mode (either human or animal) to match your reference image. Mismatching these modes may result in incorrect outputs.

    Command Line Arguments

    • -r or --reference: Path to the reference image.
    • -a or --audio: Path to the input audio file.
    • --animation_mode: Set to human or animal.
    • --cfg_scale: Adjust the scale to change expression and pose intensity (e.g., 2.0).

    Examples

    Animate an Animal:

    python inference.py -r assets/examples/imgs/joyvasa_001.png -a assets/examples/audios/joyvasa_001.wav --animation_mode animal --cfg_scale 2.0

    Animate a Human:

    python inference.py -r assets/examples/imgs/joyvasa_003.png -a assets/examples/audios/joyvasa_003.wav --animation_mode human --cfg_scale 2.0

  10. Batch process videos with process_videos

    main

    The process_videos function allows for parallelized audio extraction from a list of video files. It utilizes a multiprocessing.Pool with a fixed number of 12 processes to speed up the extraction task across multiple CPU cores.

    from src.prepare_data.05_extract_audio import process_videos

video_files = ["video1.mp4", "video2.mp4", "video3.mp4"]
process_videos(video_files, suffix=".wav")

  11. Use GradioPipelineAnimal for animal image animation

    main

    The GradioPipelineAnimal class is specialized for animating animal images using audio-driven facial dynamics. It extends LivePortraitPipelineAnimal and provides the execute_a2v method specifically tuned for animal animation modes.

    Initialize with inference_cfg, crop_cfg, and ArgumentConfig, then call execute_a2v with the source image and audio.

    from src.gradio_pipeline import GradioPipelineAnimal
from src.config.argument_config import ArgumentConfig

# Assuming inference_cfg and crop_cfg are already defined
args = ArgumentConfig()
pipeline = GradioPipelineAnimal(inference_cfg, crop_cfg, args)

output_video_path = pipeline.execute_a2v(
    input_image="path/to/animal.jpg",
    input_audio="path/to/audio.wav",
    animation_mode="animal"
)

  12. Extract audio from videos using extract_audio

    main

    The extract_audio function extracts audio from a video file and saves it as a .wav file. It uses ffmpeg to perform the extraction with the following specifications:

    • Sample Rate: 16000 Hz (-ar 16000)
    • Channels: Mono (-ac 1)
    • Codec: pcm_s16le (-c:a pcm_s16le)
    • Output Format: WAV

    Supported video extensions include .mp4, .mov, .avi, and .webm. If the output audio file already exists, the function skips extraction. If the input file is not a supported video type, it raises an Exception.

    from src.prepare_data.05_extract_audio import extract_audio

# Extracts audio from 'input_video.mp4' to 'input_video.wav'
extract_audio("input_video.mp4", suffix=".wav")