DOCSEARCHbyAlgolia
Submit Docs

ZEGGS (ZeroEGGS)

repository·main·Indexed 19 days ago

https://github.com/ubisoft/ubisoft-laforge-zeroeggs

A project for zero-shot, example-based gesture generation from speech. ZEGGS includes a stylized speech and gesture dataset covering 19 motion styles and provides tools for training models via main.py, generating gestures through a CLI, and rendering BVH gestures to FBX using a MotionBuilder script.

Tokens
1.4K
Snippets
7
Records
9
Agent score
65%

What's inside ZEGGS

  1. Install ZEGGS environment

    main

    To set up the ZEGGS environment, create a Conda virtual environment with Python 3.8, install PyTorch 1.12.x for CUDA 11.3, and install the remaining dependencies via pip. Note that you may need to install sox on your system.

    # Create and activate environment
conda create -n zeggs python=3.8
conda activate zeggs

# Install PyTorch 1.12.x for CUDA 11.3
conda install pytorch torchvision cudatoolkit=11.3 -c pytorch

# Install remaining requirements
pip install -r requirements.txt

  2. Access and prepare the ZEGGS dataset

    main

    The ZEGGS dataset consists of speech and gesture sequences. Because the repository contains large files, you must use git lfs to clone it.

    Data is located in ./data/Zeggs_data.zip, ./data/Zeggs_data.z01, and ./data/Zeggs_data.z02. Place all parts in the same folder and extract using WinRAR or Winzip.

    After extraction, run the data pipeline to create the necessary files for training and evaluation in the processed folder. You can customize the pipeline using data_pipeline_conf.json (suggested configs are in the configs folder).

    # Run the data preparation pipeline
python data_pipeline.py

  3. Render BVH gestures to FBX

    main

    To render generated gestures, use the MotionBuilder bvh2fbx script provided in the ./bvh2fbx folder.

    1. Modify bvh2fbx.py to set your Python environment path and the target FBX character.
    2. Modify bvh2fbx.bat to set paths to MotionBuilder plugins and the executable.
    3. Place pairs of .bvh and .wav files (with matching names) into ./bvh2fbx/Rendered.
    4. Run bvh2fbx.bat.
    # Run the batch rendering script
bvh2fbx.bat

  4. Train the ZEGGS model from scratch

    main

    To train the model from scratch, use main.py. You must provide an options file (configuration) and a run name. Pre-trained models are available in ./data/outputs/saved_models if you prefer not to train.

    # Example: Train with default configuration
python ./main.py -o "../configs/configs_v1.json" -n "zeggs_v1"

  5. Prepare BVH and WAV pairs for rendering

    main

    To use the rendering pipeline, place your motion data and audio files in the ZEGGS/bvh2fbx/Rendered/ directory.

    Requirements:

    • You must provide pairs of .bvh (motion) and .wav (audio) files.
    • Crucial: Each pair must have the exact same filename (e.g., animation_01.bvh and animation_01.wav).

    The rendering process will save the resulting .fbx files into this same folder.

  6. Generate gestures via CLI (Single Sample)

    main

    You can generate a single gesture from a specific audio and style pair using generate.py. The -o flag requires an options file (similar to a training config but containing paths to saved pretrained models).

    # Generate a single sample
python ./generate.py -o "../data/outputs/v1/options.json" -s "../data/clean/067_Speech_2_x_1_0.bvh" -a "../data/clean/067_Speech_2_x_1_0.wav"

  7. Generate gestures via CLI (Batch Processing)

    main

    To generate a batch of animated gestures, provide a CSV file containing paths to audio and style files along with other parameters using the -c flag.

    # Generate a batch from a CSV file
python ./generate.py -o "../data/outputs/v1/options.json" -c "../data/test/evaluation_example_based.csv"

  9. Train the ZEGGS Network via CLI

    main

    The main.py script serves as the CLI entrypoint for training the ZEGGS model. It requires a JSON configuration file containing training options, network options, and path definitions. You can optionally provide a name for the run using the -n or --name flag, which will be injected into the configuration.

    Configuration Structure: The input JSON file must contain the following top-level keys:

    • train_opt: Training hyperparameters and settings.
    • net_opt: Network architecture settings.
    • paths: A dictionary containing:
      • base_path: The root directory for data and outputs.
      • path_processed_data: Relative path to the processed data directory.
      • output_dir: (Optional) Specific directory for outputs. If omitted, a timestamped directory is created under base_path/outputs/.
      • models_dir: (Optional) Specific directory for saved models. If omitted and not resuming a training run, it is created under output_dir/saved_models/.

    Automatic Directory Management:

    • The script automatically creates an output_dir and a logs directory within it.
    • It saves a copy of the final configuration used for the run to output_dir/options.json.
    • It calls save_useful_info(output_dir) to persist metadata.
    python main.py -o "../configs/configs.json" -n "test"