DOCSEARCHbyAlgolia
Submit Docs

HybrIK Documentation

repository·main·Indexed 23 days ago

https://github.com/jeffffffli/hybrik

A hybrid analytical-neural inverse kinematics solution for 3D human pose and shape estimation. It includes HybrIK for SMPL body meshes and HybrIK-X for SMPL-X whole-body meshes, featuring tools for training, evaluation, and visualization via video and image demos.

Tokens
983
Snippets
4
Records
6
Agent score
32%

What's inside HybrIK

  1. Organize dataset directory structure

    main

    HybrIK requires specific datasets (Human3.6M, MPI-INF-3DHP, 3DPW, and MSCOCO) to be organized in a specific data directory structure. Ensure your local directory matches the following layout:

    |-- data
`-- |-- h36m
    `-- |-- annotations
        `-- images
`-- |-- pw3d
    `-- |-- json
        `-- imageFiles
`-- |-- 3dhp
    `-- |-- annotation_mpi_inf_3dhp_train.json
        |-- annotation_mpi_inf_3dhp_test.json
        |-- mpi_inf_3dhp_train_set
        `-- mpi_inf_3dhp_test_set
`-- |-- coco
    `-- |-- annotations
        |   |-- person_keypoints_train2017.json
        |   `-- person_keypoints_val2017.json
        `-- train2017
        `-- val2017

  2. Install HybrIK

    main

    Follow these steps to set up the HybrIK environment using Conda. Note that PyTorch3D is optional and only required for visualization purposes.

    1. Create and activate a Conda environment with Python 3.8.
    2. Install PyTorch 1.9.1 and torchvision 0.10.1.
    3. (Optional) Install PyTorch3D for visualization.
    4. Clone the repository and install the package in editable mode.
    5. Download necessary model files and unzip them in the project root directory.
    # 1. Create a conda virtual environment.
conda create -n hybrik python=3.8 -y
conda activate hybrik

# 2. Install PyTorch
conda install pytorch==1.9.1 torchvision==0.10.1 -c pytorch

# 3. Install PyTorch3D (Optional, only for visualization)
conda install -c fvcore -c iopath -c conda-forge fvcore iopath
conda install -c bottler nvidiacub
pip install git+ssh://git@github.com/facebookresearch/pytorch3d.git@stable

# 4. Pull our code
git clone https://github.com/Jeff-sjtu/HybrIK.git
cd HybrIK

# 5. Install
pip install pycocotools
python setup.py develop  # or "pip install -e ."

  3. Evaluate HybrIK models

    main

    To evaluate a pretrained model, use the validate_smpl_cam.sh script. You must provide the configuration file and the path to the pretrained .pth file.

    ./scripts/validate_smpl_cam.sh ./configs/256x192_adam_lr1e-3-hrw48_cam_2x_w_pw3d_3dhp.yaml ./pretrained_hrnet.pth

  4. Run HybrIK-X SMPL-X Demo (Video)

    main

    To run the HybrIK-X demo for whole-body mesh recovery (SMPL-X), place the pretrained model (e.g., ./pretrained_models/hybrikx_rle_hrnet.pth) in the ./pretrained_models/ directory and run the video script.

    python scripts/demo_video_x.py --video-name examples/dance.mp4 --out-dir res_dance --save-pk --save-img

  5. Run HybrIK SMPL Demo (Video and Image)

    main

    To run the demo, ensure you have placed the pretrained model in the ./pretrained_models/ directory (e.g., ./pretrained_models/hybrik_hrnet.pth).

    SMPL Video Demo

    Visualize HybrIK on videos (processing single frames) and save results. You can use the saved .pk files with the HybrIK Blender add-on.

    SMPL Image Demo

    Visualize HybrIK on a directory of images.

  6. Train HybrIK from scratch

    main

    To start training the SMPL model with a predicted camera, use the train_smpl_cam.sh script, providing the target dataset and the configuration YAML file.

    ./scripts/train_smpl_cam.sh test_3dpw configs/256x192_adam_lr1e-3-res34_smpl_3d_cam_2x_mix_w_pw3d.yaml