GVHMR (World-Grounded Human Motion Recovery)

To install GVHMR, clone the repository, create a Conda environment with Python 3.10, and install the required dependencies. If you want to use GVHMR as an editable package in another repository, add "python.analysis.extraPaths": ["path/to/your/package"] to your VS Code settings.json.

git clone https://github.com/zju3dv/GVHMR
cd GVHMR

conda create -y -n gvhmr python=3.10
conda activate gvhmr
pip install -r requirements.txt
pip install -e .

The relpose preprocessing module requires pycolmap. You can install it using pip:

pip install pycolmap

Note: For updates regarding this specific component, follow the upstream repository at https://github.com/zehongs/RelativePose.

To train the model, use the tools/train.py script. The release checkpoint gvhmr_siga24_release.ckpt was trained using 2x4090 GPUs for 420 epochs; be aware that different GPU settings may lead to different results.

Note: Training does not employ the same post-processing used in the test scripts, so global metrics during training may differ from test results, though they remain valid for baseline comparison.

python tools/train.py exp=gvhmr/mixed/mixed

DPVO is an optional dependency. Note that installing it is not recommended if you require fast inference speeds. Installation requires setting CUDA_HOME and installing specific packages like torch-scatter, numba, and pypose.

cd third-party/DPVO
wget https://gitlab.com/libeigen/eigen/-/archive/3.4.0/eigen-3.4.0.zip
unzip eigen-3.4.0.zip -d thirdparty && rm -rf eigen-3.4.0.zip
pip install torch-scatter -f "https://data.pyg.org/whl/torch-2.3.0+cu121.html"
pip install numba pypose
export CUDA_HOME=/usr/local/cuda-12.1/
export PATH=$PATH:/usr/local/cuda-12.1/bin/
pip install -e .

You can run demonstrations using the scripts provided in tools/demo.

• Use tools/demo/demo.py for a single video.
• Use tools/demo/demo_folder.py to process an entire folder of videos.

Note on Visual Odometry: By default, the camera is estimated using DPVO. If you know the camera is static, use the -s flag to skip visual odometry for better efficiency and compatibility with the new SimpleVO implementation.

To reproduce the results for specific datasets (3DPW, RICH, and EMDB), use the tools/train.py script with the appropriate global/task configuration.

• All datasets: Set global/task=gvhmr/test_3dpw_emdb_rich.
• Individual datasets: Set global/task to gvhmr/test_3dpw, gvhmr/test_rich, or gvhmr/test_emdb respectively.

You must provide the checkpoint path using ckpt_path.

Preprocessed data for training (AMASS, BEDLAM, H36M) and testing (3DPW, EMDB, RICH) must be downloaded from Google Drive and extracted into the inputs/ folder. The resulting structure should place each dataset's hmr4d_support subdirectory directly under its respective dataset name in inputs/.

mkdir inputs

# For Training
cd inputs
tar -xzvf AMASS_hmr4d_support.tar.gz
tar -xzvf BEDLAM_hmr4d_support.tar.gz
tar -xzvf H36M_hmr4d_support.tar.gz

# For Testing
tar -xzvf 3DPW_hmr4d_support.tar.gz
tar -xzvf EMDB_hmr4d_support.tar.gz
tar -xzvf RICH_hmr4d_support.tar.gz

GVHMR requires several pretrained models and body models. You must manually sign up for and download SMPL and SMPLX models. Other pretrained weights (DPVO, GVHMR, HMR2, VitPose, YOLO) must be downloaded from the provided Google Drive link and placed in the inputs/checkpoints/ directory following this specific structure:

inputs/checkpoints/
├── body_models/smplx/
│   └── SMPLX_{GENDER}.npz
├── body_models/smpl/
│   └── SMPL_{GENDER}.pkl
├── dpvo/
│   └── dpvo.pth
├── gvhmr/
│   └── gvhmr_siga24_release.ckpt
├── hmr2/
│   └── epoch=10-step=25000.ckpt
├── vitpose/
│   └── vitpose-h-multi-coco.pth
└── yolo/
    └── yolov8x.pt

The Renderer class in hmr4d.utils.vis.renderer provides a way to render 3D meshes onto a 2D image plane using Pytorch3D. To use it, initialize the Renderer with the image dimensions (width, height), the camera focal length, the device (e.g., "cuda"), and the mesh faces. You can then use render_mesh to draw vertices onto a target image buffer.

from hmr4d.utils.vis.renderer import Renderer
import imageio

fps = 30
focal_length = data["cam_int"][0][0, 0]
width, height = img_hw
faces = smplh[data["gender"]].bm.faces
renderer = Renderer(width, height, focal_length, "cuda", faces)
writer = imageio.get_writer("tmp_debug.mp4", fps=fps, mode="I", format="FFMPEG", macro_block_size=1)

for i in tqdm(range(length)):
    img = np.zeros((height, width, 3), dtype=np.uint8)
    img = renderer.render_mesh(smplh_out.vertices[i].cuda(), img)
    writer.append_data(img)
writer.close()