DOCSEARCHbyAlgolia
Submit Docs

MotionBERT Documentation

repository·main·Indexed 23 days ago

https://github.com/walter0807/motionbert

A unified framework for learning human motion representations. MotionBERT supports 3D human pose estimation, action recognition, and mesh recovery. It includes a pretrained encoder for extracting motion representations from 2D skeletons in H36M format and provides tools for in-the-wild inference on custom RGB videos using AlphaPose. The framework supports training and finetuning on datasets such as NTURGB+D, 3DPW, and Human3.6M.

Tokens
11.3K
Snippets
19
Records
75
Agent score
80%

What's inside MotionBERT

  1. Expected data directory structure

    main

    The processed data should be organized according to the following directory tree structure:

    .
└── data/
    ├── motion3d/
    │   └── MB3D_f243s81/
    │       ├── AMASS
    │       └── H36M-SH
    ├── motion2d/
    │   ├── InstaVariety/
    │   │   ├── motion_all.npy
    │   │   └── id_all.npy
    │   └── posetrack18_annotations/
    │       ├── train
    │       └── ...
    └── ...

  2. Install MotionBERT

    main

    To set up the MotionBERT environment, create a new Conda environment with Python 3.7, install PyTorch compatible with your CUDA version, and install the required dependencies via requirements.txt.

    Note: Ensure you install PyTorch according to your specific CUDA version requirements.

    conda create -n motionbert python=3.7 anaconda
conda activate motionbert
# Please install PyTorch according to your CUDA version.
conda install pytorch torchvision torchaudio pytorch-cuda=11.6 -c pytorch -c nvidia
pip install -r requirements.txt

  3. Prepare data for Skeleton-based Action Recognition

    main

    To perform skeleton-based action recognition using the NTURGB+D dataset, you must download the 2D detection results (generated via HRNet by pyskl) and place them in the data/action/ directory.

    1. Download ntu60_hrnet.pkl and ntu120_hrnet.pkl to data/action/.
    2. Download the 1-shot split and place it in data/action/.

  4. Train, Finetune, or Evaluate NTURGB+D-120 (1-shot) models

    main

    For 1-shot action recognition on the NTURGB+D-120 dataset, use the train_action_1shot.py script. This follows a similar pattern to the standard NTURGB+D tasks but uses specific 1-shot configurations and scripts.

    # Train from scratch
python train_action_1shot.py \
--config configs/action/MB_train_NTU120_oneshot.yaml \
--checkpoint checkpoint/action/MB_train_NTU120_oneshot

# Finetune from a pretrained model
python train_action_1shot.py \
--config configs/action/MB_ft_NTU120_oneshot.yaml \
--pretrained checkpoint/pretrain/MB_release \
--checkpoint checkpoint/action/FT_MB_release_MB_ft_NTU120_oneshot

# Evaluate
python train_action_1shot.py \
--config configs/action/MB_train_NTU120_oneshot.yaml \
--evaluate checkpoint/action/MB_train_NTU120_oneshot/best_epoch.bin

  5. Preprocess AMASS data

    main

    To use AMASS (SMPL+H) data for pretraining, download the data from the official website and use the provided preprocessing scripts. Note that minor modifications to these scripts may be required depending on your specific data version.

    Available tools:

    • tools/compress_amass.py: Downsamples the frame rate.
    • tools/preprocess_amass.py: Renders mocap data and extracts 3D keypoints.
    • tools/convert_amass.py: Slices data into motion clips.

  6. Train Human Mesh Recovery from scratch

    main

    You can train the mesh recovery model from scratch using either the 3DPW or H36M datasets by specifying the corresponding configuration file and checkpoint directory via train_mesh.py.

    # with 3DPW
python train_mesh.py \
--config configs/mesh/MB_train_pw3d.yaml \
--checkpoint checkpoint/mesh/MB_train_pw3d

# H36M
python train_mesh.py \
--config configs/mesh/MB_train_h36m.yaml \
--checkpoint checkpoint/mesh/MB_train_h36m

  7. Perform Mesh In-the-wild Inference

    main

    To infer human mesh from extracted 2D keypoints, follow these steps:

    1. Download Checkpoint: Download the required checkpoint and place it in the directory: checkpoint/mesh/FT_MB_release_MB_ft_pw3d/.
    2. Prepare Data: Ensure your data is prepared according to the Mesh Data Guide.
    3. Run Inference: Execute infer_wild_mesh.py using your video file and the AlphaPose JSON results.

    CLI Arguments:

    • --vid_path: Path to your input video file.
    • --json_path: Path to the AlphaPose JSON results.
    • --out_path: Directory where the output will be saved.
    • --ref_3d_motion_path (Optional): Path to a .npy file containing estimated 3D motion results. Use this to provide the estimated 3D motion for the root trajectory.
    python infer_wild_mesh.py \
--vid_path <your_video.mp4> \
--json_path <alphapose-results.json> \
--out_path <output_path> \
--ref_3d_motion_path <3d-pose-results.npy> # Optional, use the estimated 3D motion for root trajectory.

  8. Extract 2D Poses for In-the-wild Inference

    main

    Before running MotionBERT inference on custom videos, you must extract 2D keypoints using AlphaPose.

    Requirements:

    • Use the Fast Pose model trained on the Halpe dataset (26 keypoints).
    • Single Person Only: MotionBERT currently only supports single-person inference. If your video has multiple people, use the Pose Tracking Module for AlphaPose and use the --focus flag to specify the target person ID.

  9. Evaluate 3D Human Pose Estimation model

    main

    To evaluate a trained model, run train.py with the --evaluate flag pointing to the desired checkpoint file (e.g., the best epoch saved during training).

    python train.py \
--config configs/pose3d/MB_train_h36m.yaml \
--evaluate checkpoint/pose3d/MB_train_h36m/best_epoch.bin

  10. Train MotionBERT for 3D Human Pose Estimation

    main

    You can train the model for 3D Human Pose Estimation either from scratch or by finetuning a pretrained MotionBERT model.

    Train from scratch

    Use the train.py script with a specific configuration and checkpoint path.

    Finetune from pretrained MotionBERT

    Use the --pretrained flag to specify the base MotionBERT weights and --checkpoint to specify the target checkpoint path for finetuning.

    # Train from scratch
python train.py \
--config configs/pose3d/MB_train_h36m.yaml \
--checkpoint checkpoint/pose3d/MB_train_h36m

# Finetune from pretrained MotionBERT
python train.py \
--config configs/pose3d/MB_ft_h36m.yaml \
--pretrained checkpoint/pretrain/MB_release \
--checkpoint checkpoint/pose3d/FT_MB_release_MB_ft_h36m

  12. Preprocess InstaVariety data

    main

    To prepare InstaVariety data, follow these steps:

    1. Download data from human_dynamics and place it in data/motion2d.
    2. Run tools/convert_insta.py to preprocess the 2D keypoints (you must specify the name_action argument).
    3. Load the resulting .pkl files, concatenate them into a motion_list, and generate the consolidated .npy files using the script below.

    Alternatively, you can download preprocessed 2D keypoints from this link and unzip them directly to data/motion2d/.

    import numpy as np
ids = []
for i, x in enumerate(motion_list):
    ids.append(np.ones(len(x))*i)
motion_all = np.concatenate(motion_list)
id_all = np.concatenate(ids)
np.save('data/motion2d/InstaVariety/motion_all.npy', motion_all)
np.save('data/motion2d/InstaVariety/id_all.npy', id_all)