MotionAGFormer: Transformer-GCNFormer for 3D Human Pose Estimation

To prepare the Human3.6M dataset for MotionAGFormer:

1. Download the fine-tuned Stacked Hourglass detections from MotionBERT and unzip them into the data/motion3d directory.
2. Slice the motion clips using h36m.py in the data/preprocess directory. The number of frames required depends on the model scale you intend to use.

# For MotionAGFormer-Base and MotionAGFormer-Large
python h36m.py --n-frames 243

# For MotionAGFormer-Small
python h36m.py --n-frames 81

# For MotionAGFormer-XSmall
python h36m.py --n-frames 27

You can generate GIF visualizations of the processed poses using visualize.py in the data/preprocess directory. This requires the sequence number you wish to visualize.

# For Human3.6M (expects 243 frames)
python visualize.py --dataset h36m --sequence-number <AN ARBITRARY NUMBER>

# For MPI-INF-3DHP
python visualize.py --dataset mpi --sequence-number <AN ARBITRARY NUMBER>

To set up the environment for MotionAGFormer, ensure you have Python 3.8.10, PyTorch 2.0.0, and CUDA 12.2 installed. You can install the required project dependencies using pip.

pip install -r requirements.txt

Train the model on the MPI-INF-3DHP dataset using train_3dhp.py. Configuration files are located in configs/mpi.

python train_3dhp.py --config <PATH-TO-CONFIG>

To run the demo on in-the-wild videos, follow these setup steps:

1. Download Pretrained Models: Download YOLOv3 and HRNet pretrained models and place them in ./demo/lib/checkpoint.
2. Download MotionAGFormer Checkpoint: Download the base model checkpoint and place it in ./checkpoint.
3. Prepare Videos: Place your target videos in the ./demo/video directory.
4. Run Demo: Execute demo/vis.py pointing to your video file.

python demo/vis.py --video sample_video.mp4

Train the model on the Human3.6M dataset using train.py. You must provide a configuration file from the configs/h36m directory. You can optionally integrate with Weights & Biases (wandb) for logging.

# Basic training
python train.py --config <PATH-TO-CONFIG>

# Training with Weights & Biases logging
python train.py --config configs/h36m/MotionAGFormer-base.yaml --use-wandb --wandb-name MotionAGFormer-base

To evaluate a trained model without starting a new training session, use the --eval-only flag. You must specify the checkpoint directory, the specific checkpoint file, and the corresponding configuration file.

Human3.6M Evaluation:

python train.py --eval-only --checkpoint <CHECKPOINT-DIRECTORY> --checkpoint-file <CHECKPOINT-FILE-NAME> --config <PATH-TO-CONFIG>

MPI-INF-3DHP Evaluation:

python train_3dhp.py --eval-only --checkpoint <CHECKPOINT-DIRECTORY> --checkpoint-file <CHECKPOINT-FILE-NAME> --config <PATH-TO-CONFIG>

To prepare the MPI-INF-3DHP dataset:

1. Follow the setup instructions provided by P-STMO.
2. Ensure the generated .npz files (data_train_3dhp.npz and data_test_3dhp.npz) are located in the data/motion3d directory.

The KalmanBoxTracker class maintains the internal state of an individual tracked object using a Kalman Filter with a constant velocity model.

State Representation

The Kalman Filter operates on a 7-dimensional state vector:

• [x, y, s, r, vx, vy, vs]
  • x, y: Center coordinates
  • s: Scale (area)
  • r: Aspect ratio
  • vx, vy, vs: Velocities of center, scale, and aspect ratio respectively.

Key Methods

• update(bbox): Updates the state vector with a new observation in [x1, y1, x2, y2] format.
• predict(): Advances the state vector using the constant velocity model and returns the predicted bounding box in [x1, y1, x2, y2] format.
• get_state(): Returns the current bounding box estimate in [x1, y1, x2, y2] format.

The coco_h36m function transforms keypoints from the COCO format to the Human3.6M format. It calculates specific htps_keypoints (head, thorax, pelvis, and spine) based on the input keypoints and reorders them using h36m_coco_order and spple_keypoints.

Returns:

• keypoints_h36m: The transformed keypoints in Human3.6M format.
• valid_frames: An array of indices representing frames that contain non-zero keypoint data.

# keypoints shape expected: (temporal, num_keypoints, 2)
keypoints_h36m, valid_frames = coco_h36m(keypoints)

The load_model function initializes and loads a YOLOv3 Darknet model. It supports loading weights from a file and moving the model to a GPU if available.

Parameters

• args (optional): An object containing configuration (must have cfg_file and weight_file attributes). If None, it calls arg_parse() to generate defaults.
• CUDA (optional): A boolean indicating whether to use GPU. Defaults to torch.cuda.is_available() if None is passed.
• inp_dim (optional): The input resolution for the network (default: 416).

Returns

Returns a Darknet model instance in evaluation mode.

from lib.yolov3.human_detector import load_model

# Load model with default settings
model = load_model()

# Load model with specific resolution and explicit CUDA setting
model = load_model(CUDA=True, inp_dim=416)

The revise_kpts function improves the quality of Human3.6M keypoints by handling low-confidence joints. If certain joints fall below a confidence threshold of 0.3, the function applies heuristic replacements (e.g., setting a low-confidence joint to the position of a nearby stable joint) based on which specific combination of joints is unreliable.

This is particularly useful for handling 'half body' scenarios or occlusions where specific limb joints are missing.

Returns:

• new_h36m_kpts: A numpy array of the same shape as h36m_kpts with revised joint positions.

# h36m_kpts: (B, T, K, 2), h36m_scores: (B, T, K), valid_frames: list of arrays
new_h36m_kpts = revise_kpts(h36m_kpts, h36m_scores, valid_frames)