Strided Transformer for 3D Human Pose Estimation

To run the demo, you must prepare the checkpoint and video directories with the required assets:

1. Download Pretrained Models: Download the YOLOv3 and HRNet pretrained models from the provided Google Drive link.
2. Place Checkpoints: Move the downloaded models into the ./demo/lib/checkpoint directory.
3. Prepare Input Videos: Place your in-the-wild videos into the ./demo/video/ directory.

The project is tested on Ubuntu 18 with PyTorch 1.7.1 and Python 3.9. To set up the environment:

1. Install PyTorch 1.7.1 and Torchvision 0.8.2 following the official PyTorch instructions.
2. Install the remaining dependencies using pip.

pip3 install -r requirements.txt

Pretrained models are hosted on Google Drive. To use them, download the model files from the following link and place them in the ./checkpoint/pretrained directory of your project root:

Google Drive Link

mkdir -p checkpoint/pretrained
# Place downloaded files into this directory

To use the model, you must prepare the Human3.6M dataset in a ./dataset directory. You have two options for obtaining the data:

1. Manual Setup: Download the raw dataset from the Human3.6M website and follow the setup procedures described in the VideoPose3D repository.
2. Preprocessed Data: Download the processed files directly from this Google Drive folder.

Your directory structure should look like this:

${POSE_ROOT}/
|-- dataset
|   |-- data_3d_h36m.npz
|   |-- data_2d_h36m_gt.npz
|   |-- data_2d_h36m_cpn_ft_h36m_dbb.npz

To start training on Human3.6M, run:

python main.py

After training for several epochs, you can add the refinement module by running:

python main.py --refine --lr 1e-5 --reload --previous_dir [your model saved path]

# Initial training
python main.py

# Training with refinement module
python main.py --refine --lr 1e-5 --reload --previous_dir [your model saved path]

To run a visual demo on your own videos, follow these steps:

1. Download the YOLOv3 and HRNet pretrained models from this Google Drive folder and place them in the ./demo/lib/checkpoint directory.
2. Place your in-the-wild videos in the ./demo/video/ directory.
3. Run the visualization script using the --video flag.

python demo/vis.py --video sample_video.mp4

Run the following command to test the pretrained model on the Human3.6M dataset. This command includes the --refine and --reload flags to utilize the refinement module and the pretrained checkpoints.

python main.py --test --refine --reload --refine_reload --previous_dir 'checkpoint/pretrained'

Run the evaluation script to test the pretrained model on the Human3.6M dataset. Ensure you have downloaded the models to ./checkpoint/pretrained before running this command.

python main.py --test --refine --reload --refine_reload --previous_dir 'checkpoint/pretrained'

When initializing the Sort class, you can tune the following parameters:

• max_age (int): The number of consecutive frames a tracker can be without a detection before it is removed from the tracker list.
• min_hits (int): The number of consecutive detections required before a track is officially considered a 'hit' and returned in the output.

from sort import Sort

# High max_age allows for longer gaps in detection
# High min_hits reduces false positives from noise
tracker = Sort(max_age=5, min_hits=10)

The workflow is driven by an opt object parsed from common.opt.opts. Key configuration flags include:

• opt.train: Boolean to toggle between training and testing modes.
• opt.refine: Boolean to enable/disable the refinement stage (model['refine']).
• opt.gpu: Sets CUDA_VISIBLE_DEVICES.
• opt.dataset: Specifies the dataset name (e.g., for loading .npz files).
• opt.reload / opt.refine_reload: Boolean flags to load previously saved weights for the transformer or refinement models.
• opt.batch_size, opt.workers, opt.lr, opt.nepoch: Standard training hyperparameters.

The SORT implementation uses two different bounding box representations internally and provides utility functions to convert between them:

1. Standard BBox: [x1, y1, x2, y2] (top-left and bottom-right coordinates).
2. Kalman State (z): [x, y, s, r] where x, y is the center, s is the scale (area), and r is the aspect ratio.

Use convert_bbox_to_z(bbox) to convert from standard to Kalman format, and convert_x_to_bbox(x, score=None) to convert back.

import numpy as np
from sort import convert_bbox_to_z, convert_x_to_bbox

# Standard [x1, y1, x2, y2]
bbox = np.array([10, 10, 50, 50])

# Convert to [x, y, s, r]
z = convert_bbox_to_z(bbox)

# Convert back to [x1, y1, x2, y2]
bbox_back = convert_x_to_bbox(z)

The coco_h36m function transforms keypoints from the COCO format to the Human3.6M (H36M) format. It calculates intermediate htps_keypoints (head, thorax, pelvis, spine) to map joint positions and reorders the keypoints according to the H36M structure.

Input Requirements:

• keypoints: A NumPy array of shape (temporal, num_keypoints, 2) representing the sequence of keypoints.

Returns:

• keypoints_h36m: The transformed keypoints in H36M format.
• valid_frames: An array of indices representing frames that contain non-zero keypoint data.

import numpy as np
from demo.lib.preprocess import coco_h36m

# Assuming keypoints is a numpy array of shape (T, 17, 2)
keypoints_h36m, valid_frames = coco_h36m(keypoints)