repository·main·Indexed 26 days ago
https://github.com/facebookresearch/videopose3dA framework for 3D human pose estimation in video using temporal convolutions. It supports supervised and semi-supervised training approaches and is compatible with the Human3.6M and HumanEva datasets. The system allows for the evaluation of pretrained models, training from scratch using run.py, and visualization of 3D pose reconstructions as GIFs or MP4s.
The model's information flow can be configured based on the application requirements:
Since the inference script assumes a single-person scenario, you may want to extract specific clips or adjust frame rates using ffmpeg.
Extract a clip (e.g., from 1:00 to 1:30):
ffmpeg -i input.mp4 -ss 1:00 -to 1:30 -c copy output.mp4Adjust frame rate to 50 FPS (recommended for best results as the model was trained on 50-FPS videos):
ffmpeg -i input.mp4 -filter "minterpolate='fps=50'" -crf 0 output.mp4ffmpeg -i input.mp4 -ss 1:00 -to 1:30 -c copy output.mp4
ffmpeg -i input.mp4 -filter "minterpolate='fps=50'" -crf 0 output.mp4For HumanEva-I, you can use ground-truth poses or Detectron (Mask R-CNN) detections pretrained on COCO. Download the Mask R-CNN detections into the data directory:
cd data
wget https://dl.fbaipublicfiles.com/video-pose-3d/data_2d_humaneva15_detectron_pt_coco.npz
cd ..cd data
wget https://dl.fbaipublicfiles.com/video-pose-3d/data_2d_humaneva15_detectron_pt_coco.npz
cd ..To use Stacked Hourglass detections, download the archives from Martinez et al. and place them in the data directory, then run the conversion script:
cd data
# For pretrained detections (from h36m.zip)
python prepare_data_2d_h36m_sh.py -pt h36m.zip
# For fine-tuned detections
python prepare_data_2d_h36m_sh.py -ft stacked_hourglass_fined_tuned_240.tar.gz
cd ..cd data
python prepare_data_2d_h36m_sh.py -pt h36m.zip
python prepare_data_2d_h36m_sh.py -ft stacked_hourglass_fined_tuned_240.tar.gz
cd ..You can train new models for Human3.6M or HumanEva using run.py. By default, the script runs in training mode.
Human3.6M Training:
Default training uses 80 epochs with fine-tuned CPN detections. You can reduce the receptive field using the -arc flag to speed up training:
-arc 3,3,3,3,3 (243 frames, ~24 hours)-arc 3,3,3,3 (81 frames, ~11 hours)-arc 3,3,3 (27 frames, ~6 hours)HumanEva Training: Training for 1000 epochs using Mask R-CNN detections. Training typically takes about 50 minutes.
Setting up HumanEva-I requires the original dataset and MATLAB:
Release_Code_v1_1_beta\HumanEva_I to the root of the source tree (Release_Code_v1_1_beta/).dxAvi library requirement for visualization, patch @sync_stream/sync_stream.m by replacing line 20:
ImageStream(I) = image_stream(image_paths{I}, start_image_offset(I));
with
ImageStream(I) = 0;data/ConvertHumanEva.m to Release_Code_v1_1_beta/ and run it in MATLAB. This creates a converted_15j directory.python prepare_data_humaneva.py -p /path/to/dataset/Release_Code_v1_1_beta/converted_15j --convert-3dThis produces data_3d_humaneva15.npz and data_2d_humaneva15_gt.npz in the data directory.
python prepare_data_humaneva.py -p /path/to/dataset/Release_Code_v1_1_beta/converted_15j --convert-3dBefore using VideoPose3D, ensure you have the following installed:
Required:
Optional (for visualization and export):
To set up the Human3.6M dataset without MATLAB, follow these steps:
Poses -> D3 Positions for subjects 1, 5, 6, 7, 8, 9, and 11.cdflib Python library: pip install cdflib.Poses_D3_Positions_S*.tgz (for the subjects listed above) into a common directory. Ensure the structure follows this pattern:
/path/to/dataset/S1/MyPoseFeatures/D3_Positions/Directions.cdfdata directory:cd data
python prepare_data_h36m.py --from-source-cdf /path/to/dataset
cd ..Upon success, data_3d_h36m.npz (3D poses) and data_2d_h36m_gt.npz (ground-truth 2D poses) will be created in the data directory.
cd data
python prepare_data_h36m.py --from-source-cdf /path/to/dataset
cd ..--subjects-unlabeled argument. This allows you to use a subset of data for supervised training while using other subjects as unlabeled data for semi-supervision.You can render videos showing 2D input keypoints, 3D reconstructions, and 3D ground truth by combining --evaluate and --render.
Visualization Arguments:
--viz-subject: Subject to render (e.g., S11).--viz-action: Action to render (e.g., Walking).--viz-camera: Camera index (0-3 for Human3.6M, 0-2 for HumanEva). Default 0.--viz-video: Path to the 2D video file. If provided, a skeleton overlay is rendered on the video. If omitted, a black background is used.--viz-output: Output filename (must be .mp4 or .gif).--viz-size: Output resolution multiplier (default 5).--viz-downsample: Factor to reduce output frame rate (e.g., 2).--viz-limit: Number of frames to render (default: all).--viz-skip: Number of frames to skip at the start (useful for HumanEva synchronization).--viz-no-ground-truth: Removes the 3D ground truth viewport.--viz-export: Exports 3D joint coordinates in camera space to a NumPy archive.python run.py -k cpn_ft_h36m_dbb -arc 3,3,3,3,3 -c checkpoint --evaluate pretrained_h36m_cpn.bin --render --viz-subject S11 --viz-action Walking --viz-camera 0 --viz-video "/path/to/videos/S11/Videos/Walking.54138969.mp4" --viz-output output.gif --viz-size 3 --viz-downsample 2 --viz-limit 60To evaluate pretrained models, download the weights into a checkpoint/ directory. If the directory does not exist, create it first.
mkdir checkpoint
cd checkpoint
wget https://dl.fbaipublicfiles.com/video-pose-3d/pretrained_h36m_cpn.bin
wget https://dl.fbaipublicfiles.com/video-pose-3d/pretrained_humaneva15_detectron.bin
cd ..Semi-supervised learning is specifically implemented for the h36m dataset. It is automatically enabled when the --subjects-unlabeled argument is provided.
Semi-supervised Arguments:
-sun or --subjects-unlabeled: Comma-separated list of unlabeled subjects used for semi-supervision.--warmup: Number of supervised training epochs before applying the semi-supervised loss (default 1).--subset: Factor to reduce training set size (e.g., 0.1 for one tenth). Subsampling extracts random contiguous chunks from videos.--downsample: Integer factor to reduce the dataset frame rate.--no-bone-length: Disables the bone length term in the unsupervised loss.--linear-projection: Uses only focal length and principal point for 2D projection, ignoring non-linear camera distortion.--no-proj: Disables the projection consistency term in the loss function.