Pose2Sim Documentation

Pose2Sim provides a workflow for 3D markerless kinematics (human or animal), serving as an alternative to traditional marker-based Motion Capture (MoCap) methods. It allows for research-grade accuracy and production-grade robustness using low-cost hardware like phones, webcams, or GoPros.

Key Features:

• Any Cameras: Compatible with phones, webcams, GoPros, etc.
• Research-Grade Accuracy: Validated by peer-reviewed studies.
• Multi-Person Support: Track multiple people simultaneously.
• Full 3D Kinematics: Complete OpenSim skeletal analysis with joint angles.

Use Cases:

• Sports Analysis: Field-based 3D motion capture.
• Clinical Assessment: Gait analysis in clinical settings.
• Animation: Outdoor 3D capture with fully clothed subjects.
• Research: Biomechanics studies with multiple participants.

Note: For real-time analysis with a single camera (where motion is in the sagittal or frontal plane), consider using Sports2D instead.

Pose2Sim is a pipeline designed to bridge computer vision and biomechanics by converting 2D pose estimation data into 3D joint angles. The workflow follows these stages:

1. 2D Keypoint Detection: Using a model like OpenPose to extract 2D coordinates from multiple video sources.
2. Pose2Sim Core Processing:
   • Camera Calibration: Calibrating cameras via checkerboard or converting existing calibration files.
   • 2D Tracking: Identifying and isolating the specific person of interest from background subjects.
   • 3D Triangulation: Converting 2D coordinates into 3D positions (stored as a .trc file) using a weighted Direct Linear Transform (DLT) method.
   • 3D Coordinate Filtering: Cleaning the triangulated data.
3. OpenSim Integration: Using the resulting 3D coordinates to scale a musculoskeletal model and perform inverse kinematics to obtain full-body 3D joint angles.

A core feature of Pose2Sim is bridging OpenPose data to OpenSim. This process constrains kinematics to a physically accurate skeletal model, providing more robust results than raw keypoint data by:

• Constraining Bone Lengths: Ensuring bones remain at a constant length.
• Joint Angle Limits: Restricting joints to coherent, anatomically correct angle limits.

Skeletal Model Details

The provided model combines elements from the Rajagopal (2016) human gait model and the Beaucage (2019) lifting model:

• Knee Joint: Uses the Rajagopal definition where abduction/adduction and internal/external rotation angles are constrained to the flexion/extension angle.
• Spine: Uses the Beaucage definition where each lumbar vertebra is constrained to the next, allowing for coherent spine bending even with limited tracked keypoints.
• Limitations: Hand movements are locked because standard OpenPose models do not provide hand detection.

Systematic Error Correction

The model automatically accounts for systematic labeling errors in OpenPose by offsetting model markers relative to true joint centers.

You can use any RTMlib model (hand, face, animal, or custom trained models) by providing a dictionary to the mode configuration. The dictionary must be enclosed in triple quotes and can contain local paths or URLs to .onnx or .zip files.

Key dictionary keys:

• det_class: Detection class (e.g., 'YOLOX').
• det_model: Path/URL to the detection model.
• det_input_size: List of [width, height] for detection.
• pose_class: Pose class (e.g., 'RTMPose', 'RTMO').
• pose_model: Path/URL to the pose model.
• pose_input_size: List of [width, height] for pose estimation.

Example: Custom Balanced Mode

mode = """{'det_class':'YOLOX',
       'det_model':'https://huggingface.co/datasets/DavidPagnon/rtmlib_models/resolve/main/mmpose/rtmposev1/onnx_sdk/yolox_m_8xb8-300e_humanart-c2c7a14a.onnx',
       'det_input_size':[640, 640],
       'pose_class':'RTMPose',
       'pose_model':'https://huggingface.co/datasets/DavidPagnon/rtmlib_models/resolve/main/mmpose/rtmposev1/onnx_sdk/rtmpose-m_simcc-body7_pt-body7-halpe26_256x192-4d3e73dd_20230605.onnx',
       'pose_input_size':[192,256]}"""

Example: One-stage RTMO model

mode = """{'pose_class':'RTMO', 
       'pose_model':'https://huggingface.co/datasets/DavidPagnon/rtmlib_models/resolve/main/mmpose/rtmo/onnx_sdk/rtmo-m_16xb8-600e_body7-640x640-39e78cc4_20231211.onnx', 
       'pose_input_size':[640, 640]}"""

# Equivalent to mode='balanced', with body_with_feet pose model
mode = """{'det_class':'YOLOX',
       'det_model':'https://huggingface.co/datasets/DavidPagnon/rtmlib_models/resolve/main/mmpose/rtmposev1/onnx_sdk/yolox_m_8xb8-300e_humanart-c2c7a14a.onnx',
       'det_input_size':[640, 640],
       'pose_class':'RTMPose',
       'pose_model':'https://huggingface.co/datasets/DavidPagnon/rtmlib_models/resolve/main/mmpose/rtmposev1/onnx_sdk/rtmpose-m_simcc-body7_pt-body7-halpe26_700e-256x192-4d3e73dd_20230605.onnx',
       'pose_input_size':[192,256]}"""

# With one-stage RTMO model 
mode = """{'pose_class':'RTMO', 
       'pose_model':'https://huggingface.co/datasets/DavidPagnon/rtmlib_models/resolve/main/mmpose/rtmo/onnx_sdk/rtmo-m_16xb16-600e_body7-640x640-39e78cc4_20231211.onnx', 
       'pose_input_size':[640, 640]}"""

# With animal pose estimation:
mode = """{'pose_class':'RTMPose',
       'pose_model':'https://huggingface.co/datasets/DavidPagnon/rtmlib_models/resolve/main/mmpose/rtmposev1/onnx_sdk/rtmpose-m_simcc-ap10k_pt-aic-coco_210e-256x256-7a041aa1_20230206.onnx',
       'pose_input_size':[256,256]}"""

If your cameras are not natively synchronized, use this module to find the time offset for which the correlation of mean vertical speed for chosen keypoints is highest.

How to use:

1. Open a terminal in your project folder.
2. Activate your environment.
3. Run ipython and execute the synchronization command.

Configuration: You can tune parameters (keypoints to synchronize on, reference person, and time window) via a GUI by setting synchronization_gui = true in your Config.toml, or directly in the Config.toml file.

Best practices for success:

• Ensure the participant performs clear vertical movement.
• The capture should last at least 5 seconds but no more than a few minutes to avoid camera drift.
• Avoid participants moving towards or away from the cameras.
• Alternatively, use a physical event (flashlight, clap) or GoPro-specific methods (timecode/GPS).

from Pose2Sim import Pose2Sim
Pose2Sim.synchronization()

To reduce storage requirements without significantly impacting accuracy, follow these guidelines:

• Compression: Increase the CRF (Constant Rate Factor) from 20 to 30. This can decrease storage use by up to 10x.
• Resolution: Decrease resolution from HD (1080 x 1920) to SD (720 x 1280). This can decrease storage use by about 8x.
• Audio: Audio tracks are unnecessary and can be removed to save 1-10% of file size.
• Framerate: Maintain a high framerate (at least 60 Hz). Framerates below 60 Hz lead to accuracy drops, especially for fast motions.
• Privacy: Use the provided face_blurring.py utility script to preserve privacy with minimal impact on accuracy.

You can use ffmpeg to compress and downscale your videos simultaneously.

ffmpeg -i original_vid.MP4 \
  -vcodec libx264 -crf 30 -preset fast \
  -vf "scale='min(1280,iw)':'min(720,ih)':force_original_aspect_ratio=decrease" \
  -movflags +faststart \
  lighter_vid.mp4