DOCSEARCHbyAlgolia
Submit Docs

ViTPose and ViTPose++

repository·main·Indexed 22 days ago

https://github.com/vitae-transformer/vitpose

Vision Transformer-based frameworks for generic human body pose estimation, achieving up to 81.1 AP on the MS COCO test-dev set. The library supports 2D and 3D keypoint detection for humans and animals, including top-down heatmap-based estimation, associative embedding for bottom-up detection, and video-based temporal pose propagation via PoseWarper. It also includes implementations for VoxelPose (multi-view 3D), VideoPose3D, HMR for 3D mesh recovery, and DeepPose for 2D face landmark detection.

Tokens
74K
Snippets
179
Records
307
Agent score
84%

What's inside ViTPose

  2. Overview of DeepPose for face keypoint regression

    main

    DeepPose (CVPR 2014) is a human pose estimation method that utilizes deep neural networks (DNNs) to directly regress keypoint coordinates. It operates using a top-down paradigm, which involves two distinct stages:

    1. Detection: Identifying bounding boxes for the target (e.g., human or face).
    2. Estimation: Estimating the pose/keypoints within those detected bounding boxes.

    In this specific configuration (configs/face/2d_kpt_sview_rgb_img/deeppose), the model is used for 2D keypoint estimation from RGB images.

  4. Overview of 3D human pose estimation in video (VideoPose3D)

    main
    This configuration implements the VideoPose3D (CVPR 2019) approach for 3D human pose estimation in video. The method works by 'lifting' a sequence of 2D keypoints directly into 3D keypoints using temporal convolutions and semi-supervised training. This is designed to improve temporal consistency in 3D pose estimation from video sequences.

  6. Perform 2D Animal Keypoint Detection

    main

    ViTPose supports 2D animal pose estimation for various species such as rats, dogs, macaques, and cheetahs. This capability is used for behavioral analysis in neuroscience, medical, and ecology applications.

    To use this configuration, you must complete two main steps:

    1. Data Preparation: Prepare your dataset according to the specific requirements for animal keypoint detection.
    2. Demo Generation: Use the provided demo tools to visualize the results.

    Refer to the following guides for detailed instructions:

  7. Video-based Single-view 3D Human Body Pose Estimation Overview

    main

    This configuration is designed for video-based 3D human body pose estimation, which involves detecting and analyzing the X, Y, and Z coordinates of human body joints from a sequence of RGB images. This task can be approached via three methodologies:

    1. 2D-to-3D pose lifting: Converting existing 2D poses into 3D poses.
    2. Joint learning: Learning 2D and 3D poses simultaneously.
    3. Direct regression: Regressing 3D poses directly from RGB images.

  10. Multi-view 3D Human Body Pose Estimation with VoxelPose

    main
    Multi-view 3D human body pose estimation predicts the X, Y, Z coordinates of human body joints using multiple RGB image views. For this specific task, the repository currently supports the VoxelPose implementation. You can find the specific configuration and implementation details in the /configs/body/3d_kpt_mview_rgb_img/voxelpose directory.