MMHuman3D Documentation

MMHuman3D is an open-source PyTorch-based toolbox for using 3D human parametric models in computer vision and computer graphics. It is part of the OpenMMLab project and is designed for reproducing state-of-the-art (SOTA) methods through a modular framework.

Key capabilities include:

• Modular Framework: Reimplement popular methods and prototype new architectures/hyperparameters without modifying core code.
• Unified Data Convention: Uses the HumanData format to align various datasets via a convention toolbox.
• Visualization Toolbox: Provides differential tools for rendering human parametric models (part segmentation, depth maps, point clouds) and 2D/3D keypoints.

MMHuman3D provides configuration files, log files, and pretrained models for all supported methods. Pretrained models are evaluated on the following three benchmarks:

• 3DPW
• Human3.6M
• MPI-INF-3DHP

Specific configuration directories for each baseline method can be found within the configs/ directory of the repository.

The MMHuman3D Model Zoo provides configuration files, log files, and pre-trained models for all supported algorithms. Pre-trained models are evaluated across three primary benchmarks:

• 3DPW
• Human3.6M
• MPI-INF-3DHP

To use a specific algorithm, you should locate its corresponding configuration directory within the repository's configs/ folder.

HumanData is a subclass of the Python built-in dict designed to hold single-view, image-based data for a human being. It provides a standardized base structure for universal human data while remaining compatible with customized features.

Key Characteristics:

• Data Types: Native HumanData instances store values as numpy.ndarray or Python built-in types. They do not store torch.Tensor objects directly.
• Tensor Conversion: You can easily convert all numpy.ndarray values to torch.Tensor (including to GPU) using the .to() method.
• Coordinate Space: All annotations are transformed from world space to OpenCV camera space.

# Convert all ndarray values to CPU Tensors
dict_of_tensor = human_data.to()

# Convert all ndarray values to a specific GPU device
gpu0_device = torch.device('cuda:0')
dict_of_gpu_tensor = human_data.to(gpu0_device)

MMHuman3D supports several camera models, each with a specific intrinsic matrix K format:

• Perspective: Uses focal lengths (fx, fy) and principal points (px, py).

  K = [[fx, 0, px, 0], [0, fy, py, 0], [0, 0, 0, 1], [0, 0, 1, 0]]

• WeakPerspective: An orthographic projection used primarily for SMPL(x) models. It uses scale (sx, sy) and translation (tx, ty).
• FoVPerspective: Defined by Field of View parameters (fov, znear, zfar).
• Orthographics: Uses focal lengths and principal points in a different layout.
• FoVOrthographics: Defined by FoV parameters (min_x, max_x, etc.).

The human_data convention is the unified format used for conversion.

Key characteristics:

• The first 144 keypoints correspond to the SMPL-X keypoints.
• Keypoints with the _extra suffix are derived from Jregressor_extra.
• Keypoints with the _openpose suffix are derived from OpenPose predictions.
• To avoid ambiguity with SMPL-X, keypoints from MPI-INF-3DHP, Human3.6M, and Posetrack that share names with SMPL-X but have different meanings are distinguished using the head_h36m suffix.

The UVRenderer is a specialized wrapper for SMPL UV topology. It is used for two main tasks:

1. Warping: Mapping a 2D texture image onto an SMPL mesh.
2. Sampling: Resampling attributes (like normals) from a map back to vertex attributes.

Initialization requires a uv_param_path pointing to an smpl_uv.npz file.

from mmhuman3d.core.renderer.torch3d_renderer.builder import build_renderer

# Initialize
uv_renderer = build_renderer(dict(
    type='uv', 
    resolution=resolution, 
    device=device, 
    model_type='smpl', 
    uv_param_path='data/body_models/smpl/smpl_uv.npz'
))

# Warping a texture to a mesh
smpl_mesh.textures = uv_renderer.warp_texture(texture_image=texture_image)

# Sampling vertex normals from a normal map
vertex_normals = uv_renderer.vertex_resample(normal_map)

Regardless of the algorithm used, all preprocessed .npz files should be placed in the data/preprocessed_datasets directory within your project root. The general structure is:

mmhuman3d
├── ...
└── data
    ├── datasets
    └── preprocessed_datasets
        ├── <dataset_name_1>.npz
        ├── <dataset_name_2>.npz
        └── ...

Unlike HumanData where images and data have a one-to-one correspondence, MultiHumanData supports a many-to-one relationship. This allows a single image to correspond to multiple human body mesh recovery data blocks.

To manage this, MultiHumanData introduces a 'frame_range' key. This key is a np.ndarray with shape (-1, 2), where each element contains two pointers (indices) that define the range of data blocks associated with a specific image.

To access data for the $i$-th image:

1. Index frame_range with the primary index $i$ to get two pointers: start_idx and end_idx.
2. Use these pointers to slice the data blocks from start_idx to end_idx (inclusive of the start, up to the end pointer) to retrieve all human data corresponding to that image.