MogaNet: Efficient Convolutional Neural Networks

MogaNet (Multi-order Gated Aggregation Network) is a family of efficient ConvNets designed using multi-order game-theoretic interaction. It is optimized for informative context mining with a focus on the complexity-performance trade-off.

Key features include:

• High scalability.
• Competitive performance on ImageNet and various vision benchmarks.
• Efficient parameter usage.
• Support for multiple downstream tasks including object detection, semantic segmentation, human pose estimation (2D & 3D), and video prediction.

The repository provides PyTorch implementations and is integrated with tools like timm and OpenMixup.

The Moganet Model Zoo provides configuration files, log files, and pre-trained models for all supported algorithms. Pre-trained models are evaluated across three primary benchmarks:

• 3DPW
• Human3.6M
• MPI-INF-3DHP

MogaNet supports multi-person human pose estimation, which involves detecting the poses (keypoints) of all people within an input image. The task can be approached using two methodologies:

1. Top-down approaches: These divide the task into two stages: first performing human detection, and then performing single-person pose estimation within the detected human bounding boxes.
2. Bottom-up approaches: These first detect all keypoints in the image and then group or associate those keypoints into individual person instances.

The mmhuman3d library is organized into several core modules that handle different aspects of 3D human pose estimation and modeling. Developers can interact with the following primary namespaces:

• mmhuman3d.apis: High-level APIs for common tasks.
• mmhuman3d.core: Foundational components including cameras, conventions, evaluation, filter, optimizer, parametric_model, and visualization.
• mmhuman3d.models: Model definitions, including architectures, backbones, discriminators, necks, heads, and losses.
• mmhuman3d.data: Data management components including datasets, data_converters, and data_structures.
• mmhuman3d.utils: General utility functions.

The project supports a variety of video prediction methods and MetaFormer architectures used within the SimVP framework.

Supported Video Prediction Methods:

• ConvLSTM
• PredRNN, PredRNN++, PredRNN.V2
• E3D-LSTM
• MAU
• CrevNet
• PhyDNet
• SimVP, SimVP.V2

Supported MetaFormers (for SimVP): These architectures can replace the hidden Translator in SimVP to achieve token mixing and channel mixing. Supported models include ViT, Swin-Transformer, MLP-Mixer, ConvMixer, UniFormer, PoolFormer, ConvNeXt, VAN, IncepU (SimVP.V1), gSTA (SimVP.V2), HorNet, and MogaNet.

The Moganet repository provides a Model Zoo containing configuration files, log files, and pretrained models for several supported human pose estimation methods. All pretrained models are evaluated on the following three benchmarks:

• 3DPW
• Human3.6M
• MPI-INF-3DHP

For specific implementation details and configurations of individual baseline methods, refer to the external documentation links provided below.

A HumanData object is a subclass of the Python built-in dict designed to hold single-view, image-based data for a human being. It stores values as numpy.ndarray or Python built-in types. While it does not natively hold torch.Tensor data, it provides a .to() method for easy conversion to CPU or GPU tensors.

Key characteristics:

• Base Structure: Follows a well-defined schema for universal human data.
• Customization: Supports custom features via a misc dictionary or by disabling key strictness.
• Tensor Conversion: Can be converted to a dictionary of tensors using .to().

# Convert all ndarray values to CPU tensors
dict_of_tensor = human_data.to()

# Convert all ndarray values to a specific GPU device
import torch
gpu0_device = torch.device('cuda:0')
dict_of_gpu_tensor = human_data.to(gpu0_device)

Top-down pose estimation methods approach the task in two distinct stages:

1. Human Detection: The system first identifies humans within an image and generates bounding boxes for each person.
2. Pose Estimation: Using the detected bounding boxes, the system performs single-person pose estimation for each individual.

Unlike coordinate-based methods that predict $(x, y)$ values directly, this approach produces heatmaps. These heatmaps represent the spatial likelihood of a keypoint existing at any given pixel location. Common neural network architectures used for this task include stacked hourglass networks and HRNet.

For multi-person visualization, add an extra dimension for num_person to your input tensors/arrays:

• verts: (num_frame, num_person, 6890, 3)
• poses: (num_frame, num_person, 72)
• betas: (num_frame, num_person, 10)
• vibe pred_cam: (num_frame, num_person, 3)

Note that camera parameters K, R, and T are applied per frame and do not require the num_person dimension.