SmoothNet

SmoothNet is a plug-and-play temporal refinement network designed to mitigate jitter in human pose estimation from videos. It functions as a dedicated temporal-only refinement layer that can be attached to existing pose estimators (2D, 3D, or SMPL body recovery) to improve temporal smoothness and accuracy, especially in challenging or occluded frames.

Key characteristics:

• Temporal-only model: It learns long-range temporal relations for every joint without relying on noisy inter-joint correlations, making it highly transferable across different estimators and datasets.
• Plug-and-Play: It can be used as a smoothing strategy for various backbones.
• Supported Tasks: 2D human pose estimation, 3D human pose estimation, and SMPL body recovery.
• Integration: SmoothNet is supported as a smoothing strategy in MMPose (v0.25.0+) and MMHuman3D.

To use your own dataset, you must organize it into a specific directory structure based on the body representation (2D, 3D, or SMPL) and the estimator used.

1. Directory Structure

Your dataset folder should follow this pattern:

|-- [your dataset]_[estimator]_[2D/3D/smpl]
    |-- detected
        |-- [your dataset]_[estimator]_[2D/3D/smpl]_test.npz
        |-- [your dataset]_[estimator]_[2D/3D/smpl]_train.npz
    |-- groundtruth
        |-- [your dataset]_gt_[2D/3D/smpl]_test.npz
        |-- [your dataset]_gt_[2D/3D/smpl]_train.npz

2. File Formats (.npz)

Each .npz file must contain imgname and the corresponding pose data. The imgname format is [sequence_name]/[image_name] (or an empty string if unavailable).

• 3D keypoints: Contains keypoints_3d (shape: sequence_length * (keypoints_number * 3)).
• 2D keypoints: Contains keypoints_2d (shape: sequence_length * (keypoints_number * 2)).
• SMPL: Contains pose (shape: sequence_length * 72) and shape (shape: sequence_length * 10).

3. Configuration

If using 3D keypoints, add the root path to your dataset in the following config files:

• lib/core/evaluate_config.py
• lib/core/train_config.py
• lib/core/visualize_config.py

Use the key cfg.DATASET.ROOT_[your dataset]_[estimator]_3D.

SmoothNet is a plug-and-play post-processing network designed to smooth outputs from existing pose estimators. Because it is a temporal-only network without spatial modeling, it is trained on 3D position representations but can be tested on 2D, 3D, or 6D representations.

To accommodate different datasets, backbones, and modalities, three pre-trained models are provided. Choose the model based on your input data and target estimator:

1. AIST-VIBE-3D: Use with configs/aist_vibe_3D.yaml. Best for AIST++ or MuPoTS datasets.
2. 3DPW-SPIN-3D: Use with configs/pw3d_spin_3D.yaml. Best for 3DPW or MPI-INF-3DHP datasets.
3. H36M-FCN-3D: Use with configs/h36m_fcn_3D.yaml. Best for Human3.6M datasets.

Note: SmoothNet consistently achieves lower acceleration (Accel) values compared to backbone estimators. When using SmoothNet with SOTA single-frame estimators (like PARE), you can achieve significantly improved performance.

# Example configuration selection (refer to specific yaml files in /configs)
# For Human3.6M tasks:
config: configs/h36m_fcn_3D.yaml

SmoothNet evaluates pose refinement performance using several error metrics, typically expressed in millimeters (multiplied by a m2mm conversion factor). The metrics compare the input (noisy) poses against the denoised (SmoothNet) poses and, optionally, a baseline filter.

Key Error Metrics

• MPJPE (Mean Per Joint Position Error): Measures the average Euclidean distance between predicted and ground truth joints.
• PAMPJPE (Procrustes-Aligned MPJPE): Measures error after Procrustes alignment.
• MPVPE (Mean Per Vertex Position Error): Measures error at the vertex level.
• Accel (Acceleration Error): Measures the error in joint acceleration, used to assess temporal smoothness.
• Top1 Accuracy: Measures the percentage of frames where error is below a certain threshold (e.g., input_mpjpe_top1).

Performance Metrics

• Inference Time: Reported in microseconds per frame (10e6 * time / num_frames).
• Improvement: The delta between the denoised error and the input error (e.g., improvement_mpjpe).

If cfg.EVALUATE.TRADITION is non-empty, the evaluation includes comparison against a filter baseline.

Run train_smoothnet.py to start training. You can train on multiple datasets, estimators, or body representations simultaneously by separating values with commas.

Arguments:

• --cfg: Path to the configuration file.
• --dataset_name: Name of the dataset (or comma-separated list).
• --estimator: The backbone estimator used (or comma-separated list).
• --body_representation: smpl, 3D, or 2D (or comma-separated list).
• --slide_window_size: The size of the sliding window.

Example (Single Dataset): Train on Human3.6M using FCN estimator with 3D representation and a window size of 8:

python train_smoothnet.py --cfg configs/h36m_fcn_3D.yaml --dataset_name h36m --estimator fcn --body_representation 3D --slide_window_size 8

Example (Multiple Datasets): Train on AIST++ (VIBE, 3D) and 3DPW (SPIN, 3D) simultaneously:

python train_smoothnet.py --cfg configs/h36m_fcn_3D.yaml --dataset_name aist,pw3d --estimator vibe,spin --body_representation 3D,3D --slide_window_size 8

Use visualize_smoothnet.py to generate video visualizations of processed poses. Note that this demo is based on offline processed detected poses for specific datasets (AIST++, Human3.6M, 3DPW). For arbitrary video inference, use MMHuman3D.

Arguments:

• --cfg: Path to the configuration file.
• --checkpoint: Path to the pretrained checkpoint.
• --dataset_name: Name of the dataset.
• --estimator: The backbone estimator used.
• --body_representation: smpl, 3D, or 2D.
• --slide_window_size: The size of the sliding window.
• --visualize_video_id: The ID of the sequence to visualize.
• --output_video_path: Path where the output video will be saved.

Example: Visualize the second sequence of 3DPW - SPIN - 3D and save to ./visualize:

python visualize_smoothnet.py --cfg configs/pw3d_spin_3D.yaml --checkpoint data/checkpoints/pw3d_spin_3D/checkpoint_8.pth.tar --dataset_name pw3d --estimator spin --body_representation 3D --slide_window_size 32 --visualize_video_id 2 --output_video_path ./visualize

Run eval_smoothnet.py to evaluate the model and compare it against traditional filters.

Arguments:

• --cfg: Path to the configuration file.
• --checkpoint: Path to the pretrained checkpoint.
• --dataset_name: Name of the dataset (or comma-separated list).
• --estimator: The backbone estimator used (or comma-separated list).
• --body_representation: smpl, 3D, or 2D (or comma-separated list).
• --slide_window_size: The size of the sliding window.
• --tradition: Traditional filter to compare against (savgol, oneeuro, or gaus1d).

Example: Evaluate MPI-INF-3DHP using both TCMR and VIBE estimators (3D) using a model trained on 3DPW-SPIN-3D, comparing against the oneeuro filter:

python eval_smoothnet.py --cfg configs/pw3d_spin_3D.yaml --checkpoint data/checkpoints/pw3d_spin_3D/checkpoint_8.pth.tar --dataset_name mpiinf3dhp,mpiinf3dhp --estimator tcmr,vibe --body_representation 3D,3D --slide_window_size 8 --tradition oneeuro