VideoMimic: Humanoid Robot Control via Visual Imitation

VideoMimic Real-to-Sim is a vision pipeline designed to reconstruct 3D environments and human motion from single-camera (monocular) RGB videos. The system extracts human poses in world coordinates and retargets that motion to humanoid robots (such as the G1) for imitation learning.

Key Capabilities:

• 3D Reconstruction: Generates dense 3D scene geometry (pointclouds and meshes) from video.
• Human Motion: Extracts accurate 3D human poses aligned with a scaled environment.
• Multi-Human Support: Tracks and reconstructs multiple people simultaneously using detection and re-identification.
• Robot Retargeting: Performs collision and contact-aware human-to-humanoid motion retargeting.
• Simulation-Ready: Produces datasets containing environment meshes and robot configurations ready for simulation.

Sequential processing in the VideoMimic Real-to-Sim pipeline processes multiple videos one after another rather than in parallel. This approach is specifically designed to:

• Amortize model loading time: Models and checkpoints are loaded once and reused across all videos.
• Share JAX compilation: JAX optimization code is compiled once (during the first video), making subsequent videos significantly faster.
• Maintain stable memory usage: Prevents GPU memory overflow by processing one video at a time.
• Build datasets efficiently: It is the recommended method for creating large training datasets for policy learning, as it reduces the per-video overhead from minutes to seconds after the first video.

Note: This is NOT batch/parallel processing. It is a sequential loop designed for efficiency and stability.

The standard workflow for achieving motion control using reinforcement learning in this project follows these four stages:

1. Train: Use the Gym simulation environment to train a policy that maximizes designed rewards. Note: Real-time visualization is not recommended during this stage as it reduces training efficiency.
2. Play: Use the Play command to verify the trained policy and ensure its behavior meets expectations.
3. Sim2Sim: Deploy the policy trained in Gym to different simulators to verify that the policy is robust and not overfitted to Gym-specific characteristics.
4. Sim2Real: Deploy the policy onto a physical robot to achieve actual motion control.

By default, the pipeline only processes the largest person (by average bounding box area). To process multiple humans, use the --multihuman flag along with --top-k N to specify the number of largest humans to process.

Warning: Memory usage scales with the number of humans processed.

# Example for Stage 0
python stage0_preprocessing/vitpose_2d_poses.py ... --multihuman --top-k 3

# Example for Stage 2
python stage2_optimization/megahunter_optimization.py ... --multihuman --top-k 3

The VideoMimic Real-to-Sim pipeline requires two separate Conda environments due to conflicting dependency requirements (specifically regarding CUDA versions and xformers).

• vm1rs (Main Environment): Uses Python 3.12 and CUDA 12.4+. It is used for human preprocessing, optimization, and motion retargeting.
• vm1reocn (Reconstruction Environment): Uses Python 3.10 and CUDA 11.8. It is used for MegaSam reconstruction, NKSR meshification, and GeoCalib operations.

Mental Model: Always activate the specific environment required for your current task. Use vm1rs for most operations and vm1reocn only when performing reconstruction or postprocessing.

The pipeline ensures consistent person tracking through the following mechanism:

1. Detection: SAM2 detects all humans and calculates their bounding box area per frame.
2. Ranking: An average bounding box area is computed for each person across all frames. Humans are ranked by this average area (largest first).
3. Metadata: This ranking is stored in meta_data.json (under sorted_by_avg_area) and per-frame JSON files (under area_ranking).
4. Selection: Subsequent stages (ViTPose, VIMO, MegaHunter) use this ranking to select the top-k humans, ensuring the same person ID is tracked throughout the pipeline.

SAM2 meta_data.json structure:

{
  "all_instance_ids": [1, 2, 3],
  "sorted_by_avg_area": [2, 1, 3],
  "avg_areas": {
    "1": 15234.5,
    "2": 18956.2,
    "3": 12456.8
  },
  "frame_counts": {
    "1": 95,
    "2": 100,
    "3": 87
  }
}

Navigate to the sloper4d_eval_script/ directory to run the evaluation. You must provide the path to the predicted SMPL data, the output directory, and the ground truth .pkl file path.

Arguments:

• --pred_smpl_path: Path to the .npy file containing predicted SMPL data.
• --output_dir: Directory where evaluation results will be saved.
• --gt_pkl_path: Path to the ground truth .pkl labels file.

cd sloper4d_eval_script/

# Example evaluation for sequence 008
python run_eval_sloper4d.py --pred_smpl_path ../results/sloper4d_seq008/hps_combined_track_0.npy --output_dir ../results/sloper4d_seq008 --gt_pkl_path ../demo_data/sloper4d/seq008_running_001/seq008_running_001_labels.pkl