SAM 3D Body

To prepare the AI Challenger Keypoint Dataset for use with SAM-3D-Body Data, you must organize the images into a specific directory structure and set an environment variable for easy referencing.

1. Set the AIC_IMG_DIR environment variable to the path where your images are stored.
2. Download the AI Challenger Keypoint images.
3. Ensure the directory structure under $AIC_IMG_DIR matches the required format (containing test/images and train/images subdirectories).

export AIC_IMG_DIR=/path/to/ai/challenger/images

To prepare the MPII Dataset for use with SAM-3D-Body Data, you must set up the directory structure and download the required assets.

1. Set the COCO_IMG_DIR environment variable to point to your image directory.
2. Download the mpii_human_pose_v1.tar.gz archive.
3. Extract the archive into your target directory ($MPII_IMG_DIR).

The resulting directory structure must be:

$MPII_IMG_DIR
└── images

# Set environment variable
export COCO_IMG_DIR=/path/to/coco/images

# Download and extract the dataset
wget https://datasets.d2.mpi-inf.mpg.de/andriluka14cvpr/mpii_human_pose_v1.tar.gz
tar zxvf mpii_human_pose_v1.tar.gz -C $MPII_IMG_DIR 
rm mpii_human_pose_v1.tar.gz

To prepare the COCO dataset for use with SAM-3D-Body, you must set the COCO_IMG_DIR environment variable to your target directory, download the COCO 2014 image zips (train, val, and test), and extract them so that the directory structure contains test2014, train2014, and val2014 directly under the path specified by $COCO_IMG_DIR.

# 1. Set the environment variable
export COCO_IMG_DIR=/path/to/coco/images

# 2. Download and extract the dataset
cd $COCO_IMG_DIR
wget http://images.cocodataset.org/zips/train2014.zip
wget http://images.cocodataset.org/zips/val2014.zip
wget http://images.cocodataset.org/zips/test2014.zip
unzip train2014.zip 
unzip val2014.zip 
unzip test2014.zip 
rm train2014.zip val2014.zip test2014.zip 

Use scripts/download.py to download annotations from HuggingFace. Note that you must first request access on the SAM 3D Body Hugging Face repo and be authenticated.

To download all available splits, use the --save_dir flag. To download a specific split (e.g., coco_train, sa1b_train), use the --splits flag.

Set the following environment variables to define where annotations are stored and where WebDataset archives should be outputted:

• SAM3D_BODY_ANN_DIR: Path to the SAM 3D Body annotations directory.
• SAM3D_BODY_WDS_DIR: Path to the directory for WebDataset output.

export SAM3D_BODY_ANN_DIR=/path/to/sam3d/body/annotations
export SAM3D_BODY_WDS_DIR=/path/to/sam3d/body/webdatasets

If your workflow requires MoGe, install it directly from the Microsoft GitHub repository.

pip install git+https://github.com/microsoft/MoGe.git

Model checkpoints are hosted on Hugging Face. You must request access on the specific repository page before you can download them. Once access is granted, ensure you are authenticated via Hugging Face to download the weights.

Available models:

• facebook/sam-3d-body-dinov3
• facebook/sam-3d-body-vith

Note: Access may be rejected if you are in a comprehensively sanctioned jurisdiction.

Create and configure a conda environment specifically for dataset preparation. This includes installing PyTorch with CUDA support and several dependencies for processing 3D data, images, and datasets.

conda create --name sam_3d_body_data python=3.9 -y
conda activate sam_3d_body_data
conda install pytorch=2.4.0 torchvision torchaudio pytorch-cuda=11.8 -c pytorch -c nvidia -y
pip install hdbscan pyntcloud==0.3.1 pykalman torchgeometry colour==0.1.5 flask==3.1.1 trimesh==4.7.3
pip install pandas==2.0.3 numpy==1.26 pycolmap==0.3.0 yacs==0.1.8 projectaria-tools==1.3.3 opencv-python==4.7.0.72
pip install datasets huggingface_hub webdataset pycocotools

To install SAM 3D Body, first create and activate a dedicated Conda environment with Python 3.11. You must also install PyTorch following the official PyTorch instructions before proceeding with other dependencies.

conda create -n sam_3d_body python=3.11 -y
conda activate sam_3d_body