GigaPose: Fast and Robust 6D Novel Object Pose Estimation

To estimate object poses without existing CAD models, you can reconstruct 3D models from a single image using Wonder3D and then use the GigaPose pipeline.

Workflow Steps

1. Select Reference Image: Choose an input reference image for each object.
2. Crop Image: Crop the input image and save the cropping matrix (required to recover the correct scale later).
3. Reconstruct 3D Model: Use Wonder3D to reconstruct the model. Note that outputs are in the input image's coordinate frame and use an orthographic camera.
4. Recover Scale: Use the cropping matrix from Step 2 to restore the correct scale of the reconstructed model.
5. Estimate Pose: Run the GigaPose pipeline on the reconstructed models.

Reproducing Results on LM-O Dataset

If you want to reproduce the results shown in the paper for the LM-O dataset, follow these steps:

1. Revert to specific commit (if working on this specific setting):

   git checkout 388e8bddd8a5443e284a7f70ad103d03f3f461c5

2. Download CNOS detections:

   python -m src.scripts.download_cnos_bop23

3. Setup the reconstructed dataset:

   mkdir $ROOT_DIR/datasets/lmoWonder3d
   wget https://huggingface.co/datasets/nv-nguyen/gigaPose/resolve/main/lmoWonder3d.zip -P $ROOT_DIR/datasets/lmoWonder3d
   unzip -j $ROOT_DIR/datasets/lmoWonder3d/lmoWonder3d.zip -d $ROOT_DIR/datasets/lmoWonder3d/models -x "*/._*"
   
   # Create symlinks to existing LM-O data
   ln -s $ROOT_DIR/datasets/lmo/test $ROOT_DIR/datasets/lmoWonder3d/test
   ln -s $ROOT_DIR/datasets/lmo/test_targets_bop19.json $ROOT_DIR/datasets/lmoWonder3d/test_targets_bop19.json

4. Onboard the dataset (render templates):

   python -m src.scripts.render_custom_templates custom_dataset_name=lmoWonder3d

5. Run Estimation:

   python test.py test_dataset_name=lmoWonder3d run_id=$NAME_RUN
   python refine.py test_dataset_name=lmoWonder3d run_id=$NAME_RUN

# Onboarding by rendering templates from reconstructed 3D models
python -m src.scripts.render_custom_templates custom_dataset_name=lmoWonder3d

# Test the dataset
python test.py test_dataset_name=lmoWonder3d run_id=$NAME_RUN
python refine.py test_dataset_name=lmoWonder3d run_id=$NAME_RUN

To test GigaPose on BOP 2024 datasets, you can perform coarse prediction followed by refinement.

1. Coarse Prediction

• 6D Detection task: Use test_setting=detection.
• 6D Localization task: Use test_setting=localization (available for core19 datasets).

2. Refinement

Run refine.py with the same test_dataset_name and test_setting used in the coarse step.

3. Evaluation

Use the bop_toolkit to evaluate your results. You must export INPUT_DIR (path to prediction file) and FILE_NAME (name of the prediction file).

# 1. Running coarse prediction (6D detection)
python test.py test_dataset_name=hope run_id=$NAME_RUN test_setting=detection

# 2. Running refinement
python refine.py test_dataset_name=hope run_id=$NAME_RUN test_setting=detection

# 3. Evaluating with BOP toolkit
export INPUT_DIR=DIR_TO_YOUR_PREDICTION_FILE
export FILE_NAME=NAME_PREDICTION_FILE
cd $ROOT_DIR_OF_TOOLKIT
python scripts/eval_bop24_pose.py --results_path $INPUT_DIR --eval_path $INPUT_DIR --result_filenames=$FILE_NAME

For BOP 2023 datasets, you can run individual datasets or all 7 core datasets at once.

Individual Dataset

Run test.py for coarse prediction and refine.py for refinement.

All Core Datasets

Run the batch evaluation script: python -m src.scripts.eval_bop.

Evaluation with BOP toolkit

Use bop_toolkit/scripts/eval_bop19_pose.py with the --renderer_type=vispy flag.

# 1. Running coarse prediction
python test.py test_dataset_name=lmo run_id=$NAME_RUN

# 2. Running refinement
python refine.py test_dataset_name=lmo run_id=$NAME_RUN

# 3. Running all steps for all 7 core datasets
python -m src.scripts.eval_bop

# 4. Evaluating with BOP toolkit
export INPUT_DIR=DIR_TO_YOUR_PREDICTION_FILE
export FILE_NAME=NAME_PREDICTION_FILE
python bop_toolkit/scripts/eval_bop19_pose.py --renderer_type=vispy --results_path $INPUT_DIR --eval_path $INPUT_DIR --result_filenames=$FILE_NAME

The megapose.lib3d.symmetries module provides data structures to represent different types of object symmetries used in 3D pose estimation.

• DiscreteSymmetry: Represents symmetries that occur at specific, fixed poses. It is defined by a pose attribute, which is a (4, 4) homogeneous transformation matrix.
• ContinuousSymmetry: Represents symmetries that occur along a continuous axis (e.g., a cylindrical object). It is defined by an offset and an axis (a unit vector representing the rotation axis).

These classes are used as inputs to make_symmetries_poses to generate a complete set of symmetry transformation matrices.

from megapose.lib3d.symmetries import DiscreteSymmetry, ContinuousSymmetry
import numpy as np

# Example Discrete Symmetry (a 180-degree rotation)
discrete_sym = DiscreteSymmetry(pose=np.eye(4))

# Example Continuous Symmetry (rotation around Z-axis)
continuous_sym = ContinuousSymmetry(offset=np.array([0.0, 0.0, 0.0]), axis=np.array([0.0, 0.0, 1.0]))

MegaPose uses PandasTensorCollection (aliased as PoseEstimatesType and DetectionsType) to manage pose estimates and object detections. These collections combine a pandas.DataFrame for metadata (infos) and a dictionary of torch.Tensor objects for geometric data (tensors).

DetectionsType (Detections)

Used to provide object locations to the model.

• infos fields:
  • 'label': str, the object label.
  • 'batch_im_id': int, image index in the batch.
  • 'instance_id': int, unique ID for the instance in the scene.
  • 'score': Optional[float], detection confidence.
  • 'scene_id': Optional[str], dataset scene identifier.
  • 'view_id': Optional[str], dataset view identifier.
• tensors fields:
  • 'bboxes': [B, 4] integer tensor representing bounding boxes.

PoseEstimatesType (Pose Estimates)

Used to store the output of the pose estimation.

• infos fields:
  • Includes all fields from DetectionsType plus:
  • 'hypothesis_id': int, index of the specific pose hypothesis.
  • 'pose_score': Optional[float], score from the coarse module.
  • 'coarse_logit': Optional[float].
  • 'refiner_batch_idx': Optional[int], index for refiner outputs.
  • 'refiner_instance_idx': Optional[int], index for refiner crops.
• tensors fields:
  • 'poses': [B, 4, 4] float32 tensor of transformation matrices.