DOCSEARCHbyAlgolia
Submit Docs

DexGraspNet 2.0

repository·main·Indexed 41 days ago

https://github.com/pku-epic/dexgraspnet2

A framework for learning generative dexterous grasping in large-scale synthetic cluttered scenes. It includes tools for grasp data preprocessing, neural network training for grippers and dexterous hands, and performance evaluation in simulation environments like IsaacGym. Key components include the GraspnessSample module for sampling gripper poses via diffusion, ISA, or CVAE architectures, a CollisionChecker for scene penetration analysis, and a SimulationEvaluator for validating grasp success.

Tokens
11.1K
Snippets
29
Records
39
Agent score
51%

What's inside DexGraspNet 2.0

  1. Set up the DexGrasp environment

    main

    To use DexGraspNet 2.0, you need Ubuntu 20.04 and CUDA 11.7. If using different CUDA versions, you must manually adjust torch, pytorch3d, and the CUDA_HOME environment variable.

    Follow these steps to create the conda environment and install dependencies:

    1. Create and activate the environment:
    conda create -n DexGrasp python=3.8
conda activate DexGrasp
    1. Install PyTorch and PyTorch3D:
    conda install pytorch==2.0.1 torchvision==0.15.2 torchaudio==2.0.2 pytorch-cuda=11.7 -c pytorch -c nvidia

# Install PyTorch3D locally
wget https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/pytorch3d/linux-64/pytorch3d-0.7.5-py38_cu117_pyt201.tar.bz2
conda install -y --use-local ./pytorch3d-0.7.5-py38_cu117_pyt201.tar.bz2

    1. Install specialized libraries (TorchSDF, torchprimitivesdf, IsaacGym, MinkowskiEngine, etc.). Note that MinkowskiEngine requires libopenblas-dev and setting CUDA_HOME.

    2. Install Python packages via pip (plotly, open3d, graspnetAPI, wandb, etc.).

    Note on NumPy: Install numpy==1.23.0. If you encounter low AP results in gripper experiments due to graspnetapi bugs, update to numpy==1.24.1 and manually replace np.float with float and np.int with int in the relevant files.

    conda create -n DexGrasp python=3.8
conda activate DexGrasp
conda install pytorch==2.0.1 torchvision==0.15.2 torchaudio==2.0.2 pytorch-cuda=11.7 -c pytorch -c nvidia

  2. Run preprocessing for gripper and dexterous grasping

    main

    Use the following commands to generate or refine grasp datasets.

    Gripper Data: Requires graspnet data. You can download gripper_grasps and gripper_graspness directly to skip this.

    python src/preprocess/extract_gripper_grasp.py --start 0 --end 100
python src/preprocess/refine_dataset.py
python src/preprocess/gripper_graspness.py --start 0 --end 100

    Dexterous Hand Data: You can download dex_graspness_new and dex_grasps_new directly to skip this.

    python src/preprocess/dex_graspness.py --start 0 --end 100
python src/preprocess/dex_graspness.py --start 1000 --end 8500

    Evaluation Preprocessing: To prepare data for evaluation (computing edges and network inputs):

    # Compute edges
python src/preprocess/compute_edges.py --dataset graspnet --start 100 --end 190
python src/preprocess/compute_edges.py --dataset acronym 

# Collect network input
python src/preprocess/compute_network_input_all.py --dataset graspnet --scene_id_start 100 --scene_id_end 190
python src/preprocess/compute_network_input_all.py --dataset acronym 
    python src/preprocess/dex_graspness.py --start 0 --end 100

  3. Evaluate and visualize DexGraspNet models

    main

    Gripper Evaluation:

    python src/eval/eval_gripper.py --ckpt experiments/gripper_ours/ckpt/ckpt_50000.pth --split test_seen

    Dexterous Grasping Evaluation:

    # Predict all categories
python src/eval/predict_dexterous_all_cates.py --ckpt experiments/dex_ours/ckpt/ckpt_50000.pth 

# Evaluate in IsaacGym (requires editing ckpt_path_list in evaluate_dexterous_all.py)
python src/eval/evaluate_dexterous_all_cates.py

# Print simulation results
python src/eval/print_dexterous_result.py --ckpt experiments/dex_ours/ckpt/ckpt_50000.pth

    Visualization: Use the tests/visualize_*.py scripts to inspect results:

    python tests/visualize_scene.py
python tests/visualize_dex_grasp.py
python tests/visualize_gripper_pred.py --ckpt_path=experiments/gripper_ours/ckpt/ckpt_50000.pth
python tests/visualize_dex_pred.py --ckpt_path=experiments/dex_ours/ckpt/ckpt_50000.pth
    python src/eval/eval_gripper.py --ckpt experiments/gripper_ours/ckpt/ckpt_50000.pth --split test_seen

  4. Organize the DexGraspNet 2.0 data directory

    main

    Download the dataset from HuggingFace and unzip the contents into a data/ directory. The directory structure must follow this pattern:

    data/
    meshdata/
    acronym_test_scenes/
    scenes/
    dex_graspness_new/
    dex_grasps_new/
    gripper_graspness/
    gripper_grasps/
    models/ (this should be a link to meshdata)
    data/
    meshdata/
    acronym_test_scenes/
    scenes/
    dex_graspness_new/
    dex_grasps_new/
    gripper_graspness/
    gripper_grasps/
    meshdata/
    models/ (link to meshdata)

  5. Train DexGraspNet models

    main

    Training is performed using src/train.py by specifying an experiment name and a YAML configuration file.

    Gripper Training:

    python src/train.py --exp_name exp_gripper_ours --yaml configs/network/train_gripper_ours.yaml

    Dexterous Hand Training:

    # Ours
python src/train.py --exp_name exp_dex_ours --yaml configs/network/train_dex_ours.yaml

# Isagrasp with graspness
python src/train.py --exp_name exp_dex_isagrasp --yaml configs/network/train_dex_isagrasp.yaml

# Graspcvae with graspness
python src/train.py --exp_name exp_dex_grasptta --yaml configs/network/train_dex_grasptta.yaml
    python src/train.py --exp_name exp_dex_ours --yaml configs/network/train_dex_ours.yaml

  6. Implement or use the Simulator base class

    main

    The Simulator class is an abstract base class used to interface with different physics simulators (e.g., IsaacGymSimulator). To use a simulator, you typically follow a lifecycle of: initializing with a config, registering assets, creating environments, creating actors, and finally preparing the simulation.

    Key lifecycle methods include:

    • register_asset(...): Defines an asset's location and configuration. Returns an asset_info dict containing DOF (Degrees of Freedom) limits.
    • create_env(): Instantiates the simulation environments.
    • create_actor(...): Adds an actor (using a registered asset) to the environments.
    • prepare_sim(): Finalizes the simulator setup before the control loop begins.
    • close(): Shuts down the simulator.
    from src.utils.simulator.simulator import Simulator

# Example lifecycle pattern
sim = simulator_class(config=my_config, num_envs=1024, device_id=0, headless=True)
sim.register_asset(asset_name="robot", asset_root="/path", asset_path="robot.urdf", asset_config={})
sim.create_env()
sim.create_actor(actor_name="agent", asset_name="robot", actor_config={})
sim.prepare_sim()

  7. Configure GraspnessSample architecture types

    main

    The behavior of GraspnessSample is determined by the type key in the configuration dictionary. This determines which sampling mechanism is used:

    config.typeDescription
    graspness_diffusionUses GaussianDiffusion1D. Supports various rotation representations via config.diffusion.rot_type (e.g., svd, sixd, quat, aa, euler).
    graspness_isaUses ConditionalTransform for joint and pose estimation.
    graspness_cvaeUses GraspCVAE for sampling.

    Important Config Keys

    • backbone: Name of the backbone network.
    • feature_dim: Dimension of the extracted features.
    • joint_num: Number of gripper joints.
    • dist_joint: Boolean indicating if joints are modeled as relative displacements.
    • voxel_size: Size of the voxel grid (defaults to 0.005 if not provided).

  8. Add object meshes to the Observer

    main

    Before rendering, you must register meshes into the Observer's vertex buffers using add_object_mesh. This method uploads the mesh geometry to the GPU.

    Parameters:

    • object_code (str): A unique identifier for the object.
    • object_mesh (trimesh.Trimesh): A trimesh object containing the geometry.
    • id (int): An integer ID used in the shader to color the object.
    import trimesh

# Create a mesh
mesh = trimesh.creation.box()

# Register it with the observer
observer.add_object_mesh('my_object', mesh, id=1)

  11. The evaluate_data method signature

    main

    The evaluate_data method evaluates a batch of grasps on a cluttered scene.

    Arguments:

    • object_pose_dict (dict[str, np.ndarray[4, 4]]): A mapping from object codes to their 4x4 pose matrices.
    • grasps (dict[str, np.ndarray]): A dictionary containing grasp properties. Expected keys include:
      • 'translation': np.ndarray of shape [batch_size, 3]
      • 'rotation': np.ndarray of shape [batch_size, 3, 3]
      • 'jointxxx': np.ndarray of shape [batch_size] (representing joint values)

    Returns:

    • successes (np.ndarray[batch_size, np.bool]): An array indicating whether each grasp in the batch was successful.

  12. Calculate robot self-penetration

    main

    The self_penetration method calculates the degree of self-collision within the robot. It uses cal_self_distance to find how much the robot's own surface points are penetrating its own collision volumes, excluding adjacent links (defined by the URDF/meta adjacency mask).

    Arguments:

    • robot_surface_points: A RobotPoints instance containing the sampled surface points.
    • local_translations: dict (link name -> translation, (batch_size, 3))
    • local_rotations: dict (link name -> rotation, (batch_size, 3, 3))
    • dilation_spen: float (self penetration dilation)