GraspGen: A Diffusion-based Framework for 6-DOF Grasping

GraspGen is a modular framework for diffusion-based 6-DOF robotic grasp generation. It is designed to scale across:

• Embodiments: Supports 3 gripper types (Franka-Panda, Robotiq-2f-140, and suction cup).
• Observability: Robust to both partial and complete 3D point clouds.
• Complexity: Handles both single-object grasping and cluttered scenes.

Note on Grippers: This repository specifically supports the three grippers mentioned above. For other grippers (e.g., DROID, robotiq-2f-85), use the GraspGen-X extension.

The GraspGen MCP Server acts as a lightweight bridge between LLMs (like Claude or Cursor) and the GraspGen inference engine. It does not require CUDA or model weights itself; instead, it connects to a running GraspGen ZMQ inference server via TCP.

Architecture Flow:

1. LLM/AI Agent calls tools via MCP (stdio).
2. MCP Server receives the call and forwards requests to the GraspGen Server via ZMQ (tcp).
3. GraspGen Server (running on GPU with model loaded) processes the point cloud and returns grasps to the MCP server, which then returns them to the LLM.

GraspGen requires a dataset organized into two main components: Splits and a Grasp Dataset.

1. Splits

Objects must be divided into training and validation/test sets using *.txt files. Each line in these files should contain either:

• A uuid (if using Objaverse).
• A relative path to the object mesh file (.obj or .stl), relative to the root of the object dataset.

If an object is used for both training and testing, it must be included in both text files.

2. Grasp Dataset

Grasp data must be stored in a separate directory containing *.json files. Each JSON file represents the grasp data for a specific object.

GraspGen can be deployed as a standalone ZMQ server. This allows any application (Python, C++, Rust, etc.) to request 6-DOF grasp predictions over a network without needing a GPU or CUDA installed on the client machine.

Architecture:

• Server: Loads a specific gripper model (e.g., Franka Panda, Robotiq 2F-140) and listens on a ZMQ REP socket. It handles the GPU-intensive inference.
• Client: Sends point clouds (as numpy arrays serialized with msgpack) to the server and receives back 6-DOF grasp poses and confidence scores.
• Communication: Uses ZMQ (TCP) with msgpack serialization.

If your gripper is similar to an existing one, you may be able to retarget the model instead of full re-training:

Antipodal Grippers (e.g., Franka/Robotiq): If the stroke length/width is similar, you can apply a Z-axis offset to align the base link frames:

import trimesh.transformations as tra
new_grasp = grasp @ tra.translation_matrix([0, 0, -Z_OFFSET])

Suction Grippers: If using a single-cup suction gripper, you can retarget the 30mm suction model by rescaling the object point cloud/mesh input:

import trimesh.transformations as tra
mat = tra.scale_matrix(r / 0.030)  # where r is your suction cup radius

import trimesh.transformations as tra
# For antipodal offset
new_grasp = grasp @ tra.translation_matrix([0, 0, -Z_OFFSET])

# For suction rescaling
mat = tra.scale_matrix(r / 0.030)

Visualize grasp predictions from your trained model on an object mesh using an interactive 3D visualization.

Parameters:

• --mesh_file: Path to the object mesh file.
• --mesh_scale: Scale factor for the object mesh (default: 1.0).
• --gripper_config: Path to the config file created in the checkpoint generation step (e.g., /results/tutorial/models/tutorial_model_config.yaml).

Visualization Features:

• Interactive 3D view of the object mesh.
• Generated grasp poses overlaid on the object.
• Color-coded grasp quality scores.
• Highlighting of the best grasps.

cd /code && python scripts/demo_object_mesh.py \
    --mesh_file /models/sample_data/meshes/box.obj \
    --mesh_scale 1.0 \
    --gripper_config /results/tutorial/models/tutorial_model_config.yaml

If you prefer not to use Docker, you must manually manage the CUDA environment and dependencies.

1. Start the Server

Ensure you have an environment with CUDA and all GraspGen dependencies installed (e.g., via Conda):

conda activate GraspGen
# or source .venv/bin/activate

# Install serving dependencies
pip install pyzmq msgpack msgpack-numpy

# Start the server
python client-server/graspgen_server.py \
    --gripper_config /path/to/GraspGenModels/checkpoints/graspgen_robotiq_2f_140.yml \
    --port 5556

2. Run the Client

Follow the same uv setup steps as the Docker instructions to create a lightweight client environment without CUDA requirements.

# Start the server:
python client-server/graspgen_server.py \
    --gripper_config /path/to/GraspGenModels/checkpoints/graspgen_robotiq_2f_140.yml \
    --port 5556

Use Docker for a complete environment suitable for both training and inference. This is the recommended method for training tasks.

git clone https://github.com/NVlabs/GraspGen.git && cd GraspGen
bash docker/build.sh

git clone https://github.com/NVlabs/GraspGen.git && cd GraspGen
bash docker/build.sh # This will take a while

While GraspGen is designed for zero-shot generalization, you can fine-tune the model on specific object/grasp datasets. To do this:

1. Pass the existing pretrained checkpoint to the train.checkpoint argument in the training script.
2. Update the paths to point to your new grasp/object dataset following the docs/GRASP_DATASET_FORMAT.md convention.