DOCSEARCHbyAlgolia
Submit Docs

ManiSkill 3 Documentation

repository·main·Indexed 25 days ago

https://github.com/mani-skill/maniskill

An open-source framework for robot simulation and training powered by SAPIEN, optimized for manipulation skills. It features high-throughput GPU-parallelized simulation and visual data collection. The framework includes baseline implementations for online Reinforcement Learning (RL) and Imitation Learning, such as Action Chunking with Transformers (ACT), Behavior Cloning (BC), Diffusion Policy, and SAC-MoE.

Tokens
60.9K
Snippets
133
Records
307
Agent score
85%

What's inside ManiSkill 3

  1. Overview of the SO100 (Koch v1.1) robot arm

    main

    The SO100 is a 6DOF (6 Degrees of Freedom) Koch v1.1 robotic arm. The ManiSkill implementation includes several refinements over the original assets:

    • Fixed joint limits and axes: Adjusted to accurately reflect real-world physical behavior.
    • Improved collision geometry: Collision meshes have been decomposed into separate convex meshes for better simulation accuracy.
    • Semantic remodeling: Certain links have been remodeled to separate the motor components from the 3D printer parts, including semantic annotations for better task interaction.

  3. Overview of ManiSkill 3 features

    main

    ManiSkill is an open-source robot simulation and training framework powered by SAPIEN, optimized for manipulation tasks. Key capabilities include:

    • High-throughput Data Collection: GPU parallelized visual data collection (RGBD + Segmentation) capable of 30,000+ FPS on high-end GPUs (e.g., RTX 4090).
    • Parallelized Simulation: Supports GPU parallelized simulation for high-throughput state-based data collection and heterogeneous simulation (where each parallel environment can have unique scenes/objects).
    • Diverse Robot & Task Support: Covers various embodiments (humanoids, mobile manipulators, single-arm robots) and tasks (table-top, drawing/cleaning, dexterous manipulation).
    • Flexible Task API: An object-oriented API that abstracts complex GPU memory management.
    • Sim2Real & Real2Sim: Tools for evaluating real-world policies 100x faster via GPU simulation and deploying simulation-trained policies to real hardware.
    • Integrated Baselines: Support for Reinforcement Learning (PPO, SAC, TD-MPC2), Imitation Learning (Behavior Cloning, Diffusion Policy), and Vision Language Action (VLA) models (Octo, RDT-1B, RT-x).

  4. Overview of Inspire Hand assets

    main
    The inspire_hand assets folder contains models for the dexterous Inspire Robotics Hands. Currently, the supported models are the RH56DFX-2L/R with a wrist model. The assets include a floating base version of the hand. The URDF has been modified from the original Inspire Robotics version to tune joint axes/signs, joint limits, and mimic tag dependencies to ensure compatibility with PhysX simulation.

  5. Overview of Control Tasks in ManiSkill

    main

    ManiSkill provides a suite of classic control tasks designed for reinforcement learning. These tasks involve controlling a robot to reach specific states or perform motions (e.g., walking, balancing, standing). Unlike the DM Control suite, ManiSkill implements these with GPU-parallelized simulation and rendering for high-throughput training.

    Key features of these tasks include:

    • Dense Rewards: Many tasks provide dense reward signals to facilitate learning.
    • Randomizations: Tasks include various randomizations (e.g., joint noise, initial poses, or rotations) to improve robustness.
    • Success/Fail Conditions: Specific criteria for task completion or episode termination.

  6. Overview of Environment Utilities

    main
    The mani_skill.envs.utils module provides a suite of utilities designed for building robotic simulation scenes and designing reward functions or success conditions. These utilities are implemented to be dual-compatible, supporting both numpy (for CPU-based simulation) and pytorch (for GPU-based simulation).

  7. Overview of SO100 robot assets

    main

    The SO100 is a 6DOF (six degrees of freedom) robotic arm asset available in ManiSkill. The assets are derived from the original SO-ARM100 project under an Apache 2.0 License.

    ManiSkill provides a modified version of these assets with the following improvements for simulation accuracy:

    • Joint limits are fixed to reflect real-world behavior.
    • Joint tags are set to revolute (instead of continuous) to correctly permit joint limits.
    • Joint directions and orientations are aligned with the real robot's joints.
    • Link names have been cleaned (spaces removed).
    • Gripper link collision meshes have been manually decomposed into simpler meshes for better simulation performance.

  9. Overview of Drawing Tasks

    main

    Drawing tasks in ManiSkill involve controlling a robot to draw specific shapes or patterns on a canvas using a stick. These tasks vary in complexity, reward structure, and success criteria.

    Available drawing environments include:

    • TableTopFreeDraw-v1: A reference environment with a white canvas and a red-drawing stick. It has no success conditions, no dense rewards, and no sparse rewards. Max steps: 1000.
    • DrawSVG-v1: The robot must draw a specified SVG path. It uses sparse rewards but no dense rewards. Max steps: 500.
    • DrawTriangle-v1: The robot must draw a goal triangle. It uses sparse rewards but no dense rewards. Max steps: 300.

  10. Overview of ManiSkill

    main

    ManiSkill is an open-source framework for robot simulation and training powered by SAPIEN, specifically optimized for manipulation skills. It provides high-throughput GPU parallelized simulation and visual data collection (RGBD + Segmentation), supporting heterogeneous simulation where each parallel environment can have unique scenes and objects.

    Key capabilities include:

    • High-throughput data collection: Up to 30,000+ FPS for RGBD + Segmentation on high-end GPUs.
    • Diverse Robot Support: Humanoids, mobile manipulators, and single-arm robots.
    • Task Variety: Table-top manipulation, drawing/cleaning, and dexterous manipulation.
    • Sim2Real/Real2Sim: Tools for scalable evaluation of real-world policies and deploying simulation-trained policies to real hardware.
    • Learning Baselines: Support for Reinforcement Learning (PPO, SAC, TD-MPC2), Imitation Learning (Behavior Cloning, Diffusion Policy), and Vision Language Action (VLA) models (Octo, RDT-1B, RT-x).

  11. Overview of ManiSkill Utilities

    main

    ManiSkill provides a suite of utility modules designed to assist with task and scene construction, training, and evaluation. The core modules include:

    • building/: Utilities for building tasks and scenes, including asset/articulation loading from datasets and randomization functions for task initialization.
    • scene_builder/: Tools for the SceneBuilder class, including prebuilt builders for tabletop scenes, ReplicaCAD, and AI2THOR (via HSSD dataset).
    • geometry/: Functions for geometric operations, such as sampling primitive shapes and retrieving axis-aligned bounding boxes (AABB) for articulations or actors.
    • wrappers/: Classes for extending environment functionality, such as recording videos/episodes, modifying observation spaces, and adapting the API for compatibility with RL libraries like Stable Baselines 3.
    • visualization/: Tools for visualizing ManiSkill environments.
    • gym_utils.py: Utilities specifically for working with the Gymnasium/Gym API.
    • common.py: General-purpose utilities frequently used for reward functions, success evaluation, and manipulating nested dictionaries.

  12. Understand GPU Simulation and Sub-scenes

    main

    ManiSkill uses NVIDIA PhysX to perform parallelized physics simulation on the GPU. Instead of simulating tasks sequentially, ManiSkill places all actors and articulations into a single PhysX scene, where each task is assigned its own workspace called a sub-scene.

    Key Concepts:

    • Sub-scenes: Small workspaces within the main PhysX scene. Data (like actor poses) is automatically pre-processed to be relative to the center of the sub-scene rather than the global PhysX scene.
    • Spacing: The distance between sub-scenes is controlled by the sim_config.spacing value.
    • Collision Warning: If objects in one sub-scene move beyond its workspace (e.g., due to a low spacing value), they may collide with objects in adjacent sub-scenes. This is a common source of bugs in large-scale or outdoor simulations.