SimplerEnv Documentation

To add a new robot, follow these three main phases: asset integration, agent implementation, and system identification.

1. Asset Integration

• Place your robot URDF in ManiSkill2_real2sim/mani_skill2_real2sim/assets/descriptions.
• Use a test script in tools/robot_object_visualization to verify the robot in the SAPIEN viewer.
• Visual Matching Tip: To reduce the real-to-sim gap, consider recoloring the robot (e.g., using GIMP) to match real-world observations.
• Collision Issues: If links behave strangely due to mesh penetration, either remesh in Blender or ignore collisions between specific links (see agents/robots/widowx.py for implementation patterns).

2. Agent Implementation

• Create a new robot agent in ManiSkill2_real2sim/mani_skill2_real2sim/agents/robots by inheriting from ManiSkill2_real2sim/mani_skill2_real2sim/agents/base_agent.py.
• Controllers: Add configurations for the arm and gripper in ManiSkill2_real2sim/mani_skill2_real2sim/agents/controllers/. Use base_controller.py as a reference.
• Key Methods to Implement/Override:
  • set_action and before_simulation_step in base_agent.py.
  • step_action in ManiSkill2_real2sim/mani_skill2_real2sim/envs/sapien_env.py.
• Cameras: Add cameras to the robot. Poses are relative to the link specified by actor_uid. SAPIEN uses the ROS convention: x forward, y left, z up. It is highly recommended to calibrate and input intrinsic parameters to minimize the evaluation gap.

3. System Identification (SysID)

• Dataset Creation: Modify tools/sysid/prepare_sysid_dataset.py to create a subset of trajectories from existing datasets (e.g., TensorFlow) using the required pickle file format.
• Running SysID: Use tools/sysid/sysid.py (which uses simulated annealing) to find optimal stiffness and damping parameters.
• Analysis: Use tools/sysid/analyze_sysid_results.py to examine logs. Iteratively refine parameters by reducing the search range based on the best results from previous rounds.

To implement a new environment, follow this workflow:

1. Backgrounds: Add custom scene backgrounds to ManiSkill2_real2sim/data/hab2_bench_assets/stages. Note that backgrounds follow the Habitat convention (y-axis up).
2. Visual Matching (Greenscreen): If using this setup, add the inpainted background image (with robot/objects removed) to ManiSkill2_real2sim/data/real_inpainting.
3. Environment Implementation: Add the new environment class to ManiSkill2_real2sim/mani_skill2_real2sim/envs/custom_scenes.
   • Lifecycle: The reset function should assess if reconfiguration is needed (calling reconfigure in sapien_env.py to load assets) and then call initialize_episode.
   • Metadata: Populate or create JSON metadata files for object assets in ManiSkill2_real2sim/data/custom/info_*.json.
4. Ray Tracing: For Drawer tasks, enable ray-tracing via env = gym.make(**kwargs, shadow_dir='rt') to ensure proper shadow/contrast for depth inference. Note that this is slow on non-RTX GPUs.
5. Testing: Use the interactive script ManiSkill2_real2sim/mani_skill2_real2sim/examples/demo_manual_control_custom_envs.py to manually control the robot and verify visual alignment.
6. Policy Integration:
   • Update simpler_env/utils/env/env_builder.py and simpler_env/utils/env/observation_utils.py to support your new environment.
   • If implementing new metrics, modify tools/calc_metrics.py.

To add new objects for simulation, follow these steps:

1. Storage: Place assets in ManiSkill2_real2sim/data/custom.
2. Asset Components: Each object requires:
   • Visual Mesh: A textured.dae (with .png textures) or a single textured.glb.
   • Collision Mesh: A collision.obj. This mesh should be watertight and convex.
3. Processing:
   • If the collision mesh is not convex, process it using tools/coacd_process_mesh.py.
   • Use tools/robot_object_visualization/test_object.py to visualize the object and toggle the collision mesh visibility.
4. Optimization:
   • For the visual matching setup, use GeTex to bake real-world textures onto simulation assets.
   • Geometry Tips: Collision meshes don't need to match visual geometry perfectly. You can use simple geometric shapes (like frustums) or make the bottom of objects (like cans or bottles) flat to prevent wobbling or rolling when placed on surfaces.

Blender Exporting Guide

When exporting from Blender to SAPIEN (which uses x forward, z up):

• Axis Convention: If your Blender model is y forward, z up, export using the x forward, z up option in Blender. This ensures the x-axis becomes the longest side in SAPIEN.
• Transformations: Clear parents (alt+p > clear parent), move to origin (object > set origin > geometry to origin), and apply all transforms (object > apply > all transforms) before exporting.
• Collision Mesh Refinement: In Blender, you can make a collision mesh a convex hull (edit mode > mesh > convex hull) or use the decimate modifier to simplify it and reduce 'slipping' during grasping.

Follow these steps to set up a minimal installation of SimplerEnv.

Prerequisites:

• CUDA version >= 11.8 and < 13 (required for full installation including RT-1/Octo inference).
• An NVIDIA GPU (RTX preferred; ray tracing may be slow on non-RTX GPUs like 1080Ti or A100). TPU is not supported.

Setup Steps:

1. Create and activate an Anaconda environment with Python 3.10 or 3.11:

   conda create -n simpler_env python=3.10
   conda activate simpler_env

2. Clone the repository with submodules:

   git clone https://github.com/simpler-env/SimplerEnv --recurse-submodules

3. Install numpy<2.0 to avoid IK errors in pinocchio:

   pip install numpy==1.24.4

4. Install ManiSkill2 real-to-sim environments:

   cd {this_repo}/ManiSkill2_real2sim
   pip install -e .

5. Install the SimplerEnv package:

   cd {this_repo}
   pip install -e .

conda create -n simpler_env python=3.10
conda activate simpler_env
git clone https://github.com/simpler-env/SimplerEnv --recurse-submodules
pip install numpy==1.24.4
cd ManiSkill2_real2sim
pip install -e .
cd ..
pip install -e .