SAPIEN Documentation

SAPIEN supports two levels of object segmentation via get_uint32_texture():

1. Mesh-level segmentation: Provides unique IDs for individual meshes within an actor.
2. Actor-level segmentation: Provides unique IDs for the entire actor.

This is useful for tasks requiring fine-grained part identification versus coarse object identification.

In the SAPIEN ray tracing pipeline, scene and camera information are provided via descriptor sets. Unlike rasterization, all object and material information must be available simultaneously for ray tracing.

Key requirements:

• The Vertices buffer must use std430 layout to ensure compatibility with the rasterization pipeline if using both.
• The set number and binding order are flexible, but the data structures must match the expected GLSL definitions.

Core data structures include GeometryInstance, Material, TextureIndex, Object, and various light types (PointLight, DirectionalLight, SpotLight).

layout(std430, set = 1, binding = 8) readonly buffer Vertices {
  Vertex v[];
} vertices[];

Ray tracing enables realistic material properties that rasterization cannot faithfully model. To achieve these effects, use the following material properties when building your scene:

• Transparency/Refraction: Set a large transmission value on a material to create transparent objects (e.g., glass spheres).
• Metallic/Reflection: Assign highly metallic materials to objects to produce realistic reflections.

Ensure the ray_tracing flag is enabled in your rendering setup to see these effects.

# Conceptual example of setting material properties for ray tracing
# (Based on rt_mat.py logic)

# Transparent material
transparent_material = sapien.render.Material(transmission=0.9)

# Metallic material
metallic_material = sapien.render.Material(metallic=1.0)

An articulation in SAPIEN is a tree-structured collection of links connected by joints.

• Links: Act as rigid bodies. They can have visual and collision shapes attached.
• Joints: Define the relationship between a parent link and a child link. Common types include:
  • revolute: A hinge joint that allows rotation around a common axis.
  • prismatic: A slider joint that allows translation along a common axis.
  • fixed: Locks the parent and child links together rigidly.
• Tree Structure: The articulation is represented as a tree where each node is a link and each edge is a joint. A root link is a link created without a specified parent.

SAPIEN uses a hierarchical structure for physical simulation:

1. engine: The most basic interface for physical simulation. It is used to create simulation scenes.
2. scene: A simulation instance where individual physics runs occur. You can call step() on a scene to advance the simulation. Multiple scenes can be created to run simulation steps independently (similar to an env in OpenAI Gym).
3. renderer: The interface for rendering. A single renderer typically manages the visualization for all scenes.

import sapien
engine = sapien.Engine()
scene = engine.create_scene()

To create a custom reinforcement learning environment, you should inherit from gym.Env and implement a base class (e.g., SapienEnv).

Key characteristics of a SapienEnv implementation:

• Constructor: Responsible for setting up the SAPIEN engine, scene, and renderer, and then calling self._build_world().
• _build_world(): A virtual function that you must implement to build the simulation world (e.g., creating ground, lighting, or specific articulations).
• _setup_viewer(): A virtual function used for on-screen visualization.

Note: Unlike Mujoco, SAPIEN does not support creating a simulation world directly from an XML file. You must implement your own parsers if you wish to use specific file formats.

SAPIEN 3 introduces a major API and infrastructure overhaul based on an Entity-Component System (ECS).

Core Changes:

• Entities vs Actors: The Actor from SAPIEN 2 is now an Entity in SAPIEN 3. Functionalities previously on the Actor are now components attached to an Entity.
• Builders: Actor and articulation builder APIs remain mostly unchanged, but color and material parameters for visual shapes are now unified under material.
• Simulation: Support for PhysX 5 GPU simulation is available.

While robots are often loaded from URDF files, you can also construct articulated objects directly using the SAPIEN Python API. This is done by using an ArticulationBuilder in conjunction with LinkBuilder for each link in the articulation chain.

This approach allows for programmatic generation of complex robot models without requiring external files.

# Conceptual pattern for building an articulated robot
builder = sapien.articulation.ArticulationBuilder()
# ... configure builder with links and joints ...
# builder.add_link(link_builder)
robot = builder.build()

In SAPIEN, simulated rigid bodies are referred to as Actors. You can add objects like ground planes or boxes to your Scene to populate the physical environment.

# Example of adding a ground and a box
ground = scene.create_actor(name="ground")
# (Note: Actual geometry/shape configuration follows actor creation)

box = scene.create_actor(name="box")

A shader pack is a directory containing GLSL files that defines the rendering behavior of the SAPIEN renderer. SAPIEN identifies the type of pipeline based on the presence of specific files:

• Rasterization shader pack: Requires gbuffer.frag.
• Ray-tracing shader pack: Requires camera.rgen.

To find the default shader packs provided by SAPIEN, you can use the following Python commands:

Default Rasterization Pack Path:

python -c 'import os,sapien; print(os.path.dirname(sapien.__file__) + "/vulkan_shader/ibl")'

Default Ray-tracing Pack Path:

python -c 'import os,sapien; print(os.path.dirname(sapien.__file__) + "/vulkan_shader/rt")'

Note: The current design for renderer customization is experimental and subject to breaking changes.