Isaac Lab Documentation

Isaac Lab Mimic (Mimic) is a trajectory data generation tool designed to augment human demonstrations by generating synthetic data. It automatically creates new demonstrations of the same task but with different spatial configurations.

How it works:

• Splits human demonstrations into subtasks based on reference objects.
• Applies rigid body transformations to the robot's actions.
• Evaluates new demonstrations for success and adds them to the dataset.

Requirements:

• OS: Supported on Linux only.
• Action Space: The embodiment's action space must be defined in task space. If using joint space, actions must be converted to task space using forward kinematics.
• Compatibility: Supports single-eef (manipulators) and multi-eef (humanoids) embodiments.

Isaac Teleop is a unified framework for high-fidelity egocentric and robot data collection. It provides a standardized device interface and a flexible graph-based retargeting pipeline that works across simulated and real-world robots. It replaces the previous native XR teleop stack (isaaclab.devices.openxr) in Isaac Lab.

Quick Start: For installation and first-run steps, refer to the cloudxr-teleoperation how-to guide.

PhysX is the primary and most broadly supported backend in Isaac Lab. It serves as the reference for behavior parity and supports all public assets, sensors, and renderer surfaces. Most existing isaaclab_tasks default to the PhysX preset.

Core Simulation Support:

• Articulation API: Supports multi-link, fixed-base, floating-base, and single-body articulations (modeled as rigid bodies).
• Rigid Objects: Supports Rigid Object and Rigid Object Collection APIs.
• Pipelines: Supports both CPU and GPU pipelines. The GPU pipeline is the default for vectorized Reinforcement Learning (RL) workloads.
• Soft-body/Particles: Available via PhysX schemas, though not directly exposed through the standard Isaac Lab asset surface.

The Isaac Lab reference architecture provides an end-to-end workflow for robot learning, covering the entire process from initial asset input to model deployment. It is designed for robotics developers and researchers to build foundational environments, design tasks, and train/test policies using Isaac Sim and Isaac Lab.

The workflow consists of six major components:

1. Asset Input: Providing the raw assets.
2. Configuration: Defining assets and the scene.
3. Robot Learning Task Design: Designing the learning task.
4. Register with Gymnasium: Making the environment compatible with the Gymnasium API.
5. Environment Wrapping: Wrapping environments for specific training needs.
6. Run Training/Testing: Executing the training and evaluation loops.

Newton is a GPU-accelerated, extensible, and differentiable physics simulation engine integrated into Isaac Lab. It is built on top of NVIDIA Warp and integrates MuJoCo Warp. In Isaac Lab, Newton support is currently in beta and is focused on a validated set of classic Reinforcement Learning (RL) and flat-terrain locomotion environments.

Key characteristics:

• High Performance: GPU-accelerated simulation.
• Differentiable: Designed for research and advanced simulation workflows.
• Validated: Newton simulation has been validated against PhysX via policy transfer.
• Extensible: Supports multiple solvers through the NewtonManager abstraction.

Isaac Lab provides a specialized set of interfaces and wrappers around Isaac Sim classes, specifically optimized for reinforcement learning (RL) requirements. While Isaac Sim is a general-purpose simulator, Isaac Lab specializes in:

• Augmenting simulators with non-ideal actuator models.
• Managing observation and reward settings for learning tasks.
• Integrating various sensors into the simulation loop.
• Providing interfaces to physics features not natively available in Isaac Sim (e.g., deformable bodies).

This specialization aims to provide a unified description for different simulation systems tailored for learning workflows.

Isaac Lab provides integration with Ray to streamline the dispatching of multiple training jobs (in parallel or in series) and to perform hyperparameter tuning on both local and remote clusters.

Key capabilities include:

• Parallel/Sequential Job Dispatching: Run multiple training jobs with minimal interaction.
• Hyperparameter Tuning: Support for multiple GPUs and/or multiple GPU nodes.
• Resource Isolation: Use resource-wrapped jobs to define specific requirements for each sub-job.
• Consistency: Use the same training setup across local and cloud environments.

Note: This functionality is experimental and has only been tested on Linux. Due to security risks, Ray clusters should only be deployed in trusted, isolated networks.