DOCSEARCHbyAlgolia
Submit Docs

LeIsaac Documentation

repository·main·Indexed 20 days ago

https://github.com/lightwheelai/leisaac

A teleoperation and imitation learning framework bridging IsaacLab and LeRobot. LeIsaac enables high-quality robotic data collection in simulation, conversion for training, and deployment of fine-tuned policies to real-world hardware. Key features include DigitalTwin Env for sim2real transfer, Hugging Face EnvHub support for loading tasks, and integration with LeRobot Recorder for direct data collection.

Tokens
22.4K
Snippets
68
Records
91
Agent score
72%

What's inside LeIsaac

  1. Overview of LeIsaac functionality

    main

    LeIsaac provides teleoperation capabilities within IsaacLab specifically designed for the SO101Leader (LeRobot). The project facilitates a complete pipeline from data collection to policy deployment:

    • Teleoperation: Uses the SO101 Follower robot (and related hardware) in IsaacLab for practical teleoperation.
    • Automated Data Collection: Uses state machine scripted policies to collect data without requiring human teleoperation.
    • Data Conversion: Provides scripts to convert HDF5 data into the LeRobot dataset format.
    • Policy Training & Deployment: Simulation data can be used to fine-tune GR00T N1.5 for deployment on real hardware.

  2. Overview of the LeIsaac × Cosmos Video-to-Action Pipeline

    main

    This pipeline integrates LeIsaac, Cosmos-Predict2.5, and GR00T-Dreams IDM to create a scalable synthetic data generation loop. The workflow follows these stages:

    1. Data Collection (LeIsaac): Collect teleoperated demonstrations in HDF5 format and convert them to LeRobot format.
    2. Video Generation (Cosmos-Predict2.5): Post-train Cosmos on the LeRobot videos, then run inference to generate synthetic rollout videos conditioned on text prompts and initial video frames.
    3. Action Inference (IDM): Fine-tune IDM on the dataset to infer robot actions (trajectories in parquet format) from the Cosmos-generated videos.
    4. Evaluation (LeIsaac): Convert both the original HDF5 and the new IDM-generated trajectories back into a replayable LeIsaac HDF5 dataset for evaluation in Isaac Sim.

  4. How LeRobot Recorder works

    main

    LeRobot Recorder is an enhanced feature in LeIsaac that records data directly in LeRobot format during teleoperation, eliminating the need for post-hoc data conversion. It uses the LeRobotRecorderManager to manage the recording lifecycle.

    The recording process follows these steps per environment step:

    1. Data Collection: Captures observations (joint positions, camera images, etc.) and actions.
    2. Format Conversion: Uses build_lerobot_frame to convert data into the LeRobot frame format.
    3. Buffer Management: Stores frames in a LeRobot Dataset buffer.
    4. Episode Processing:
      • On Success: Calls flush() to save the episode to the dataset.
      • On Failure: Calls clear() to discard the buffer, preventing failed attempts from polluting the dataset.

    Note on Data Quality: LeIsaac automatically skips the first 5 frames of every episode to prevent initial state instability from affecting the dataset.

  5. Getting Started with LeIsaac

    main

    To begin using LeIsaac, refer to the official documentation for specific setup and usage guides:

    • Installation and Setup: Follow the guide to install dependencies and configure your environment.
    • Extra Features: Explore advanced capabilities like the datagen module for programmatic motion trajectories or lerobot recorder integration.
    • Policy Support: Learn how to perform inference with supported policies (e.g., GR00T N1.6).
    • Resources: Browse the lists of Available Robots, Environments, and Policies.

  6. Convert HDF5 teleoperation data to LeRobot Dataset v2

    main

    To convert collected teleoperation data (stored in HDF5 format) into the LeRobot Dataset v2 format, use the isaaclab2lerobot.py script. Note that only successful episodes are converted.

    Prerequisites

    Install the required dependencies:

    pip install lerobot==0.3.3
pip install numpy==1.26.0
    python scripts/convert/isaaclab2lerobot.py \
    --task_name=LeIsaac-SO101-PickOrange-v0 \
    --repo_id=EverNorif/so101_test_orange_pick \
    --hdf5_root=./datasets \
    --hdf5_files=dataset.hdf5

  7. Install LeIsaac as a package

    main

    To install LeIsaac as a dependency with all required components (IsaacLab, IsaacSim, etc.) provisioned, use the following workflow. This method is recommended for most users but may expose edge cases.

    conda create -n leisaac python=3.11
conda activate leisaac

# Install cuda-toolkit
conda install -c "nvidia/label/cuda-12.8.1" cuda-toolkit

# Install PyTorch (CUDA 12.8 wheels)
pip install -U torch==2.7.0 torchvision==0.22.0 --index-url https://download.pytorch.org/whl/cu128

# Install LeIsaac and IsaacLab/IsaacSim extras
pip install 'leisaac[isaaclab] @ git+https://github.com/LightwheelAI/leisaac.git#subdirectory=source/leisaac' --extra-index-url https://pypi.nvidia.com

  8. Install LeIsaac from source

    main

    For local development, clone the repository recursively and follow these steps to set up the environment, IsaacSim, and IsaacLab manually. This is the preferred method if you encounter issues with the package installation.

    # Clone repository and submodules
git clone https://github.com/LightwheelAI/leisaac.git --recursive

# Create and activate environment
conda create -n leisaac python=3.11
conda activate leisaac

# Install cuda-toolkit
conda install -c "nvidia/label/cuda-12.8.1" cuda-toolkit

# Install PyTorch
pip install -U torch==2.7.0 torchvision==0.22.0 --index-url https://download.pytorch.org/whl/cu128

# Install IsaacSim
pip install --upgrade pip
pip install "isaacsim[all,extscache]==5.1.0" --extra-index-url https://pypi.nvidia.com

# Install IsaacLab
sudo apt install cmake build-essential
cd leisaac/dependencies/IsaacLab
./isaaclab.sh --install

# Install leisaac in editable mode
cd ../..
pip install -e source/leisaac

  9. Verify scene composition via teleoperation

    main

    Once you have updated the task configuration with your custom USD scene, use the teleop_se3_agent.py script to verify that the scene is correctly composed and functional. This allows you to manually operate the robot within the new environment to check for collisions, asset placement, and rendering issues.

    python scripts/environments/teleoperation/teleop_se3_agent.py \
    --task=LeIsaac-SO101-CleanToyTable-v0 \
    --teleop_device=so101leader \
    --port=/dev/ttyACM0 \
    --num_envs=1 \
    --device=cuda \
    --enable_cameras \
    --record \
    --dataset_file=./datasets/dataset.hdf5

  10. Use Keyboard control for Single-Arm SO101 Follower

    main

    When using --teleop_device=keyboard, control is based on the target frame (gripper link) coordinate system. This allows you to command the desired pose of the link directly rather than using joint-level commands. Gripper joints are handled separately.

    Keyboard Mappings:

    Input KeyDescription
    W / SForward / backward
    A / DLeft / right
    Q / EUp / down
    J / LRotate (yaw) left / right
    K / IRotate (pitch) up / down
    U / OGripper open / close
    --teleop_device=keyboard

  12. Generate additional demonstrations using MimicGen Env

    main

    LeIsaac integrates IsaacLab MimicGen to automatically generate new demonstrations from existing expert demonstrations. The workflow follows a pipeline: converting joint actions to IK actions, annotating sub-tasks, generating new data, and converting the data back to joint actions.

    Workflow Pipeline

    1. Convert to IK Actions: Convert collected joint-position-based data to end-effector (IK) based action data. This is required because MimicGen generalizes trajectories based on end-effector and object poses.
    2. Annotate Demonstrations: Perform sub-task annotation. You can use automatic annotation by adding the --auto flag.
    3. Generate Dataset: Run the MimicGen generation process to create new trials.
    4. Convert back to Joint Actions: Transform the generated IK-based data back into joint-position-based data for use in standard training pipelines.
    5. Replay: Use replay scripts to verify the generated data (note that simulation randomness may cause variations in replay performance).

    Important Configuration

    • File Chaining: The output_file from one step typically becomes the input_file for the next step.
    • Task Type: If your initial demonstrations were collected via a specific device (e.g., keyboard), you must specify --task_type=keyboard during the annotate_demos.py and generate_dataset.py steps. This is not required when replaying the final converted dataset.
    # 1. Convert to IK
python scripts/mimic/eef_action_process.py \
    --input_file ./datasets/mimic-lift-cube-example.hdf5 \
    --output_file ./datasets/processed_mimic-lift-cube-example.hdf5 \
    --to_ik --headless

# 2. Annotate (with --auto for automatic annotation)
python scripts/mimic/annotate_demos.py \
    --device cuda \
    --task LeIsaac-SO101-LiftCube-Mimic-v0 \
    --input_file ./datasets/processed_mimic-lift-cube-example.hdf5 \
    --output_file ./datasets/annotated_mimic-lift-cube-example.hdf5 \
    --enable_cameras

# 3. Generate
python scripts/mimic/generate_dataset.py \
    --device cuda \
    --num_envs 1 \
    --generation_num_trials 10 \
    --input_file ./datasets/annotated_mimic-lift-cube-example.hdf5 \
    --output_file ./datasets/generated_mimic-lift-cube-example.hdf5 \
    --enable_cameras

# 4. Convert back to Joint
python scripts/mimic/eef_action_process.py \
    --input_file ./datasets/generated_mimic-lift-cube-example.hdf5 \
    --output_file ./datasets/final_generated_mimic-lift-cube-example.hdf5 \
    --to_joint --headless