DOCSEARCHbyAlgolia
Submit Docs

rl_sar

repository·main·Indexed 23 days ago

https://github.com/fan-ziqi/rl_sar

A framework designed to bridge the gap between robot reinforcement learning simulation and real-world deployment. It supports various robot types (including Unitree A1, Go2, Go2W, and G1) and simulation environments such as IsaacGym, IsaacSim, Gazebo, and MuJoCo. The framework provides tools for Docker-based deployment, ROS1/ROS2 integration, and real-hardware interfacing.

Tokens
14.2K
Snippets
31
Records
68
Agent score
79%

What's inside rl_sar

  1. Overview of rl_sar

    main

    rl_sar (Simulation and Real) is a framework for the simulation verification and real-world deployment of robot reinforcement learning algorithms. It is designed for quadruped, wheeled-legged, and humanoid robots.

    Key Capabilities:

    • Simulators: Supports IsaacGym, IsaacSim, Gazebo, and MuJoCo (partial support).
    • Middleware: Supports ROS-Noetic and ROS2 (Foxy/Humble).
    • Inference Engines: Supports libtorch and onnxruntime.
    • Platforms: Supports Linux and macOS (macOS only supports MuJoCo simulation).
    • Applications: Supports Locomotion and Dance tasks.

    Robot Support Matrix (rname):

    Robot NamePre-Trained PolicyGazeboMujocoReal
    Unitree-A1 (a1)legged_gym (IsaacGym)✅❌✅
    Unitree-Go2 (go2)himloco (IsaacGym) / robot_lab (IsaacSim)✅✅✅
    Unitree-Go2W (go2w)robot_lab (IsaacSim)✅✅✅
    Unitree-B2 (b2)robot_lab (IsaacSim)✅✅⚪
    Unitree-B2W (b2w)robot_lab (IsaacSim)✅✅⚪
    Unitree-G1 (g1)robomimic/locomotion (IsaacGym) / robomimic/charleston (IsaacGym) / whole_body_tracking/dance_102 (IsaacSim) / whole_body_tracking/gangnam_style (IsaacSim)✅✅✅
    FFTAI-GR1T1 (gr1t1)legged_gym (IsaacGym)✅❌⚪
    FFTAI-GR1T2 (gr1t2)legged_gym (IsaacGym)✅❌⚪
    zhinao-L4W4 (l4w4)legged_gym (IsaacGym)✅❌✅
    Deeprobotics-Lite3 (lite3)himloco (IsaacGym)✅❌✅
    Agibot-D1 (d1)robot_lab (IsaacSim)✅✅✅
    DDTRobot-Tita (tita)robot_lab (IsaacSim)✅❌⚪

    Note: ⚪ indicates partial/unconfirmed support. For training policies using IsaacLab (IsaacSim), use the robot_lab project. The joint_names in robot_lab config match the xxx/robot_lab/config.yaml in this repository.

  2. Run rl_sar using Docker

    main

    Docker supports both simulation and deployment. To use Docker:

    1. Enable X11 on host: xhost +local:docker
    2. Start containers: cd docker && docker compose up -d
    3. Enter container: docker compose exec rl_sar bash

    Inside the container, you can run:

    • MuJoCo: ./cmake_build/bin/rl_sim_mujoco g1 scene_29dof
    • Gazebo: ros2 launch rl_sar gazebo.launch.py rname:=go2 (and run ros2 run rl_sar rl_sim in a new terminal)
    • Real Robot: ./cmake_build/bin/rl_real_go2 <NETWORK_INTERFACE>
    xhost +local:docker
cd docker && docker compose up -d
docker compose exec rl_sar bash

  3. Run Gazebo simulation

    main

    To run the Gazebo simulation, you must launch the environment first, and then launch the control program rl_sim in a separate terminal. Without rl_sim, the robot will not be controlled and may fall over.

    If you are using Ubuntu 22.04 and cannot see the robot, it may have initialized outside the field of view; launching rl_sim will automatically reset its position. If the robot falls during standing, use keyboard R or gamepad RB+Y to reset.

    If Gazebo fails to open for the first time, download the model package:

    git clone https://github.com/osrf/gazebo_models.git ~/.gazebo/models
    # ROS1: Launch Gazebo
source devel/setup.bash
roslaunch rl_sar gazebo.launch rname:=<ROBOT>

# ROS1: Launch Control (New Terminal)
source devel/setup.bash
rosrun rl_sar rl_sim

# ROS2: Launch Gazebo
source install/setup.bash
ros2 launch rl_sar gazebo.launch.py rname:=<ROBOT>

# ROS2: Launch Control (New Terminal)
source install/setup.bash
ros2 run rl_sar rl_sim

  4. Configure auto-start for rl_sar on Unitree Jetson

    main

    To have the rl_real_go2 control program start automatically on boot on the onboard Jetson:

    1. Create a systemd service file at /etc/systemd/system/rl_sar.service.
    2. Define the ExecStart path to your binary (e.g., /home/unitree/rl_sar/cmake_build/bin/rl_real_go2 eth0 wheel).
    3. Reload, enable, and start the service using systemctl.

  6. Add a new robot to rl_sar

    main
    To integrate a new robot into the rl_sar framework, you must create or modify specific files following a strict naming convention. Replace <ROBOT> with your robot's name and <CONFIG> with your specific configuration name. The required files are categorized into robot description, policy files, Finite State Machine (FSM) definitions, and physical deployment code.

  7. Set up rl_sar as a systemd service for auto-start

    main

    To make rl_sar start automatically on boot (e.g., on a Unitree Jetson), create a systemd service file.

    1. Create the file: sudo touch /etc/systemd/system/rl_sar.service
    2. Configure the service: Edit the file with the following template (adjust WorkingDirectory and ExecStart to your actual paths):
    [Unit]
Description=RL SAR Service
After=network.target

[Service]
Type=simple
User=unitree
WorkingDirectory=/home/unitree/rl_sar
ExecStart=/home/unitree/rl_sar/cmake_build/bin/rl_real_go2 eth0 wheel
Restart=on-failure
RestartSec=5
StandardOutput=journal
StandardError=journal

[Install]
WantedBy=multi-user.target
    1. Enable and Start:
    sudo systemctl daemon-reload
sudo systemctl enable rl_sar.service
sudo systemctl start rl_sar.service
    1. Manage Service:
    • Stop: sudo systemctl stop rl_sar.service
    • Restart: sudo systemctl restart rl_sar.service
    • View Logs: sudo journalctl -u rl_sar.service -f
    [Unit]
Description=RL SAR Service
After=network.target

[Service]
Type=simple
User=unitree
WorkingDirectory=/home/unitree/rl_sar
ExecStart=/home/unitree/rl_sar/cmake_build/bin/rl_real_go2 eth0 wheel
Restart=on-failure
RestartSec=5
StandardOutput=journal
StandardError=journal

[Install]
WantedBy=multi-user.target

  8. Control Agibot D1 real robot

    main

    Connect via WiFi (Default IP: 192.168.234.1).

    Network Configuration:

    1. SSH into the robot (ssh firefly@192.168.234.1, password: firefly).
    2. Edit /opt/export/config/sdk_config.yaml and set target_ip to your PC's IP address.
    3. Reboot the robot.

    Run Control Program:

    # ROS1
source devel/setup.bash
rosrun rl_sar rl_real_d1 [local_ip] [robot_ip]

# ROS2
source install/setup.bash
ros2 run rl_sar rl_real_d1 [local_ip] [robot_ip]

# CMake
./cmake_build/bin/rl_real_d1 [local_ip] [robot_ip]

# Example (using default IP)
./cmake_build/bin/rl_real_d1 192.168.234.2 192.168.234.1

  9. Control Deeprobotics Lite3 real robot

    main

    Lite3 is typically connected via Wireless (WiFi).

    IMPORTANT: Recheck joint mapping parameters in rl_sar/policy/himloco/config.yaml. The default Sim2Sim mapping differs from real-world mapping; mismatching these can cause safety hazards.

    Setup:

    1. Determine Lite3 IP and port.
    2. Modify lines 46-48 in rl_sar/src/rl_real_lite3.cpp.
    3. Update jy_exe/conf/network.toml on the Lite3 motion host to match your local machine's IP/port.

    Run Control Program:

    # ROS1
source devel/setup.bash
rosrun rl_sar rl_real_lite3

# ROS2
source install/setup.bash
ros2 run rl_sar rl_real_lite3

# CMake
./cmake_build/bin/rl_real_lite3

  10. Prepare trained models and configuration for rl_sar

    main

    Before running any robot or simulation, you must prepare the environment by:

    1. Copying the trained .pt model file to rl_sar/src/rl_sar/policy/<ROBOT>/<CONFIG>.
    2. Configuring the parameters in <ROBOT>/<CONFIG>/config.yaml and <ROBOT>/base.yaml.

  11. Use software rendering (Mesa llvmpipe) in Docker

    main

    If you do not have a GPU available or want to use software rendering instead of NVIDIA GPU acceleration, use the software profile in Docker Compose.

    Start software rendering mode:

    docker compose --profile software up -d rl_sar_software

    Verify the renderer: Run the following command to check if you are using llvmpipe (software) or an NVIDIA driver (GPU):

    docker exec -it rl_sar glxinfo | grep "OpenGL renderer"

    Expected outputs:

    • GPU mode: NVIDIA GeForce RTX xxxx
    • Software mode: llvmpipe (LLVM xx.x, 256 bits)
    docker compose --profile software up -d rl_sar_software

docker exec -it rl_sar glxinfo | grep "OpenGL renderer"

  12. Install ROS dependencies for rl_sar

    main

    If you intend to use ROS with rl_sar, install the following additional packages.

    ROS Noetic (Ubuntu 20.04):

    sudo apt install ros-noetic-teleop-twist-keyboard ros-noetic-controller-interface ros-noetic-gazebo-ros-control ros-noetic-joint-state-controller ros-noetic-effort-controllers ros-noetic-joint-trajectory-controller ros-noetic-joy ros-noetic-ros-control ros-noetic-ros-controllers ros-noetic-controller-manager

    ROS2 Foxy (Ubuntu 20.04) / ROS2 Humble (Ubuntu 22.04):

    sudo apt install ros-$ROS_DISTRO-teleop-twist-keyboard ros-$ROS_DISTRO-ros2-control ros-$ROS_DISTRO-ros2-controllers ros-$ROS_DISTRO-control-toolbox ros-$ROS_DISTRO-robot-state-publisher ros-$ROS_DISTRO-joint-state-publisher-gui ros-$ROS_DISTRO-gazebo-ros2-control ros-$ROS_DISTRO-gazebo-ros-pkgs ros-$ROS_DISTRO-xacro
    # ros-noetic (Ubuntu20.04)
sudo apt install ros-noetic-teleop-twist-keyboard ros-noetic-controller-interface ros-noetic-gazebo-ros-control ros-noetic-joint-state-controller ros-noetic-effort-controllers ros-noetic-joint-trajectory-controller ros-noetic-joy ros-noetic-ros-control ros-noetic-ros-controllers ros-noetic-controller-manager

# ros2-foxy (Ubuntu20.04) / ros2-humble (Ubuntu22.04)
sudo apt install ros-$ROS_DISTRO-teleop-twist-keyboard ros-$ROS_DISTRO-ros2-control ros-$ROS_DISTRO-ros2-controllers ros-$ROS_DISTRO-control-toolbox ros-$ROS_DISTRO-robot-state-publisher ros-$ROS_DISTRO-joint-state-publisher-gui ros-$ROS_DISTRO-gazebo-ros2-control ros-$ROS_DISTRO-gazebo-ros-pkgs ros-$ROS_DISTRO-xacro