DOCSEARCHbyAlgolia
Submit Docs

HomeRobot

repository·main·Indexed 23 days ago

https://github.com/facebookresearch/home-robot

An open-source robotic mobile manipulation stack for low-cost manipulators like the Hello Robot Stretch. It provides tools for Open Vocabulary Mobile Manipulation (OVMM), enabling robots to explore environments, find objects, and place them in receptacles via high-level semantic instructions. The stack includes support for Habitat ObjectNav and OVMM, data collection tools for H5 demonstration files, and utilities for URDF model export and gripper replacement.

Tokens
38.1K
Snippets
130
Records
158
Agent score
78%

What's inside HomeRobot

  1. Use Stretch robot environments for continual imitation learning

    main

    This project provides OpenAI Gym-compatible environments specifically designed for running continual imitation learning on a Stretch robot. It supports three primary workflows:

    1. Demonstration Collection: Use stretch_collect_demo_env.py to collect new demonstrations.
    2. Offline Training: Use stretch_offline_demo_env.py to train using existing demonstration datasets.
    3. Real-world Deployment: Use stretch_live_env.py to run a trained policy on actual Stretch robot hardware.

    Because these environments adhere to the OpenAI Gym API, they are compatible with standard reinforcement learning frameworks.

  2. HomeRobot Package Organization

    main

    The repository is organized into several specialized packages:

    PackageDescription
    home_robot (src/home_robot)Core package containing agents and interfaces
    home_robot_sim (src/home_robot_sim)OVMM simulation environment based on AI Habitat
    home_robot_hw (src/home_robot_hw)ROS package containing hardware interfaces for the Hello Robot Stretch
    home_robot_spot (src/home_robot_spot)Minimal package for using the Boston Dynamics Spot

  3. H5 Data Format and Custom Recording

    main

    Collected data is stored in H5 files containing Trial objects with the following structure:

    • temporal: Contains q (joint position), dq (joint velocity), ee_pose (end effector pose), base_pose, camera_pose, and user_keyframe (the ID of the image frame when the user pressed the keyframe button).
    • config: Configuration data.
    • image: Contains rgb and depth frames.

    To implement a custom data recording format, inherit from the Recorder class and use the following methods:

    • add_frame for temporal keys.
    • add_img_frame for image keys.
    • add_config for configuration keys.

  4. Network architecture for HomeRobot

    main

    HomeRobot uses a client-server architecture to manage computational loads. Because mobile robots often lack sufficient GPU power for state-of-the-art perception, the system splits tasks:

    • Robot (e.g., Stretch): Runs ROS and low-level controllers.
    • Workstation: Runs CPU- and GPU-intensive AI code.

    To function correctly, the robot and workstation must be on the same subnetwork. Recommended hardware options include:

    1. External Router: A dedicated wireless router (e.g., Netgear Nighthawk) connecting both devices.
    2. Stretch's WiFi Hotspot: Using Ubuntu's ad-hoc wireless hotspot feature for debugging.
    3. VPN: Using services like Tailscale to create a local connection if the workstation is remote (e.g., in a datacenter). Note that VPNs may introduce latency in high-resolution image streaming.

  6. Understand the H5 file structure for episodic data

    main

    H5 files are used to store episodic data generated by collect_h5.py. The data is managed by the EpisodeManager class and recorded via the Recorder class. The standard file structure follows this hierarchy:

    # H5-file-|
#         |-episode0-|
#         |          |-head_rgb/<frame_number> (RGB Image)
#         |          |-head_depth/<frame_number> (DEPTH Image)
#         |          |-head_xyz/<frame_number> (XYZ point-cloud)
#         |          |-ee_pose/<frame_number> (end-effector 6D pose)
# ...

    In this structure, episode0 represents trial-index 0.

  7. How HomeRobot environments and agents work together

    main

    HomeRobot is built around two primary abstractions that allow for embodiment-agnostic robotic tasks:

    1. Environments: These extend the Environment class. They are responsible for providing observations of the world and a mechanism to apply actions.
    2. Agents: These extend the Agent class. An agent takes an observation as input and produces an action as output.

    Most new robotic methods are implemented as Agents. The home_robot package provides the core interfaces that allow agents to interact with different environments (like simulation or real hardware) using a consistent API.

  8. Train SLAP components (IPM and APM)

    main

    SLAP consists of two components: the Interaction Prediction Module (IPM) and the Action Prediction Module (APM). You must run two separate training jobs.

    1. Train IPM: Navigate to $HOME_ROBOT_ROOT/project/slap_manipulation and run: python src/slap_manipulation/policy/interaction_prediction_module.py

    2. Train APM: Navigate to $HOME_ROBOT_ROOT/project/slap_manipulation and run: python src/slap_manipulation/policy/action_prediction_module.py

    Configuration: Ensure data-dir, template, and split in the respective YAML config files (src/slap_manipulation/policy/conf/interaction_predictor_training.yaml and src/slap_manipulation/policy/conf/action_predictor_training.yaml) point to your data and valid splits.

    python src/slap_manipulation/policy/interaction_prediction_module.py
python src/slap_manipulation/policy/action_prediction_module.py

  9. Controls for Data Collection

    main

    Use the following controller buttons to manage the demonstration recording process:

    • Start Button: Press once to start an episode (ensure the script is actively writing to a file). Press again to end the episode. The script automatically collects a final frame at the end of the episode.
    • Back Button: Press to communicate a keyframe (a moment of interest) during the episode.

  11. Install SLAP (Spatial-Language Attention Policies)

    main

    To install the SLAP project, follow these steps in order. The environment is tested on Ubuntu 18.04 with CUDA 11.6, Python 3.9, and PyTorch 1.12.1.

    1. Create the Conda environment:

      conda env create -f requirements.yaml

    2. Install PyTorch and PyG dependencies (Conda/Mamba may fail to find the correct builds, so use pip within the activated environment):

      conda activate slap_base
python -m pip install torch==1.12.1+cu116 torchvision==0.13.1+cu116 torchaudio==0.12.1 --extra-index-url https://download.pytorch.org/whl/cu116
python -m pip install --no-index pyg_lib torch_scatter torch_sparse torch_cluster torch_spline_conv -f https://data.pyg.org/whl/torch-1.12.1+cu116.html
python -m pip install torch_geometric

    3. Install home-robot components and Detectron2: Navigate to your HOME_ROBOT_ROOT and run:

      cd ../.. # Return to HOME_ROBOT_ROOT
python -m pip install -e src/home_robot
python -m pip install -e src/home_robot_hw
python -m pip install 'git+https://github.com/facebookresearch/detectron2.git'

    Note: You must follow the home-robot instructions to download the Detectron2 checkpoint.

    conda activate slap_base
python -m pip install torch==1.12.1+cu116 torchvision==0.13.1+cu116 torchaudio==0.12.1 --extra-index-url https://download.pytorch.org/whl/cu116
python -m pip install --no-index pyg_lib torch_scatter torch_sparse torch_cluster torch_spline_conv -f https://data.pyg.org/whl/torch-1.12.1+cu116.html
python -m pip install torch_geometric

  12. Install HomeRobot on a workstation

    main

    To set up a workstation for HomeRobot, you need Python 3.9, conda, and mamba. It is recommended to use CUDA 11.7.

    1. Prerequisites

    If on Ubuntu, install basic build tools:

    sudo apt update
sudo apt install build-essential zip unzip

    2. Clone the repository

    git clone https://github.com/facebookresearch/home-robot.git
cd ./home-robot

    3. Create the Conda environment

    If you are using ROS, ensure PYTHONPATH is unset before creating the environment to avoid conflicts.

    # If using ROS - make sure you don't have PYTHONPATH set
unset PYTHONPATH

# Otherwise, use the version in src/home_robot
mamba env create -n home-robot -f src/home_robot/environment.yml

# Activate the environment
conda activate home-robot

# Optionally, update this environment to install ROS
mamba env update -f src/home_robot_hw/environment.yml

    4. Run the installation script

    Set your HOME_ROBOT_ROOT and CUDA_HOME environment variables, then run the dependency installer. This script downloads submodules, model checkpoints, and builds Detic.

    export HOME_ROBOT_ROOT=$USER/home-robot
export CUDA_HOME=/usr/local/cuda-11.7

conda activate home-robot
cd $HOME_ROBOT_ROOT
./install_deps.sh