DOCSEARCHbyAlgolia
Submit Docs

Habitat-Lab

repository·main·Indexed 25 days ago

https://github.com/facebookresearch/habitat-lab

A modular high-level library for end-to-end development in embodied AI, focused on training agents in indoor environments and human-robot interaction tasks. It includes various Human-in-the-Loop (HITL) applications such as basic_viewer, sim_viewer, and VR-enabled tools like Pick_throw_vr and XR reader for Quest headsets. The library also features Variable Experience Rollout (VER) to mitigate the straggler effect during experience collection.

Tokens
42.9K
Snippets
97
Records
186
Agent score
84%

What's inside habitat-lab

  2. Use ROS-X-Habitat to bridge Habitat with ROS

    main

    ROS-X-Habitat is a framework that bridges Habitat Lab and Habitat Sim with other robotics resources via ROS. It allows leveraging Habitat Sim v2's physics-based simulation and accessing simulation assets from ROS.

    Note: This framework is maintained by the Lab for Computational Intelligence at UBC and is not officially supported by the Habitat Lab team. Refer to the ROS-X-Habitat repository for documentation.

  3. Understand the HITL Framework Architecture

    main

    The Human-in-the-Loop (HITL) framework is composed of three main parts: the habitat-hitl Python library, desktop application examples, and a Unity-based VR client.

    The library architecture is split into three primary functional components:

    1. LabDriver: A Habitat environment wrapper. It creates a habitat.Env instance and handles simulation updates via sim_update. It receives a gui_input object (encapsulating keyboard/mouse) and an AppState (containing application-specific logic like controls and help text). It returns a post_sim_update_dict containing rendering info (keyframes, camera transforms) and optional debug images.
    2. GuiApplication: Manages the OS window (currently via GLFW), handles OS-level input, and updates the display by invoking the renderer.
    3. ReplayGuiAppRenderer: An application-agnostic renderer. It consumes the post_sim_update_dict from LabDriver to render the scene from the requested camera pose, supporting 3D model rendering, 3D debug lines, 2D debug images, and 2D on-screen help text.

  4. Use Batch Rendering to save GPU memory

    main
    Batch rendering is an experimental system in Habitat 2.0 designed to centralize rendering. Instead of each simulator instance having its own renderer and graphics context, a single renderer on the main training process aggregates simulation states and renders them simultaneously. This reduces GPU memory usage, decreases episode loading time, and improves rendering efficiency by pre-loading assets once.

  5. Understand the Habitat Stack

    main

    Habitat is an embodied AI research platform composed of two primary components:

    1. Habitat-Sim: A high-performance 3D simulator that handles agents, sensors, and 3D datasets (like MatterPort3D and Gibson).
    2. Habitat Lab: A modular library for end-to-end development, including defining tasks (navigation, instruction following, etc.), configuring agents, training (RL or imitation learning), and benchmarking performance.

    To use Habitat, you must install both habitat-sim and habitat-lab.

  6. Understand Articulated Agent Design in Habitat

    main

    Habitat represents agents (robots or humanoids) as articulated objects. These objects consist of multiple rigid parts (links) connected by joints that allow for rotational or translational motion.

    Implementation locations:

    • Robots: Found in the robots folder.
    • Humanoids: Found in the humanoids folder.

    Agents inherit from articulated agent modules to implement their specific morphologies.

  8. Understand the Habitat-Lab Configuration System

    main

    Habitat-Lab uses Hydra for its configuration system, replacing the previous YACS-based system. The final configuration (Output Config) is dynamically composed at runtime from multiple Input Configs specified in a defaults list and overridden via the primary YAML file or command-line arguments.

    Key components include:

    • Structured Configs: Used as validation schemas to ensure required fields are present and match the correct types. These are registered in the ConfigStore.
    • Config Groups: Grouped configurations located in the Config Search Path (e.g., habitat/task).
    • Package: Defines the path/node where a configuration is placed in the final Output Config (e.g., task configs in habitat/task default to the habitat.task package).
    • Config Search Path: Habitat-Lab extends the search path using the HabitatConfigPlugin to ensure all habitat/config paths are visible to Hydra.

  9. Understand Robot Design and Component Hierarchy

    main

    Habitat supports three primary robot types: Fetch (Fetch Robotics), Stretch (Hello Robot), and Spot (Boston Dynamics).

    Component Hierarchy

    • fetch_robot.py and stretch_robot.py and spot_robot.py are built upon mobile_manipulator.py.
    • franka_robot.py is built upon static_manipulator.py.

    Initialization and Parameters

    • Parameters: Robot-specific parameters (camera transformations, end-effector positions, control gains, etc.) are defined within their respective robot classes (e.g., stretch_robot.py).
    • Initialization Flow: Robots are imported by habitat/tasks/rearrange/articulated_agent_manager.py, which is then imported and initialized by habitat/tasks/rearrange/rearrange_sim.py.

  12. Download test assets for debugging

    main

    To verify if a problem is related to specific scenes, download the standard test assets using the following command from the root of your habitat-lab or habitat-sim repository:

    python -m habitat_sim.utils.datasets_download --uids habitat_test_scenes --data-path data/