Habitat-Sim

Habitat-Sim is a high-performance, physics-enabled 3D simulator designed for embodied AI research. It prioritizes simulation speed, capable of achieving thousands of frames per second (FPS) for rendering and high steps per second (SPS) for rigid-body dynamics.

Key capabilities include:

• Support for 3D scans (HM3D, HSSD, MatterPort3D, Gibson, Replica) and CAD models (ReplicaCAD, YCB, Google Scanned Objects).
• Configurable sensors like RGB-D cameras and egomotion sensing.
• Robot simulation via URDF (e.g., Fetch, Franka, AlienGo).
• Rigid-body mechanics powered by Bullet physics.

It is designed to work alongside Habitat-Lab for high-level task definition and agent training.

The Habitat-Sim backend is implemented in C++ using the Magnum graphics middleware. Key architectural components include:

• Simulator: The central orchestrator that delegates resource management to the ResourceManager.
• ResourceManager: Responsible for loading and caching 3D environment data (triangle mesh geometry, textures, shaders) from various on-disk formats to ensure efficient memory usage.
• SceneGraphs: A structured representation of 3D environments. Environments are composed of SceneNodes representing distinct objects or regions.
• Agents and Sensors: These are instantiated by attaching them to specific SceneNodes within a SceneGraph.

This design allows for programmatic manipulation of object states and the ability to combine objects from different environments.

A SceneDataset is a metadata configuration structure that defines system parameters, links assets, and organizes them into scene descriptions for use in a Simulator.

Key components:

• Attributes: Objects detailing configurations for individual objects, stages, and scenes, imported from .json files.
• AttributeManager: APIs used to manage these attributes via C++ and Python.
• MetadataMediator: Aggregates all AttributeManagers and provides an API to swap the active SceneDataset. It can be passed into the Simulator constructor via SimulatorConfiguration or used independently for metadata management.

The Asset Viewer is a tool available as both a Jupyter notebook and a pure Python utility used to visualize how assets (like stages or objects) will be rendered in the Habitat-Sim engine. You can run the utility directly from the command line to inspect assets.

To run the asset viewer utility, use the following command:

python path/to/habitat-sim/examples/tutorials/nb_python/asset_viewer.py

$ python path/to/habitat-sim/examples/tutorials/nb_python/asset_viewer.py

The camera in Habitat uses a specific local coordinate frame. For many standard setups (including those used with ReplicaCAD objects), the axes are:

• Right: x+ (Red)
• Up: y+ (Green)
• Forward (into the scene): z- (the negation of the blue z+ vector)

To add new actions outside of the source code, implement a custom control functor by subclassing habitat_sim.agent.SceneNodeControl. The class must implement a __call__ method with the following signature:

def __call__(self, scene_node: habitat_sim.SceneNode, actuation_spec: habitat_sim.ActuationSpec):
    pass

• scene_node: The object being manipulated or controlled.
• actuation_spec: Contains parameters required by the control function.

For reference, existing implementations can be found in src_python/habitat_sim/agent/controls/default_controls.py.

You can bridge the gap between simulated and real-world observations by applying noise models to sensors. To do this, specify the noise model name in the sensor.SensorSpec.noise_model field and provide any necessary constructor arguments via the sensor.SensorSpec.noise_model_kwargs field.

Available noise models are accessible via a registry by their string names.

sensor_spec.noise_model = "RedwoodDepthNoiseModel"
sensor_spec.noise_model_kwargs = dict(noise_multiplier=5)

To build a SceneDataset, create configuration files using the following naming convention:

<datasetname>.scene_dataset_config.json

These files contain SceneDatasetAttributes templates which hold relative filepaths to linked assets and additional configurations.