Open X-Embodiment Documentation

The RT-1-X model operates on the following input and output spaces:

Observation Space

The model accepts:

• An RGB image from the robot workspace camera.
• A task string describing the intended robot task.

Note: The model does not currently support wrist cameras, in-hand cameras, or depth information.

Action Space

The action space consists of seven variables for gripper movement:

• x, y, z (position)
• roll, pitch, yaw (orientation)
• opening of the gripper

Each variable can represent an absolute value, a delta change, or a velocity.

Open X-Embodiment provides robotic data in a unified format using the RLDS episode format. Each dataset is a sequence of episodes.

To get started with visualization and data batching, use the provided Dataset Colab.

A full list of included datasets and their metadata can be found in the dataset spreadsheet.

To use the JAX checkpoint with the flax checkpoint loader in rt1_inference_example.py, download the checkpoints using gsutil:

If you encounter the following error when running tfds.load({dataset_name}):

tensorflow_datasets.core.registered.DatasetNotFoundError: Dataset {dataset_name} not found.

You can resolve this by manually downloading the dataset to your ~/tensorflow_datasets/ directory using gsutil:

After manual download, the standard tfds.load({dataset_name}) command will work correctly.