DOCSEARCHbyAlgolia
Submit Docs

OpenPI: Open-Source Vision-Language-Action Models

repository·main·Indexed 12 days ago

https://github.com/physical-intelligence/openpi

A repository of open-source vision-language-action (VLA) models and packages for robotics, specifically focusing on the π (pi) series of models. OpenPI provides tools for running pre-trained robotic policies and fine-tuning them on custom datasets, with support for platforms including ALOHA (real and simulated), DROID, and the LIBERO benchmark.

Tokens
31.7K
Snippets
88
Records
122
Agent score
91%

What's inside OpenPI

  1. Configure LIBERO Server and Client arguments

    main

    You can customize the LIBERO environment by passing environment variables that control the server and client behavior.

    Server Configuration (SERVER_ARGS)

    Used to customize the loaded checkpoint. These arguments are passed to scripts/serve_policy.py.

    • --env LIBERO: Sets the environment.
    • --policy:checkpoint: Indicates a checkpoint is being provided.
    • --policy.config <config_name>: Specifies the policy configuration (e.g., pi05_libero).
    • --policy.dir <path>: Specifies the directory of the custom checkpoint (must be located in the top-level openpi/ directory).

    Client Configuration (CLIENT_ARGS)

    Used to customize the task suite. These arguments are passed to examples/libero/main.py.

    • --args.task-suite-name <name>: Specifies the LIBERO task suite (e.g., libero_10).
    # Example: Loading a custom checkpoint and running the libero_10 task suite
export SERVER_ARGS="--env LIBERO policy:checkpoint --policy.config pi05_libero --policy.dir ./my_custom_checkpoint"
export CLIENT_ARGS="--args.task-suite-name libero_10"

  2. Precision settings for JAX and PyTorch

    main

    JAX Precision

    • Inference: Most weights and computations use bfloat16, with select computations in float32 for stability.
    • Training: Defaults to mixed precision (weights/gradients in float32, activations/computations in bfloat16). You can switch to full float32 by setting dtype to float32 in the config.

    PyTorch Precision

    • Inference: Matches JAX (mostly bfloat16, some float32 for stability).
    • Training: Supports full bfloat16 (default) or full float32. Use the pytorch_training_precision config key to switch. Note that mixed precision is not yet supported in PyTorch.

  3. Pi0 Model Action Space and Proprioceptive State definitions

    main

    The pi0_base and pi0_fast_base models use specific dimension mappings for actions and proprioceptive states.

    Action Space Dimensions

    Dimensions are indexed from 0. Left and right are defined looking from behind the robot towards the workspace.

    • dim_0:dim_5: left arm joint angles
    • dim_6: left arm gripper position
    • dim_7:dim_12: right arm joint angles (bi-manual only)
    • dim_13: right arm gripper position (bi-manual only)
    • dim_14:dim_15: x-y base velocity (mobile robots only)

    Proprioceptive State

    The proprioceptive state uses the same definitions as the action space, with one exception: for mobile robots, the base x-y position (the last two dimensions) is not included in the proprioceptive state.

    Robot-Specific Variations

    • 7-DoF robots (e.g., Franka): Uses the first 7 dimensions for joint actions and the 8th dimension for the gripper action.
    • DROID: Uses the original DROID configuration: joint velocity actions in the first 7 dimensions and gripper actions in the 8th dimension, with a 15 Hz control frequency.

    General Conventions

    • Joint Angles: Expressed in radians. Position zero corresponds to the zero position reported by the robot's interface library (Note: ALOHA uses a slightly different convention).
    • Gripper Positions: Range [0.0, 1.0], where 0.0 is fully open and 1.0 is fully closed.
    • Control Frequencies:
      • 20 Hz: UR5e and Franka
      • 50 Hz: ARX and Trossen (ALOHA)
      • 15 Hz: DROID

  4. Customize DROID data filtering

    main

    The DROID dataset contains 'idle' timesteps (where the robot is not moving) that can degrade policy performance. The default training recipe uses a pre-computed list of non-idle indices.

    If you are using a custom version of the DROID dataset, you must regenerate these indices using examples/droid/compute_droid_nonidle_ranges.py. To use a custom filter, provide the path via the filter_dict_path argument in src/openpi/training/config.py.

  7. Start a remote policy server

    main

    You can run a remote policy server to perform inference on powerful GPUs off-robot. This helps separate the robot environment from the policy environment.

    To start a server using a predefined environment checkpoint, use the --env flag with one of DROID, ALOHA, or LIBERO.

    To start a server using a custom checkpoint (e.g., one you trained yourself), you must specify the policy:checkpoint identifier, the --policy.config name, and the --policy.dir path. The server defaults to port 8000.

    # Using a predefined environment
uv run scripts/serve_policy.py --env=DROID

# Using a custom checkpoint
uv run scripts/serve_policy.py policy:checkpoint --policy.config=pi0_fast_droid --policy.dir=gs://openpi-assets/checkpoints/pi0_fast_droid

  9. Train on your own Aloha dataset

    main

    To fine-tune models on your own ALOHA data, follow these steps:

    1. Convert Data: Convert your dataset to the LeRobot dataset v2.0 format using the provided script ./convert_aloha_data_to_lerobot.py.
    2. Configure Training: Define a training configuration that points to your custom dataset. You can use src/openpi/training/config.py as a template.
    3. Normalization (Important): When fine-tuning a base checkpoint, use the corresponding normalization stats from that checkpoint. In your config, specify the trossen asset_id and the path to the pretrained checkpoint's asset directory within the AssetsConfig.

  10. Install Docker and NVIDIA Toolkit on Ubuntu 22.04

    main

    If you are starting from scratch on an Ubuntu 22.04 host machine, you can use the provided convenience scripts to automate the installation of Docker and the NVIDIA container toolkit.

    # Use these scripts on Ubuntu 22.04 to automate setup
./scripts/docker/install_docker_ubuntu22.sh
./scripts/docker/install_nvidia_container_toolkit.sh