DOCSEARCHbyAlgolia
Submit Docs

LeRobot: Machine Learning for Real-World Robotics

repository·main·Indexed 12 days ago

https://github.com/huggingface/lerobot

A PyTorch-based library designed to democratize real-world robotics through standardized models, datasets, and tools. LeRobot provides a hardware-agnostic interface for robot control and a scalable dataset format hosted on the Hugging Face Hub. Version 0.6.2 includes support for teleoperation with XR controllers and SO-101 arms, as well as pre-built Docker images for CPU and GPU environments.

Tokens
193.2K
Snippets
487
Records
726
Agent score
99%

What's inside LeRobot

  1. Overview of LeRobot Dataset Tools

    main

    LeRobot provides several utilities for manipulating and editing datasets, primarily implemented in lerobot.datasets.dataset_tools. Key capabilities include:

    • Delete Episodes: Remove specific episodes.
    • Split Dataset: Divide a dataset into multiple smaller subsets (e.g., train/val/test).
    • Merge Datasets: Combine multiple datasets with identical features.
    • Add/Remove Features: Modify the feature set of a dataset.
    • Modify Tasks: Change natural-language task descriptions.
    • Convert to Video: Transform image-based datasets into efficient video formats (MP4).
    • Re-encode Videos: Update encoder settings for existing video streams.
    • Show Info: View dataset summaries (episode count, frame count, etc.).

  2. Overview of the WALL-OSS model

    main

    WALL-OSS is a Vision-Language-Action (VLA) model ported to LeRobot for cross-embodiment robotic control. It is built upon the Qwen2.5-VL Vision-Language Model and supports multiple action prediction methods and multi-modal inputs.

    Key Technical Specifications:

    • Base Model: Qwen2.5-VL
    • Action Prediction Methods: Flow Matching (diffusion-based) or FAST (discrete tokens)
    • Architecture: Mixture of Experts (MoE) featuring action-specific routing
    • Supported Inputs: Vision (images/videos), Language, and Proprioception

  3. Overview of the lerobot-annotate pipeline

    main

    The lerobot-annotate tool automates the process of adding natural-language annotations to LeRobot datasets. It uses a Vision-Language Model (VLM) to watch episode videos and writes annotations directly into the dataset's parquet files (data/chunk-*/file-*.parquet).

    Specifically, it populates two language columns:

    • language_persistent: Stores long-term context like subtasks, plans, memory, and task augmentations.
    • language_events: Stores event-based annotations like interjections, speech tool-calls, and VQA (Visual Question Answering) pairs.

    The pipeline consists of three main modules (plan, interjections, and vqa) that all communicate with a single shared VLM server (such as vLLM or an OpenAI-compatible API). The process follows a flow of: Read Episodes $\rightarrow$ Module Processing $\rightarrow$ Staging (JSONL) $\rightarrow$ Validation $\rightarrow$ Writing (Parquet).

  4. What is EnvHub?

    main
    EnvHub is a feature in LeRobot that allows users to load robotics simulation environments directly from the Hugging Face Hub using a single line of code. Instead of installing monolithic libraries, environments are hosted as Git repositories on the Hub. This allows for dynamic loading, version tracking via Git, and easy discovery of community-shared simulation tasks without complex dependency management.

  5. What is VLA-JEPA architecture

    main

    VLA-JEPA is a Vision-Language-Action model consisting of three main components:

    1. Qwen3-VL backbone (Qwen3VLInterface): Fuses images and language instructions into context tokens.
    2. DiT-B action head (VLAJEPAActionHead): Performs flow-matching diffusion over the action chunk.
    3. V-JEPA2 world model (ActionConditionedVideoPredictor): A self-supervised video prediction module used during training to provide a prediction loss.

    Inference vs Training:

    • Training: Uses all three components. The world model predicts future V-JEPA2 frame embeddings based on action tokens, adding a regression loss to the action loss.
    • Inference: Only the Qwen backbone and the action head are used. The world model is not required for inference.

  6. What is SARM (Stage-Aware Reward Modeling)?

    main

    SARM is a video-based reward modeling framework designed for long-horizon robot manipulation tasks. Unlike standard behavior cloning which treats all frames equally, SARM learns a notion of task progress (a value from 0 to 1) based on video frames and task descriptions.

    This progress signal can be used for:

    1. Weighted Imitation Learning (RA-BC): High-progress frames receive more weight during policy training.
    2. Reinforcement Learning: Providing dense rewards for policy improvement.

    SARM uses a stage-aware architecture that jointly predicts high-level task stages and fine-grained progress within those stages. It uses temporal proportions ($\alpha\bar{k}$) to normalize progress across demonstrations of varying lengths, ensuring consistent task state mapping.

  7. What is VLA-JEPA and how does its architecture work?

    main

    VLA-JEPA is a Vision-Language-Action model ported to LeRobot. It integrates a language backbone, a world model for self-supervised training, and a diffusion-based action head.

    Key Components:

    • Qwen3-VL backbone (Qwen3VLInterface): Fuses images and language instructions into context tokens.
    • DiT-B action head (VLAJEPAActionHead): Performs flow-matching diffusion over action chunks.
    • V-JEPA2 world model (ActionConditionedVideoPredictor): Used only during training for self-supervised video prediction loss.

    Inference Note: At inference time, you only need the Qwen backbone and the action head; the world model is not required.

  10. What is an EnvTransition?

    main

    The EnvTransition is the fundamental, typed dictionary data structure that flows through all processors. It represents a complete robot-environment interaction and contains the following keys:

    • OBSERVATION: All sensor data (images, states, proprioception).
    • ACTION: The action to execute or that was executed.
    • REWARD: Reinforcement learning signal.
    • DONE/TRUNCATED: Episode boundary indicators.
    • INFO: Arbitrary metadata.
    • COMPLEMENTARY_DATA: Task descriptions, indices, padding flags, or inter-step data.

  11. What is ACT (Action Chunking with Transformers)?

    main

    ACT is a lightweight and efficient policy for imitation learning, specifically designed for fine-grained manipulation tasks. It is recommended as a starting point for beginners in LeRobot because it is data-efficient (often succeeding with only 50 demonstrations), fast to train (a few hours on a single GPU), and has low computational requirements (~80M parameters).

    Architecture

    ACT uses a transformer-based architecture consisting of:

    1. Vision Backbone: A ResNet-18 that processes images from multiple camera viewpoints.
    2. Transformer Encoder: Synthesizes camera features, joint positions, and a learned latent variable.
    3. Transformer Decoder: Generates a chunk of k future action sequences using cross-attention.

    Inputs and Outputs

    • Inputs: Multiple RGB images (e.g., wrist, front, or top cameras), current robot joint positions, and a latent style variable z (which is set to zero during inference).
    • Outputs: A chunk of k future action sequences.

  12. Understand the LeRobotDataset format

    main
    The LeRobotDataset is the standard data format used by all LeRobot scripts for reading and writing. It is designed to be episode-aware and supports on-the-fly decoding of video observations. It also supports seamless round-tripping to the Hugging Face Hub for storage and sharing.