DOCSEARCHbyAlgolia
Submit Docs

MolmoAct2 Documentation

repository·main·Indexed 20 days ago

https://github.com/allenai/molmoact2

An open family of action reasoning models for robot control and real-world deployment, built on the Molmo2-ER vision-language backbone. Includes documentation for the molmoact2-policy-server (v0.1.1), Bimanual YAM evaluation setup, and a video benchmark for evaluating encoding/decoding performance for robotics policies using codecs like libx264, libx265, and libsvtav1.

Tokens
133.6K
Snippets
367
Records
493
Agent score
71%

What's inside MolmoAct2

  1. Overview of MolmoAct2

    main
    MolmoAct2 is a family of open action reasoning models designed for robot control and real-world deployment. It utilizes the Molmo2-ER embodied-reasoning vision-language backbone, incorporates robot state and action modeling, and uses a flow-matching continuous action expert for closed-loop manipulation. The project provides base checkpoints for training, fine-tuned policies for deployment, and various robotics datasets.

  2. Overview of π₀ (pi0) Vision-Language-Action models

    main

    π₀ (pi0) is a Vision-Language-Action (VLA) model designed for general robot control. This repository provides a Hugging Face port adapted from the OpenPI project by Physical Intelligence.

    There are two primary model variants available:

    π₀

    • Time Conditioning: Concatenates time with actions using action_time_mlp_* layers.
    • AdaRMS: Not used.
    • Tokenizer Length: 48 tokens.
    • Discrete State Input: False (utilizes a state_proj layer instead).
    • Parameter Count: Higher, as it includes state embedding.

    π₀.₅

    • Time Conditioning: Uses time_mlp_* for AdaRMS conditioning.
    • AdaRMS: Used in the action expert.
    • Tokenizer Length: 200 tokens.
    • Discrete State Input: True.
    • Parameter Count: Lower, as it does not include state embedding.

  3. Overview of π₀.₅ (pi05) Vision-Language-Action Model

    main

    π₀.₅ (pi05) is a Vision-Language-Action (VLA) model designed for open-world generalization. This specific implementation is a Hugging Face port adapted from the OpenPI repository by Physical Intelligence.

    Key technical differences between the original π₀ and this π₀.₅ version include:

    • Time Conditioning: π₀.₅ uses time_mlp_* for AdaRMS conditioning, whereas π₀ concatenates time with actions via action_time_mlp_*.
    • AdaRMS: π₀.₅ utilizes AdaRMS in the action expert; π₀ does not.
    • Tokenizer Length: π₀.₅ uses a 200-token tokenizer, compared to 48 tokens in π₀.
    • State Input: π₀.₅ supports discrete state input, while π₀ uses a state_proj layer.
    • Parameter Efficiency: π₀.₅ has a lower parameter count because it does not include a state embedding.

  7. What is HIL-SERL and how does it work?

    main

    HIL-SERL (Human-in-the-Loop Sample-Efficient Reinforcement Learning) is a workflow designed to train robot policies on real hardware efficiently. It combines human demonstrations with online reinforcement learning and human interventions.

    Key components include:

    1. Offline demonstrations & reward classifier: Uses a small set of human teleoperation episodes and a vision-based success detector to provide a shaped starting point.
    2. On-robot actor/learner loop: Employs a distributed Soft Actor Critic (SAC) learner that updates the policy while an actor explores on the physical robot. Humans can intervene via a gamepad or keyboard to correct unsafe or unproductive behaviors.
    3. Safety & efficiency tools: Utilizes joint/end-effector (EE) bounds, crop region of interest (ROI) preprocessing, and WandB monitoring to ensure hardware safety and data quality.

  8. What is an EnvTransition?

    main

    EnvTransition is the fundamental, typed dictionary data structure that flows through all processors. It represents a complete robot-environment interaction and contains the following keys:

    • OBSERVATION: All sensor data (images, states, proprioception)
    • ACTION: The action to execute or that was executed
    • REWARD: Reinforcement learning signal
    • DONE/TRUNCATED: Episode boundary indicators
    • INFO: Arbitrary metadata
    • COMPLEMENTARY_DATA: Task descriptions, indices, padding flags, inter-step data

  9. What is WALL-OSS (WallX)?

    main

    WALL-OSS is an open-source foundation model for embodied intelligence. It uses a tightly-coupled Mixture-of-Experts (MoE) architecture that integrates discrete and continuous action modeling.

    Key technical components include:

    • Uni-CoT (Unified Cross-Level Chain-of-Thought): A framework unifying high-level reasoning, sub-task decomposition, and fine-grained action synthesis.
    • Two-stage training:
      1. Inspiration stage: Uses discrete action priors for spatial and semantic alignment.
      2. Integration stage: Uses flow matching for high-frequency continuous control.
    • MoE action heads: Dynamically activates experts based on the task phase.

  10. What is WALL-OSS?

    main
    WALL-OSS is a Vision-Language-Action (VLA) model designed for cross-embodiment robotic control. This specific repository provides the Hugging Face port of the model. It is built upon the Qwen2.5-VL Vision-Language Model and utilizes either Flow Matching (diffusion) or FAST (discrete tokens) for action prediction. The architecture employs a Mixture of Experts (MoE) approach with action-specific routing and supports multi-modal inputs including vision (images/videos), language, and proprioception.

  11. What is ACT (Action Chunking with Transformers)?

    main

    ACT is a lightweight and efficient imitation learning policy designed for fine-grained manipulation tasks. It is highly recommended for beginners due to its fast training time, low computational requirements (~80M parameters), and high data efficiency (often succeeding with only 50 demonstrations).

    Architecture

    ACT utilizes a transformer-based architecture consisting of:

    1. Vision Backbone: A ResNet-18 that processes images from multiple camera viewpoints.
    2. Transformer Encoder: Synthesizes camera features, joint positions, and a learned latent variable.
    3. Transformer Decoder: Generates coherent action sequences using cross-attention.

    Inputs and Outputs

    • Inputs: Multiple RGB images (e.g., wrist, front, or top cameras), current robot joint positions, and a latent style variable z (which is set to zero during inference).
    • Outputs: A chunk of k future action sequences.

  12. What is SARM (Stage-Aware Reward Modeling)?

    main

    SARM is a video-based reward modeling framework designed for long-horizon robot manipulation tasks. Unlike standard behavior cloning which treats all frames equally, SARM learns a generalizable notion of task progress (a signal from 0 to 1).

    This progress signal can be used for:

    1. Weighted imitation learning (RA-BC): High-progress frames receive more weight during policy training.
    2. Reinforcement learning: Providing dense rewards for policy improvement.

    SARM uses a stage-aware architecture that jointly predicts high-level task stages and fine-grained progress within those stages. It normalizes progress across variable-length demonstrations using dataset-level temporal proportions ($\alpha\bar{k}$), ensuring consistent task states map to consistent progress values.