DOCSEARCHbyAlgolia
Submit Docs

Awesome VLA Study: Vision-Language-Action Robot Foundation Models

repository·main·Indexed 43 days ago

https://github.com/milkclouds/awesome-vla-study

A structured curriculum and reading guide for researchers and developers to learn Vision-Language-Action (VLA) robot foundation models. The study is organized into six phases, covering generative model foundations, early RFMs like RT-1 and RT-2, modern architectures such as π0 and InternVLA-M1, data scaling (OXE), efficient inference (SmolVLA), and advanced topics including RL fine-tuning, reasoning, and world models.

Tokens
2.8K
Snippets
1
Records
12
Agent score
7%

What's inside Awesome VLA Study

  1. VLA Study Curriculum Overview

    main

    The study is organized into six phases covering the progression from generative model foundations to advanced robot foundation models (RFMs):

    PhaseWeeksTopicReadings
    Phase 1W1–3Generative Model FoundationsMIT 6.S184 course
    Phase 2W4–5Early Foundation RFMs & Robot PolicyRT-1, RT-2, Octo, OpenVLA, BeT, Diffusion Policy, ACT
    Phase 3W6–7Current RFM ArchitecturesCogACT, GR00T N1, X-VLA, π0, InternVLA-M1
    Phase 4W8–9Data ScalingOXE, AgiBot World, UMI, VITRA, Human to Robot Transfer
    Phase 5W10–11Efficient Inference & Dual-SystemRTC, SmolVLA, Helix, Fast-in-Slow
    Phase 6W12–14RL Fine-tuning, Reasoning & World ModelHIL-SERL, SimpleVLA-RL, π*0.6, CoT-VLA, ThinkAct, Fast-ThinkAct, UniVLA, Cosmos Policy, DreamZero

  2. Prerequisites for the VLA Study

    main

    To effectively follow the VLA (Vision-Language-Action) study guide, you should have a foundation in the following areas:

    • Basic Probability & Optimization: Sufficient knowledge to follow derivations for ELBO (Evidence Lower Bound) and score matching.
    • Deep Learning Fundamentals: Understanding of Transformers, attention mechanisms, and tokenization.

    If you are starting from scratch, the guide recommends MIT 6.S191 — Intro to Deep Learning for an intensive bootcamp covering CNNs, Transformers, and generative models.

  3. Phase 5: Efficient Inference & Dual-System (Weeks 10–11)

    main

    This phase explores methods to make VLA models faster and more efficient for real-time control.

    Week 10: Fast-Acting VLA

    Focuses on model compression and inference optimization.

    #PaperLinkKey Topic
    18SmolVLA2506.01844450M params (~1/7 of π0), model compression + async inference
    19RTC2506.07339Async inference — freezing + inpainting, no retraining needed

    Week 11: Dual-System VLA

    Separates slow reasoning (VLM) from fast execution (lightweight policy) at different frequencies.

    #PaperLinkKey Topic
    20Helixfigure.ai/news/helixS2: 7B VLM @7-9Hz, S1: 80M @200Hz, humanoid
    21Fast-in-Slow2506.01953Integrated dual-system, end-to-end trainable

  4. Recommended Weekly Study Format

    main

    For groups or individuals participating in the study, the following weekly format is recommended to ensure deep understanding:

    • Paper presentation: 1–2 participants per week. Each presentation should last approximately 30 minutes and cover the architecture, training process, and key results of the paper.
    • Discussion: A 15–20 minute session to compare design choices across the week's papers and discuss limitations or open questions.

  5. Phase 3: Current RFM Architectures (Weeks 6–7)

    main

    This phase explores modern Robot Foundation Model (RFM) architectures, specifically how Vision-Language Models (VLMs) interface with action heads.

    Week 6: VLM + Action Head

    These architectures use only the VLM's last hidden state to drive a separate action head.

    #PaperLinkKey Topic
    8CogACT2411.19650VLM + DiT action head, action token learning
    9GR00T N12503.147342B diffusion transformer, whole-body humanoid control
    10X-VLA2510.10274Soft prompts for cross-embodiment, Florence-Large + flow matching

    Week 7: VLM + Action Expert

    Unlike Week 6, these architectures use action experts that access VLM internal hidden states.

    #PaperLinkKey Topic
    11π02410.24164Flow matching + action expert accessing VLM intermediate features
    12InternVLA-M12510.13778Spatial grounding → action generation, AR-based

    Architectural Note: The architecture of π0 is based on Transfusion (Zhou et al., 2024), which combines Autoregressive (AR) and diffusion modeling in a single transformer.

  6. Phase 4: Data Scaling (Weeks 8–9)

    main

    This phase covers the datasets and collection methods required to scale robot policies.

    Week 8: Large-Scale Robot Datasets

    #PaperLinkKey Topic
    13Open X-Embodiment (OXE)2310.088641M+ trajectories, 22 embodiments, standardized data format
    14AgiBot World2503.066691M+ trajectories, 217 tasks, 5 deployment scenarios

    Data Formats Reference:

    • Recording-oriented: rosbag (ROS 1), mcap (vendor-neutral, ROS 2 default).
    • Training-oriented: RLDS (TensorFlow/OXE standard), LeRobotDataset (HuggingFace, Parquet + video).

    Week 9: Data Collection Methods

    #PaperLinkKey Topic
    15UMI2402.10329Robot-free SE(3) data collection via handheld gripper
    16VITRA2510.21571Human video → VLA training data (1M episodes from egocentric human videos)
    17Human to Robot Transfer2512.22414Human video → robot transfer emerges with VLA scaling

  7. Explore RL Fine-tuning, Reasoning, and World Models for VLA

    main

    Phase 6 of the study guide covers advanced topics in Vision-Language-Action (VLA) models, specifically focusing on Reinforcement Learning (RL) fine-tuning, reasoning capabilities, and world modeling.

    RL Fine-tuning & Human-in-the-Loop (Week 12)

    Focuses on sample-efficient training and reward structures:

    • HIL-SERL: Human-in-the-loop RL for sample-efficient real-world training.
    • SimpleVLA-RL: RL fine-tuning for autoregressive VLA using outcome-based rewards.
    • π*0.6: Advantage-conditioned RL for flow-based VLA that learns from suboptimal data.

    Reasoning VLA (Week 13)

    Explores how models can 'think' before acting using different representation methods:

    • CoT-VLA: Uses visual chain-of-thought (predicting future images) before action.
    • ThinkAct: Decouples reasoning from execution; uses RL to ground plan quality in task success rather than language supervision.
    • Fast-ThinkAct: Uses latent distillation to compress reasoning, preserving planning capacity while achieving ~10× speedup.

    World Models (Week 14)

    Covers how to integrate world knowledge into robot policies:

    • UniVLA: Uses world modeling as a training regularizer (no world prediction required at inference).
    • Cosmos Policy: Uses a pretrained video foundation model as the robot policy backbone.
    • DreamZero: A World Action Model that performs joint world and action generation in latent space.

  8. Phase 1: Generative Model Foundations (Weeks 1–3)

    main

    This phase focuses on the mathematical and generative foundations required for VLA work, primarily using the MIT 6.S184 — Introduction to Flow Matching and Diffusion Models course.

    Week 1: ODE/SDE Foundations & Diffusion Models

    • Topics: ODE/SDE basics, forward/reverse processes, conditional/marginal probability paths.
    • Hands-on: Lab 1 involves SDE simulation.

    Week 2: Flow Matching, Score Matching & Training

    • Topics: Flow Matching, Score Matching, guidance, and classifier-free guidance.
    • Hands-on: Labs 2–3 involve building a toy diffusion model from scratch.

    Week 3: Generative Robotics & Review

    • Topics: Diffusion models for robotics (Guest lecture by Benjamin Burchfiel, Toyota Research) and optional generative protein design.

  9. Recommended courses for VLA prerequisites

    main

    The following courses are recommended to build the foundational knowledge required for the VLA study, specifically focusing on those with recent (2023+) video lectures available on YouTube.

    AreaCourseInstructorLinkNotes
    DL FundamentalsMIT 6.S191: Intro to Deep LearningAlexander Aminiintrotodeeplearning.com1-week bootcamp (10 lectures) — CNN, Transformer, generative models, RL
    Andrej Karpathy: Neural Networks: Zero to HeroAndrej Karpathykarpathy.ai/zero-to-hero.htmlBackprop → GPT, build everything from scratch in code
    VisionStanford CS231n: DL for Computer VisionFei-Fei Li et al.cs231n.stanford.eduThe canonical CV course — backprop to detection/segmentation/video
    NLP / TransformersStanford CS224n: NLP with Deep LearningChristopher Manningweb.stanford.edu/class/cs224nWord vectors → Transformers → LLMs
    RLUC Berkeley CS285: Deep RLSergey Levinerail.eecs.berkeley.edu/deeprlcoursePolicy gradients, Q-learning, model-based & offline RL

  10. Phase 2: Early Foundation Robot Models & Robot Policy (Weeks 4–5)

    main

    This phase covers the transition from pure robotics transformers to Vision-Language-Action (VLA) paradigms and core robot policies.

    Week 4: Early Foundation Robot Models

    Focuses on the evolution from RT-1 (Robotics Transformer) to RT-2 (VLA paradigm), and open-source models like Octo and OpenVLA.

    #PaperLinkKey Topic
    1RT-1: Robotics Transformer2212.06817First large-scale Robotics Transformer (no VLM)
    2RT-2: Vision-Language-Action Models2307.15818VLM backbone → VLA paradigm
    3Octo2405.12213Open-source generalist policy, modular design, pretrained on OXE (no VLM)
    4OpenVLA2406.09246First open-source VLM-based VLA

    Week 5: Core Robot Policies

    Focuses on solving the multimodal action problem using different approaches.

    #PaperLinkKey Topic
    5Behavior Transformers (BeT)2206.11251Multimodal action discretization, k-means + offset
    6Diffusion Policy2303.04137Diffusion for robot control, action sequence prediction
    7ACT/ALOHA2304.13705Action Chunking Transformer, CVAE, bimanual