DOCSEARCHbyAlgolia
Submit Docs

Awesome-LLM-Robotics

repository·main·Indexed 26 days ago

https://github.com/gt-ripl/awesome-llm-robotics

A curated repository of research papers, code repositories, and project websites utilizing Large Language Models (LLMs) and Multi-Modal Models for Robotics and Reinforcement Learning (RL). The collection covers key areas including robotic planning, manipulation, navigation, reasoning, safety, red teaming, and simulation frameworks, as well as the VeRL-Omni library for RL training of diffusion and omni-modality models.

Tokens
4.9K
Snippets
4
Records
9
Agent score
39%

What's inside Awesome-LLM-Robotics

  1. Research papers for Robotic Manipulation

    main

    This section of the Awesome-LLM-Robotics repository provides a curated list of research papers, code repositories, and websites focused on Manipulation in the context of Large Language Models (LLMs) and Embodied AI. Key research areas include:

    • Vision-Action Pretraining: e.g., D2E.
    • Prompt-to-Action Mapping: e.g., Prompt2Act.
    • Reinforcement Learning for VLA: e.g., RL4VLA, LLM-TALE, Plan-Seq-Learn, ExploRLLM.
    • Spatial & Affordance Understanding: e.g., RoboSpatial, UniAff, ManipVQA, PhysObjects.
    • Task & Motion Planning: e.g., LLM-GROP, VOYAGER, DEPS.
    • Foundation Model Integration: e.g., CLIPort, Embodied-CLIP, VIMA.

    Each entry typically includes links to the original paper (arXiv), implementation code (PyTorch, TensorFlow), and project websites.

  2. Explore Reasoning research in LLM-Robotics

    main

    The repository contains a collection of research papers focused on reasoning capabilities in robotics using LLMs/VLMs, such as:

    • Spatial Reasoning: RoboTracer, RoboRefer, RoboSpatial, ReKep.
    • Failure Detection & Explanation: AHA, CoExp.
    • Task Planning & State Representation: SPINE, ELLMER, LLM-State, SayPlan, ProgPrompt.
    • Generalist Agents & Policies: RobotxR1, LEO, Robogen, LLaRP, RT-2, RT-1, PaLM-E.

  3. Explore Surveys in LLM-Robotics

    main

    The repository maintains a curated list of survey papers covering various aspects of LLMs in robotics, including:

    • Safety in Embodied AI
    • Robotic Language Grounding
    • Superalignment in Autonomous Driving
    • Neural Scaling Laws for Embodied AI
    • Automated Planning and Scheduling (APS)
    • Foundation Models for General-Purpose Robots
    • Language-conditioned Learning for Manipulation
    • Embodied Navigation

  4. Contribute papers to Awesome-LLM-Robotics

    main

    You can contribute to this repository by submitting pull requests or emailing the maintainers to add new papers regarding Large Language/Multi-Modal Models for Robotics/RL.

    When adding papers, ensure they follow these requirements:

    • Use reverse chronological order.
    • Follow the existing formatting strictly.

    For detailed instructions on the submission process, refer to the how-to-PR.md file.

  5. Explore Robot Navigation research

    main

    This section provides a curated list of research papers and codebases focused on robot navigation using language and vision models. Key topics include implicit language maps, commonsense-aware navigation, spatial understanding, and video-based planning.

    ### Instructions and Navigation
* **LAMP**: "LAMP: Implicit Language Map for Robot Navigation", *RA-L, 2025*. [[Paper](https://arxiv.org/abs/2602.11862)] [[Website](https://lab-of-ai-and-robotics.github.io/LAMP/)]
* **CANVAS**: "CANVAS: Commonsense-Aware Navigation System for Intuitive Human-Robot Interaction", *ICRA, Oct 2024*. [[Paper](https://arxiv.org/abs/2410.01273)] [[Code](https://github.com/worv-ai/canvas)] [[Website](https://worv-ai.github.io/canvas/)]
* **RoboSpatial**: "RoboSpatial: Teaching Spatial Understanding to 2D and 3D Vision-Language Models for Robotics", *CVPR, June 2025*. [[Paper](https://arxiv.org/abs/2411.16537)] [[Code](https://github.com/NVlabs/RoboSpatial)] [[Website](https://chanh.ee/RoboSpatial/)]
* **LLMxRobot**: "Autonomous Driving Systems with On-Board LLMs", *RSS, Apr 2025* [[Paper](https://www.roboticsproceedings.org/rss21/p140.pdf)][[Code](https://github.com/ForzaETH/LLMxRobot)][[Website](https://www.youtube.com/watch?v=4iGN1uBl4v4)]
* **GSON**: "GSON: A Group-based Social Navigation Framework with Large Multimodal Model", *arxiv, Sept 2024* [[Paper](https://arxiv.org/abs/2409.18084)]
* **Navid**: "NaVid: Video-based VLM Plans the Next Step for Vision-and-Language Navigation", *arxiv, Mar 2024* [[Paper](https://arxiv.org/abs/2402.15852)] [[Website](https://pku-epic.github.io/NaVid)]
* **OVSG**: "Context-Aware Entity Grounding with Open-Vocabulary 3D Scene Graphs", *CoRL, Nov 2023*. [[Paper](https://openreview.net/forum?id=cjEI5qXoT0)] [[Code](https://github.com/changhaonan/OVSG)] [[Website](https://ovsg-l.github.io/)]
* **VLMaps**: "Visual Language Maps for Robot Navigation", *arXiv, Mar 2023*. [[Paper](https://arxiv.org/abs/2210.05714)] [[Pytorch Code](https://github.com/vlmaps/vlmaps)] [[Website](https://vlmaps.github.io/)]
* "Interactive Language: Talking to Robots in Real Time", *arXiv, Oct 2022* [[Paper](https://arxiv.org/abs/2210.06407)] [[Website](https://interactive-language.github.io/)]
* **NLMap**:"Open-vocabulary Queryable Scene Representations for Real World Planning", *arXiv, Sep 2022*, [[Paper](https://arxiv.org/abs/2209.09874)] [[Website](https://nlmap-saycan.github.io/)]
* **ADAPT**: "ADAPT: Vision-Language Navigation with Modality-Aligned Action Prompts", *CVPR, May 2022*. [[Paper](https://arxiv.org/abs/2205.15509)]
* "The Unsurprising Effectiveness of Pre-Trained Vision Models for Control", *ICML, Mar 2022*. [[Paper](https://arxiv.org/abs/2203.03580)] [[Pytorch Code](https://github.com/sparisi/pvr_habitat)] [[Website](https://sites.google.com/view/pvr-control)]
* **CoW**: "CLIP on Wheels: Zero-Shot Object Navigation as Object Localization and Exploration", *arXiv, Mar 2022*. [[Paper](https://arxiv.org/abs/2203.10421)]
* **Recurrent VLN-BERT**: "A Recurrent Vision-and-Language BERT for Navigation", *CVPR, Jun 2021* [[Paper](https://arxiv.org/abs/2011.13922)] [[Pytorch Code](https://github.com/YicongHong/Recurrent-VLN-BERT)]
* **VLN-BERT**: "Improving Vision-and-Language Navigation with Image-Text Pairs from the Web", *ECCV, Apr 2020* [[Paper](https://arxiv.org/abs/2004.14973)] [[Pytorch Code](https://github.com/arjunmajum/vln-bert)]

  6. Explore LLM-based Robotic Planning research

    main

    This section of the Awesome-LLM-Robotics repository provides a curated list of research papers, code repositories, and project websites focused on Planning for embodied agents using Large Language Models. Topics include:

    • Multi-modal and Grounded Planning: e.g., FLARE, ViLa, LLM-Planner.
    • Bimanual Robot Task Planning: e.g., LLM+MAP, LABOR Agent.
    • Safety and Efficiency: e.g., SELP, CoPAL.
    • Long-Horizon Task Planning: e.g., FLTRNN, SayCanPay.
    • Behavior Tree Generation: e.g., BTGenBot, LLM-BRAIn.
    • Navigation and Movement: e.g., LM-Nav, Prompt2Walk, Beyond Text.
    • Human-Robot Interaction: e.g., Attentive Support.

    Developers can use these resources to find state-of-the-art methods for translating high-level language instructions into executable robotic plans, handling failures, and optimizing task execution.

  7. Use VeRL-Omni for RL training

    main

    VeRL-Omni is a library designed for easy, fast, and stable Reinforcement Learning (RL) training for diffusion and omni-modality models.

    ## Libraries and Tools
* **VeRL-Omni**: Easy, fast, and stable RL training for diffusion and omni-modality models. [[Code](https://github.com/verl-project/verl-omni)] [[Docs](https://verl-omni.readthedocs.io/en/latest/index.html)]

  8. Explore Robotics Simulation Frameworks

    main

    A collection of simulation environments and benchmarks for training and testing embodied AI, ranging from photo-realistic virtual worlds to specialized benchmarks for household activities and language-grounded tasks.

    ### Simulation Frameworks
* **UnrealZoo**: "UnrealZoo: Enriching Photo-realistic Virtual Worlds for Embodied AI", *ICCV Highlight, 2025*. [[Paper](https://arxiv.org/abs/2412.20977)] [[Code](https://github.com/UnrealZoo/unrealzoo-gym)] [[Website](https://unrealzoo.site/)]
* **ManiSkill3**: "ManiSkill3: GPU Parallelized Robotics Simulation and Rendering for Generalizable Embodied AI.", *arxiv, Oct 2024*. [[Paper](https://arxiv.org/abs/2410.00425)] [[Code](https://github.com/haosulab/ManiSkill)] [[Website](http://maniskill.ai/)]
 * **GENESIS**: "A generative world for general-purpose robotics & embodied AI learning.", *arXiv, Nov 2023*. [[Code](https://github.com/Genesis-Embodied-AI/Genesis)] 
 * **ARNOLD**: "ARNOLD: A Benchmark for Language-Grounded Task Learning With Continuous States in Realistic 3D Scenes", *ICCV, Apr 2023*. [[Paper](https://arxiv.org/abs/2304.04321)] [[Code](https://github.com/arnold-benchmark/arnold)] [[Website](https://arnold-benchmark.github.io/)]
 * **OmniGibson**: "OmniGibson: a platform for accelerating Embodied AI research built upon NVIDIA's Omniverse engine".*6th Annual Conference on Robot Learning, 2022*. [[Paper](https://openreview.net/forum?id=_8DoIe8G3t)] [[Code](https://github.com/StanfordVL/OmniGibson)] 
 * **MineDojo**: "MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge", *arXiv, Jun 2022*. [[Paper](https://arxiv.org/abs/2206.08853)] [[Code](https://github.com/MineDojo/MineDojo)] [[Website](https://minedojo.org/)] [[Open Database](https://minedojo.org/knowledge_base.html)]
 * **Habitat 2.0**: "Habitat 2.0: Training Home Assistants to Rearrange their Habitat", *NeurIPS, Dec 2021*. [[Paper](https://arxiv.org/abs/2106.14405)] [[Code](https://github.com/facebookresearch/habitat-sim)] [[Website](https://aihabitat.org/)]
 * **BEHAVIOR**: "BEHAVIOR: Benchmark for Everyday Household Activities in Virtual, Interactive, and Ecological Environments", *CoRL, Nov 2021*. [[Paper](https://arxiv.org/abs/2108.03332)] [[Code](https://github.com/StanfordVL/behavior)] [[Website](https://behavior.stanford.edu/)]
 * **iGibson 1.0**: "iGibson 1.0: a Simulation Environment for Interactive Tasks in Large Realistic Scenes", *IROS, Sep 2021*. [[Paper](https://arxiv.org/abs/2012.02924)] [[Code](https://github.com/StanfordVL/iGibson)] [[Website](https://svl.stanford.edu/igibson/)]
 * **ALFRED**: "ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks", *CVPR, Jun 2020*. [[Paper](https://arxiv.org/abs/1912.01734)] [[Code](https://github.com/askforalfred/alfred)] [[Website](https://askforalfred.com/)]
  * **BabyAI**: "BabyAI: A Platform to Study the Sample Efficiency of Grounded Language Learning", *ICLR, May 2019*. [[https://arxiv.org/abs/1810.08272)] [[Code](https://github.com/mila-iqia/babyai/tree/iclr19)]

  9. Research Safety and Red Teaming in Robotics

    main

    This section lists research regarding the safety, risks, and adversarial vulnerabilities of LLM-controlled robots. It covers topics such as backdoor attacks on VLA models, jailbreaking, safety guardrails, and the risk of robots enacting discrimination or violence.

    ### Safety, Risks, Red Teaming, and Adversarial Testing
* **BadVLA**: "Towards Backdoor Attacks on Vision-Language-Action Models via Objective-Decoupled Optimization", *arXiv, May 2025*. [[Paper](https://www.arxiv.org/abs/2505.16640)] [[Code](https://github.com/Zxy-MLlab/BadVLA)] [[Website](https://badvla-project.github.io/)]
* **RoboPAIR**: "Jailbreaking LLM-Controlled Robots", *International Conference on Robotics and Automation (ICRA) May 2025*. [[Paper](https://arxiv.org/abs/2410.13691)] [[Website](https://robopair.org/)
* **RoboGuard**: "Safety Guardrails for LLM-Enabled Robots", *arXiv, April 2025*. [[Paper](https://arxiv.org/abs/2503.07885)] [[Website](https://robo-guard.github.io/)
* **Safe LLM-Controlled Robots with Formal Guarantees via Reachability Analysis** *arXiv, Mar 2025* [[arXiv](https://arxiv.org/abs/2503.03911)] [[Code](https://github.com/TUM-CPS-HN/SafeLLMRA)
* **LLM-Driven Robots Risk Enacting Discrimination, Violence, and Unlawful Actions**: *arXiv, Jun 2024*. [[Paper](https://arxiv.org/abs/2406.08824)
* **Highlighting the Safety Concerns of Deploying LLMs/VLMs in Robotics**: *arXiv, Feb 2024*. [[Paper](https://arxiv.org/abs/2402.10340)
* **Robots Enact Malignant Stereotypes**: *FAccT, Jun 2022*. [[arXiv](https://arxiv.org/abs/2207.11569)] [[DOI](https://doi.org/10.1145/3531146.3533138)] [[Code](https://github.com/ahundt/RobotsEnactMalignantStereotypes)] [[Website](https://sites.google.com/view/robots-enact-stereotypes/home)
* **Exploring the Adversarial Vulnerabilities of Vision-Language-Action Models in Robotics** *arXiv, Nov 2024* [[arXiv](https://arxiv.org/abs/2411.13587)] [[Code](https://github.com/William-wAng618/roboticAttack)] [[Website](https://vlaattacker.github.io/)