DOCSEARCHbyAlgolia
Submit Docs

OpenMind OM1 Documentation

repository·main·Indexed 25 days ago

https://github.com/openmind/om1

OM1 is a modular AI runtime for creating and deploying multimodal AI agents across digital environments and physical robots, such as humanoids and quadrupeds. It supports diverse inputs including web data, camera feeds, and LIDAR, and integrates with various LLMs and hardware via plugins. The documentation covers system installation, API reference, full autonomy workflows, MCP tool integration, and monitoring via Grafana and Prometheus.

Tokens
63.6K
Snippets
157
Records
337
Agent score
83%

What's inside OM1

  1. Overview of OM1 ROS2 SDK REST APIs

    main

    The OM1 ROS2 SDK provides two distinct REST APIs for remote control and monitoring of the robot. Note that these APIs are part of the Premium Features available through the Enterprise Plan.

    APIPortDescription
    Orchestrator API5000System orchestration and high-level control (SLAM, Nav2, charging, patrol, maps)
    Nav2 API5001Direct navigation control and real-time monitoring (pose, goals, localization)

  3. Overview of OM1 Capabilities

    main

    OM1 is an open-source agent runtime system designed to deploy AI personas across both digital and physical environments (e.g., cloud services and robot hardware like Quadrupeds, TurtleBot 4, or Humanoids).

    Key capabilities include:

    • Modular Architecture: Uses natural language data buses for human-intelligible communication.
    • Language: Built entirely in Go for maintainability and extensibility.
    • Extensibility: Easy integration of new data inputs and support for new hardware via plugins.
    • Middleware Support: Compatible with ROS2, Zenoh, and CycloneDDS.
    • Preconfigured Endpoints: Out-of-the-box support for Text-to-Speech, various LLMs (OpenAI gpt-4o, DeepSeek, Gemini, Openrouter, Near AI, Ollama), and multiple VLMs.
    • Simulators: Supports Gazebo and upcoming support for Isaac Sim.

  4. Understand the OM1 Lifecycle Conceptual Model

    main

    The OM1 lifecycle manages the operational boundaries of a Mode (a functional state like Guard, SLAM, or Navigation). It ensures predictable transitions and controlled execution through a hierarchy of Stages and Hooks.

    • Lifecycle: Controls the overall flow of a mode.
    • Stage: A logical phase within the lifecycle (e.g., startup, entry, active).
    • Hook: A programmable event point that executes specific actions at key stages.
    • Mode: The operational context or behavior being governed.

  5. System Requirements for Gazebo Simulator

    main

    To run the Gazebo Quadruped Simulation and Control smoothly, the following hardware and software specifications are recommended:

    ComponentMinimumRecommendedIdeal
    CPUIntel i7 (10th gen) / AMD Ryzen 7 (8 cores)Intel i9 (12th gen+) / AMD Ryzen 9 (12 cores)AMD Ryzen 9 7950X / Intel i9-13900K (16+ cores, 24+ threads)
    RAM16 GB32 GB64 GB
    GPUNVIDIA GTX 1660 Ti (6 GB VRAM)NVIDIA RTX 3070 / RTX 4060 Ti (8-12 GB VRAM)NVIDIA RTX 4080/4090 (16+ GB VRAM) with CUDA 11.8+
    OSUbuntu 22.04Ubuntu 22.04Ubuntu 22.04

    Storage: At least 128 GB SSD is recommended.

  6. Understand the Hybrid Localization Architecture

    main

    The hybrid localization system provides fully automatic, robust robot positioning by combining three technologies to handle global localization, continuous tracking, and environmental changes:

    1. Visual Place Recognition (VPR): Uses camera images to provide a rough location hint (neighborhood estimate) to break environmental symmetry.
    2. Correlative Scan Matching (CSM): Uses LiDAR data to perform a brute-force search for the exact position, guided by the VPR hint.
    3. Nav2 AMCL: A standard ROS 2 particle filter used for smooth, continuous position tracking during active navigation.

    The system is orchestrated by a Hybrid Localization Manager which acts as a quality gate, validating initial poses and monitoring AMCL's health to trigger automatic recoveries without human intervention.

  7. Understand the OM1 Project Structure

    main

    The OM1 repository is organized into several key directories that separate the main entry point, core logic, and extensible plugins:

    • cmd/: Contains the main entry point (main.go).
    • config/: Stores agent configuration files.
    • internal/: Contains core packages including actions, backgrounds, fuser, hooks, inputs, llm, runtime, and zenoh integration.
    • plugins/: Contains modular implementations for actions (e.g., speak, arm_g1), backgrounds, inputs (e.g., face_presence), and llm providers (e.g., OpenAI, Gemini, Ollama).
    • Makefile: The build system for the project.

  8. Understand the OM1 Core Architecture Layers

    main

    OM1 is organized into a layered architecture that transforms raw sensor data into high-level robot actions. The data flow follows this sequence:

    1. Raw Sensor Layer: Collects raw inputs (Vision, Sound, Battery/System, Location/GPS, LIDAR).
    2. AI Captioning and Compression Layer: Converts raw data into natural language descriptions using VLMs (Vision), ASR (Audio), Platform State, Spatial/NAV, and 3D environment models.
    3. Natural Language Data Bus (NLDB): A centralized bus that manages the structured natural language data generated by the captioning layer.
    4. State Fuser: Combines multiple short NLDB inputs into a single, unified paragraph to provide situational awareness.
    5. Multi AI Planning/Decision Layer: Processes fused data using specialized LLMs to make decisions.
    6. Hardware Abstraction Layer (HAL): Translates high-level AI decisions into specific hardware commands (e.g., ROS2, CycloneDDS, or Zenoh).
    7. Robot Actions: The final execution of movement policies, action models, or foundational models.

  9. Understand OM1 operational modes

    main
    OM1 uses 'Modes' to define its primary behavioral state and functional context. Modes adjust how the system perceives the environment, processes inputs, and prioritizes tasks. Modes can be selected by the user via voice commands or the UI. Switching modes allows the system to transition between social interaction, exploration, patrol, or autonomous operation.

  10. Understand Background Tasks in OM1

    main

    The Background Tasks system allows for running continuous, long-running processes (like sensor data collection or state monitoring) independently of the main control loop.

    Key components include:

    • BackgroundOrchestrator: Manages task lifecycles (startup and graceful shutdown).
    • Plugins: Tasks are loaded dynamically from the backgrounds/plugins directory.
    • Threading: Each task runs in its own thread using thread pooling for efficiency.

    Commonly available background tasks include:

    • GPS (GPS data processing)
    • ODOM (Odometry data)
    • RF Mapper (RF signal mapping)
    • RPLIDAR (RPLIDAR sensor interface)
    • RTK (Real-Time Kinematic positioning)
    • Unitree Go2 State (Unitree Go2 robot state management)

  11. Understand the OM1 Full Autonomy Architecture

    main

    The OM1 full autonomy stack is a modular, containerized system where services communicate through well-defined interfaces.

    Data Flow Overview:

    1. Sensors: Hardware data flows into the ROS2 SDK, which publishes structured topics.
    2. Intelligence: OM1 consumes these topics and user input, processing them through an LLM.
    3. Action: OM1 emits action commands back to the robot.
    4. Parallel Pipelines: The Video Processor handles media in parallel, while the OM1 Avatar renders the robot's state on the display.

  12. Compare Google ASR API versions

    main

    The Google ASR service provides real-time speech-to-text via WebSockets. There are two versions available:

    V2 (Recommended)

    • Endpoints: /api/core/google/asr or /api/core/google/asr/v2
    • Model: Chirp 3 (high accuracy, 100+ languages)
    • Features: Voice activity detection (speech_start, speech_end, end_of_utterance), configurable timeouts, and multi-language support.
    • Best for: New integrations requiring high accuracy and advanced speech detection.

    V1 (Legacy)

    • Endpoint: /api/core/google/asr/v1
    • Model: Standard Google Cloud Speech-to-Text v1.
    • Features: Standard recognition, stable, alternative language support.
    • Best for: Backward compatibility with existing v1 implementations.