DOCSEARCHbyAlgolia
Submit Docs

Mosaico Documentation

website·Indexed 32 days ago

https://docs.mosaico.dev

Mosaico is a platform for data ingestion and management, featuring Mosaico Alchemy for ready-to-use ingestion pipelines and a Manipulation Pack for robotic manipulation datasets. It provides a resource-oriented CLI for managing sequences, topics, and connection profiles, and implements administrative protocols on top of Apache Arrow Flight. The platform offers a Python SDK, with C++ and Rust SDKs in active design, and supports access control via a structured API key system.

Tokens
54.3K
Snippets
196
Records
340
Agent score
57%

What's inside Mosaico

  1. Overview of Mosaico Alchemy data ingestion

    Mosaico Alchemy is a collection of ready-to-use data ingestion pipelines designed for Physical AI and Robotics. It uses a system of 'Packs'—each targeting a specific domain (such as Robotic Manipulation)—to translate various heterogeneous dataset formats into the standardized ontology required by the Mosaico SDK.

  2. Overview of Mosaico Alchemy and Data Ingestion

    Mosaico Alchemy provides a coherent representation for robot data coming from disparate sources (such as ROS .bag files, HDF5 archives, and custom binary formats). It abstracts away the manual work of timestamp reconciliation, coordinate frame alignment, and schema mapping by bringing data under a single ontology. This allows users to run consistent queries across multiple collection pipelines and integrate data directly into training pipelines without writing custom ingestion code for supported formats.

  3. Overview of Mosaico data ingestion capabilities

    Mosaico's data ingestion system is engineered for write-heavy workloads. It is designed to absorb high-bandwidth sensor data—such as 4K video streams or high-frequency Lidar point clouds—while ensuring that this data flow does not contend with administrative traffic.

  4. Understand the role of the Mosaico Daemon (mosaicod)

    The Mosaico Daemon, also known as mosaicod, serves as the central engine of the Mosaico data platform. It operates on a client-server model, acting as a high-performance arbiter between high-level applications and the low-level storage infrastructure. Its primary responsibilities include ensuring that robotics data is strictly typed, atomically stored, and efficiently retrievable.

  5. Use Connection Profiles to manage Mosaico environments

    A Connection Profile in Mosaico is a named, saved configuration containing a host, a port, and optional API keys and TLS settings. Profiles are required for any mosaico command that interacts with the platform, specifically those under the sequence and topic command groups. Using profiles allows developers to switch between environments (e.g., dev, staging, prod) without manually specifying connection flags for every command invocation.

  6. Understand and use Mosaico Sessions for data versioning

    Mosaico uses 'Sessions' instead of traditional versioning to manage data evolution and updates. A session is an immutable data layer that acts as a logical container for topics uploaded together as a single unit. This architecture prevents ambiguity in queries and allows for high-concurrency environments where multiple independent workloads (e.g., data cleaning and real-time logging) can contribute data to the same sequence without data races or state corruption.

    Key hierarchy:

    • Sequence: The primary data container.
    • Session: An immutable layer of data within a sequence.
    • Topic: A specific instance of an ontology model within a session.

  7. Mosaico Platform Overview

    Mosaico is an open-source data platform designed for Robotics and Physical AI. It replaces linear message storage (e.g., ROS bags) with a structured, queryable archive optimized for high-throughput multi-modal sensor data. The platform uses a code-first approach, exposing ingestion, retrieval, and querying via an SDK rather than a proprietary query language.

  8. Capture fault snapshots using ros2_medkit

    ros2_medkit is a ROS 2 node designed for fault forensics. It monitors /diagnostics topics and maintains a ring buffer of recent sensor data. When a fault is confirmed, it persists only the relevant time window around the event as an .mcap snapshot. These snapshots are ingested into Mosaico with structured metadata, including robot ID, fault code, severity, and timestamp, and can be queried via the standard Mosaico SDK.

  9. Understand the mosaicod architectural design

    mosaicod is built on the Apache Arrow Flight protocol and gRPC, utilizing Apache Arrow columnar data to enable high-throughput data exchange. This architecture provides three primary benefits:

    1. Zero-Copy Serialization: Data is transmitted in the same columnar format used by tools like pandas and Polars, removing CPU overhead from serialization/deserialization.
    2. Parallelized Transport: Data transfer can be striped across multiple connections to maximize bandwidth.
    3. Snapshot-Based Schema Enforcement: A rigorous schema handshake validates data against specific schema snapshots stored with the sequence, avoiding data type guessing.

  10. Understand the purpose and benefits of the mosaicod daemon

    The mosaicod daemon is a dedicated, persistent data management layer designed specifically for robotics data profiles. It sits between data producers and the disk to solve several infrastructure challenges:

    1. Optimized Storage & Retrieval: Instead of treating data as generic byte streams, mosaicod applies compression based on specific data models. This reduces storage bloat and enables partial retrieval, allowing users to extract short events from long logs without scanning the entire dataset.
    2. Schema Versioning & Lineage: It anchors type enforcement to the specific schema snapshot present at the time of recording. This ensures that historical data remains readable even as software evolves and schemas change.
    3. Concurrency Management: It manages contention when multiple uncoordinated processes attempt to write to the same sink simultaneously, ensuring data integrity.
    4. Developer Efficiency: By centralizing these responsibilities, mosaicod makes data integrity, lineage, and schema enforcement built-in properties of the recording process rather than manual developer tasks.

  11. Understand Mosaico's chunking and skip index strategy

    Mosaico uses a chunking and indexing strategy to optimize data retrieval. When a data chunk is written to disk, the server generates 'skip indices' stored in the metadata database. These indices contain ontology-specific statistics—such as coordinate bounding boxes for GPS data or value ranges for sensors—enabling the query engine to perform content-based filtering without reading the entire bulk dataset from disk.

  12. Understand the Mosaico query engine filtering architecture

    The Mosaico query engine utilizes a three-tier filtering model to perform complex, multi-dimensional searches across large datasets without requiring full file scans. The tiers operate as follows:

    1. Sequence Filtering: Targets recording sessions based on structural attributes such as sequence name, creation timestamp, or user-defined metadata tags.
    2. Topic Filtering: Refines the search to specific data streams within the identified sequences using topic names, ontology tags (data types), serialization formats, or topic-level metadata.
    3. Ontology Filtering: Queries the actual physical values within sensor data. This tier uses statistical indices created during ingestion and min/max bounds stored in the metadata cache for each chunk to rapidly include or exclude data segments.