DOCSEARCHbyAlgolia
Submit Docs

Overture Documentation

website·Indexed 32 days ago

https://docs.overture.bio

Developer and user documentation for the Overture ecosystem, including services such as Song for metadata management, Lyric for tabular data submission, Arranger for federated search and Elasticsearch queries, Maestro for data indexing, Lectern for dictionary and schema validation, Score, and Stage, a React-based user interface framework.

Tokens
84.7K
Snippets
343
Records
589
Agent score
61%

What's inside Overture

  1. Overview of Lyric API capabilities

    The Lyric API provides a system for submitting, validating, querying, and managing tabular data aligned with Lectern dictionary schemas. Its core capabilities include:

    • Data Submission: Validate and submit tabular data files using a staged submission workflow.
    • Data Management: Perform full CRUD (Create, Read, Update, Delete) operations on committed data.
    • Data Querying: Search and retrieve committed data, including support for SQON-based search.
    • Auditability: Track a complete audit history of all data changes.
    • Dictionary Management: Register Lectern dictionaries for use and manage dictionary categories.

  2. Overview of Arranger

    Arranger is a model-agnostic data discovery API designed to simplify the creation of search interfaces for complex datasets using OpenSearch or Elasticsearch. It consists of two primary modules:

    1. Arranger Server: A back-end service that generates a GraphQL API from Elasticsearch mappings, acting as middleware to simplify querying via Serializable Query Object Notation (SQON).
    2. Arranger Components: A React component library for building interactive search UIs (including faceted search, data tables, and SQON viewers) that communicate with the Arranger Server.

  3. Overview of Lectern Data Dictionary Manager

    Lectern is a Data Dictionary Manager designed to manage collections of data dictionaries—organized schemas that define the structure, constraints, and relationships of data models. It serves as a central repository for dictionary schemas, tracking versions and relationships in a MongoDB database. Lectern is typically integrated with Lyric (Overture's tabular data submission service) to validate incoming data submissions against hosted schemas. Key capabilities include:

    • Schema Definition and Validation: Define schemas and validate them against the Lectern base meta-schema.
    • Version Control: Track changes to data structures over time.
    • Difference Computation: Compare different versions of schemas to identify changes in data requirements.
    • Integration: Provides a RESTful API (Swagger) for integration into larger data management systems.

  4. Overview of the Overture stack deployment components

    Deploying a full Overture instance involves setting up several interconnected components to handle authorization, data transfer, submission, and discovery. The stack consists of the following modules:

    • Authorization: Handled by Keycloak.
    • File Transfer: Managed by Song and Score.
    • Tabular Data Submission: Handled by Lectern and Lyric.
    • Search Portal: Comprised of Maestro, Arranger, and Stage.
    • AI Discovery (Optional): An MCP server for AI-assisted, conversational data discovery.

  5. Overview of AI-Assisted data discovery in Overture

    AI-Assisted data discovery allows researchers to query Overture portals using plain language. The system converts natural language requests into structured Arranger search queries (similar to text-to-SQL). This is achieved via an MCP-compatible client connected to the Arranger MCP server, enabling a unified interface for data exploration, code execution, and visualization.

  6. Overview of the Stage UI Framework

    Stage is a React-based user interface framework designed for building browser-accessible data portals and data management platforms. It serves as a scaffold that provides fundamental UI components for navigation, authentication, and data exploration.

  7. Overview of Overture developer components

    Overture consists of several specialized components for managing biological data pipelines, ranging from schema definition to front-end portals:

    • Lectern: Manages data dictionaries, including schemas that define data structure, constraints, and relationships.
    • Lyric: A model-agnostic service used for submitting, validating, and managing tabular data against Lectern dictionaries.
    • Song: A file catalog service that tracks files and manages their associated metadata.
    • Score: A file transfer service designed for secure, large-file uploads and downloads to and from object storage.
    • Maestro: Indexes metadata from Song into Elasticsearch to enable search and exploration.
    • Arranger: A model-agnostic search API and UI component library used for building data portals.
    • Stage: A React-based UI framework for building the front-end of data portals.

  8. Overview of Overture Platform APIs

    The Overture Platform provides six distinct APIs for managing music-related data. Five of these are RESTful APIs documented via Swagger, while the Arranger API uses GraphQL.

    • Song API: Used for managing file metadata stored in Song.
    • Score API: Used for communicating with the Score Server.
    • Maestro API: Used for managing Song data indexing operations.
    • Lyric API: Used for tabular data submission, validation, and querying.
    • Lectern API: Used for managing data dictionary schemas.
    • Arranger API: A GraphQL API used for search, which also includes a set of read-only introspection REST endpoints.

  9. Overview of the Overture Song API

    The Song API is used to manage and query file metadata within the Overture system. Its primary capabilities include:

    • Managing metadata for study, analysis, and file-centric entities.
    • Validating and submitting analysis metadata, featuring schema versioning and publication controls (published, unpublished, or suppressed).
    • Managing the schemas used for metadata validation.
    • Monitoring the health status of the Song-server.

  10. Overview of Score file transfer service

    Score is a file transfer service that acts as an intermediary between object storage systems and user authorization mechanisms. It enables secure, large-file uploads and downloads by validating user access rights via OAuth and generating time-limited pre-signed URLs for direct object storage access. It is part of the Overture.bio suite and typically integrates with Song (metadata management), Keycloak (OAuth2 authentication), and the Score Client (CLI tool).

  11. Overview of the Lectern Viewer

    The Lectern Viewer is a React component library used to display Lectern data dictionaries. It transforms a dictionary's JSON schema into interactive, readable views (tables and diagrams) to help researchers and data submitters understand data requirements. It is designed to work alongside the Lectern Server (which stores and versions schemas) and Lyric (which validates submissions against those schemas).
    npm install @overture-stack/lectern-ui

  12. Overview of SQON query language

    SQON is a JSON format used by Arranger to describe filters as a portable tree structure. Unlike a flat key-value map, SQON allows for complex logic by using a tree of nodes. Each node is either a filter clause (a single field-level condition) or a combination (an AND, OR, or NOT operator that joins other nodes).
    {
  "op": "and",
  "content": [
    { "op": "in",      "content": { "fieldName": "province",          "value": ["Ontario", "Quebec"] } },
    { "op": "between", "content": { "fieldName": "age",               "value": [20, 50] } },
    { "op": "not-in",  "content": { "fieldName": "consent_withdrawn", "value": [true] } }
  ]
}