DOCSEARCHbyAlgolia
Submit Docs

Apache Ossie

repository·main·Indexed 22 days ago

https://github.com/apache/ossie

A standardizing effort for semantic model exchange in the data analytics, AI, and BI ecosystem. It provides a vendor-agnostic JSON/YAML specification to ensure consistent data definitions across different tools. The project includes converters for Databricks Unity Catalog Metric Views, dbt MetricFlow Semantic Interface (MSI), GoodData LDM, and NVIDIA GSF.

Tokens
60.6K
Snippets
157
Records
279
Agent score
78%

What's inside Apache Ossie

  1. Overview of Apache Ossie

    main

    Apache Ossie (formerly Open Semantic Interchange or OSI) is an open-source initiative designed to standardize semantic model exchange across the data analytics, AI, and BI ecosystem. It provides a vendor-agnostic specification to ensure that data definitions, metrics, and business logic remain consistent as they move between different tools (e.g., AI agents, BI platforms, and data engineering tools).

    By using a single source of truth, Ossie aims to solve:

    • Metric Drift: Conflicting KPI definitions across platforms.
    • Manual Translation: The need to manually reconcile definitions between systems.
    • AI Hallucinations: Unreliable AI outputs caused by inconsistent data logic.
    • Integration Debt: The cost of maintaining brittle, point-to-point connectors.

  2. Convert Ossie YAML models to Snowflake Cortex Analyst YAML

    main

    The Apache Ossie to Snowflake Converter performs an offline conversion of Ossie YAML semantic models into Snowflake Cortex Analyst semantic model YAML. This process does not require an active Snowflake connection.

    Note: This tool is under active development and may not handle all edge cases; use with caution in production environments.

    uv run ossie-snowflake -i input.yaml -o output.yaml

  3. Understand fidelity and data loss in Ossie ↔ GSF conversion

    main

    Conversion between Ossie and GSF involves specific trade-offs regarding metadata and schema details.

    Round-trip Fidelity (GSF $\rightarrow$ Ossie $\rightarrow$ GSF)

    To enable high-fidelity round trips, the converter stores the entire native GSF document in an NVIDIA_GSF custom extension within the Ossie file. This allows the reuse of live identifiers, catalog properties, SQL text, and zones. Note that Ossie files containing this extension are significantly larger and not intended for manual review.

    Unavoidable Losses (Ossie $\rightarrow$ GSF)

    The following elements cannot be represented in a native GSF document and are lost during conversion:

    • Semantic-model envelopes
    • ai_context
    • Dimensions and synonyms
    • Ossie custom-extension storage
    • Expression-dialect variants

    Type Mapping

    • Ossie datatype $\leftrightarrow$ GSF physical type: For fields backed by a single column, these map to each other.
    • Logical Reduction: GSF $\rightarrow$ Ossie reduces physical types to Ossie's logical vocabulary (e.g., NUMBER(38,0) $\rightarrow$ Integer, NUMBER(12,2) $\rightarrow$ Decimal). Unknown types like Snowflake's VARIANT become Opaque.
    • Computed Fields: Since computed fields/metrics have no single underlying column, their datatype is not carried in GSF.

    SQL and Dialects

    • SQL is parsed using sqlglot across multiple candidate dialects.
    • If SQL cannot be parsed, it is carried through verbatim as opaque text.
    • On GSF $\rightarrow$ Ossie, expressions are labeled with the source connection's dialect (e.g., SNOWFLAKE, DATABRICKS, BIGQUERY) or ANSI_SQL if unknown.

  4. Mapping between GoodData LDM and Ossie Semantic Model

    main

    When converting between formats, the following conceptual mappings are applied:

    GoodData LDMOssie Semantic Model
    DatasetDataset
    Attribute (+ Labels)Field with dimension metadata
    FactField without dimension metadata
    Source column data typeField datatype
    Reference (FK)Relationship
    Date InstanceDataset with GOODDATA custom_extension (date_dimension: true)
    MAQL expressionDialect entry (dialect: MAQL)

  5. Understand the AI-Native Semantic Layer roadmap

    main

    Ossie is evolving to provide structured semantic context for AI-driven analytics and grounded query generation.

    Key roadmap deliverables include:

    • Standardized AI context metadata to provide LLMs with semantic grounding.
    • verified_queries: A core element of the spec allowing for curated, reliable query definitions.
    • Mechanisms to control how much semantic information is exposed to AI agents.

  6. How AI Context is preserved during conversion

    main

    OSI's ai_context (containing instructions, synonyms, and examples) is preserved losslessly during conversion by leveraging OBML's customExtensions mechanism.

    Mapping Pattern:

    • OSI $\rightarrow$ OBML: The ai_context object is serialized into a JSON string and stored in customExtensions with vendor: "OSI".
    • OBML $\rightarrow$ OSI: The converter looks for customExtensions where vendor == "OSI" and restores the data to the native OSI ai_context field.

    This applies to all levels of the model: datasets, fields, and the model itself.

    # OSI input
ai_context:
  synonyms:
    - "sales transactions"
    - "store purchases"

# OBML output (via customExtensions)
customExtensions:
  - vendor: OSI
    data: '{"synonyms": ["sales transactions", "store purchases"]}'

  7. Identify OBML to OSI conversion limitations

    main

    While the converter aims for lossless round-tripping, certain OBML-specific constructs are moved to custom_extensions (obml_* payloads) and are not interpreted by other OSI consumers:

    • Many-to-many joins: Represented via OBML join cardinality; flagged on export.
    • Named secondary join paths: OBML-specific topology features.
    • Ontology layer metrics: Measures, metrics, and column-level value concepts in the ontology layer are not represented in the OSI ontology export.
    • Non-SQL Metrics: If an OSI metric's expression is in a non-SQL dialect (MDX, TABLEAU, MAQL) or cannot be decomposed into OBML measures/metrics, it is preserved in a model-level OSI-vendor custom_extension named obml_unconverted_metrics. A LOSSY: warning is raised because these metrics are not queryable through OBML.

    Supported SQL Dialects for Import:

    • ANSI_SQL
    • SNOWFLAKE
    • DATABRICKS

  8. Map Ossie core constructs to vendor equivalents

    main

    When building or using a converter, you must map the following core Apache Ossie constructs to their corresponding vendor-specific representations:

    • Semantic Model: The top-level container (maps to the vendor's root object/project).
    • Datasets: Logical tables (fact/dimension) containing fields and structure.
    • Fields: Row-level attributes (columns or computed expressions).
    • Relationships: Foreign key connections between datasets.
    • Metrics: Aggregate measures defined at the semantic model level.
    • Custom Extensions: Vendor-specific metadata stored as JSON strings.
    • AI Context: Instructions and synonyms for AI/LLM tools.

  9. Understand the relationship between `datatype` and `is_time`

    main

    In Apache Ossie, datatype and dimension.is_time serve different purposes:

    1. datatype (Data Type): Describes the kind of values held (e.g., Date, Integer, String). Used for casting and serialization.
    2. dimension.is_time (Temporal Role): A marker indicating if the field should be treated as a time dimension for time-series analysis or temporal filtering.

    Default Behavior for is_time: If is_time is omitted, it defaults to true if the datatype is Date, Time, DateTime, or DateTimeTz. Otherwise, it defaults to false. Explicitly setting is_time: false allows you to opt-out of the temporal role for temporal-typed columns (e.g., an audit created_at timestamp).

  10. Understand aggregation decomposability

    main

    Aggregations are categorized by their decomposability, which determines how they can be used in multi-stage aggregation processes:

    • Distributive: SUM, COUNT, MIN, MAX (can be computed by aggregating partial results).
    • Algebraic: AVG, STDDEV, VARIANCE (can be computed from partial results and their counts/sums).
    • Holistic: MEDIAN, PERCENTILE, COUNT DISTINCT (require seeing the whole dataset or complex merging).
    • Sketch-based: APPROX_COUNT_DISTINCT, APPROX_PERCENTILE (use mergeable probabilistic sketches).

  11. Configure the ai_context field for AI agents

    main

    The ai_context field provides semantic guidance to AI agents. It can be defined in two ways:

    1. Simple String: A comma-separated list of terms or a brief description.
    2. Structured Object: A detailed object used to provide specific instructions, synonyms, and sample queries to improve agent accuracy.

    When using the structured object, the following fields are recommended:

    • instructions (string): Specific guidance on how the AI should interpret or use the entity.
    • synonyms (array): A list of alternative names or terms that refer to the same concept.
    • examples (array): A list of sample questions or use cases that the agent might encounter.
    # Simple String format
ai_context: "orders, purchases, sales"

# Structured Object format
ai_context:
  instructions: "Use this for sales analysis"
  synonyms:
    - "orders"
    - "purchases"
    - "sales"
  examples:
    - "Show total sales last month"
    - "What's the revenue by region?"

  12. Integrate Ossie with Data Catalogs and Semantic Services

    main

    To ensure semantic models are discoverable and governable, the roadmap includes integration with data catalogs and the establishment of centralized semantic services.

    Planned deliverables include:

    • Catalog Integration: Implementing patterns to work with catalogs like Polaris.
    • Semantic Registry: A standalone service or registry for managing Ossie models.
    • Model Management: Providing discovery, versioning, and access control for all Ossie-defined models.