repository·main·Indexed 19 days ago
https://github.com/datacontract/datacontract-specificationA YAML-based, platform-neutral standard for defining the structure, quality, and terms of data exchange between providers and consumers. It supports various technologies including S3, BigQuery, and Snowflake, and follows OpenAPI and AsyncAPI conventions. The specification is used for semantic expectations, code generation, schema validation, and computational governance. Note: This specification is being deprecated in favor of the Open Data Contract Standard v3.1.0, with support in the Data Contract CLI and Entropy Data continuing until the end of 2026.
Several tools are available to assist with the lifecycle of data contracts:
A data contract is a document that defines the structure, format, semantics, quality, and terms of use for exchanging data between a data provider and their consumers. It acts similarly to an API but for data sets.
The data contract specification is a YAML-based format used to describe these attributes. It is platform-neutral and can be used with any data technology (e.g., AWS S3, Google BigQuery, Snowflake, Databricks). It follows OpenAPI and AsyncAPI conventions.
Key uses include:
Service levels are agreed-upon, measurable performance levels for provided data. You can describe them informally using the description field or use predefined fields to enable automation via the Data Contract CLI.
Supported service level objects include:
availability: System uptime guarantees.retention: How long data remains available.latency: Maximum time from source to destination.freshness: Maximum age of the youngest entry.frequency: Update frequency (e.g., batch, streaming).support: Availability of support contact.backup: Details regarding backup procedures.The specification allows for extensibility. You can add custom fields with any name to extend the specification at specific points. The value for a custom field can be:
null$ref field.You can define your data contract schema using the json-schema type. The specification can be provided as an inline YAML object (following the OpenAPI Schema Object dialect) or as a single string containing a JSON string. This allows you to define properties, types, descriptions, and required fields for your data models.
schema:
type: json-schema
specification:
orders:
description: One record per order.
type: object
properties:
order_id:
type: string
description: Primary key
order_total:
type: integerA Definition Object provides a central reference for business terminology, ensuring consistent usage and identifying join-able fields across different models. Models can use the $ref field in a Field Object to link to these definitions.
Definition Object Fields:
domain: The domain of validity (defaults to global).name: Technical name.title: Business name.type: Logical data type.description: Domain-specific explanation.example: Example value.pii: Boolean for PII status.classification: Sensitivity level.tags: Custom metadata.You can define quality attributes using natural language descriptions. This is useful for expressing requirements to stakeholders or providing prompts for AI-driven data quality checks. These descriptions can later be translated into executable checks like SQL.
Quality objects can be specified at the field level or the model level. Note that the top-level quality object is deprecated.
models:
my_table:
fields:
account_iban:
quality:
- type: text
description: Must be a valid IBAN. Must not be empty.A Model Object describes the structure of a data model (like a table or view). The name of the model is the key used to refer to the object.
Model Object Fields:
type: The type of model (e.g., table, object). Defaults to table.description: Optional semantic description.fields: A map where keys are field names and values are Field Objects.Field Object Fields:
type: The logical data type.description: Semantic description of the field.pii: Boolean indicating if the field contains Personal Identifiable Information.classification: Sensitivity level (e.g., sensitive, restricted, internal, public).tags: Array of strings for custom metadata.$ref: A reference URI to a definition in the specification to inherit properties.The lineage object provides fine-grained information about data origins and transformations, based on the OpenLinage Column Level Lineage Dataset Facet. It is defined within a field's configuration.
An inputFields array describes the source data points:
namespace: The source system or domain (e.g., com.example.crm, snowflake://{account}).name: The source dataset name (e.g., a table name or Kafka topic).field: The specific source field or column.transformations: An optional array of Transformation objects describing how the data was processed.Describes the relationship between the input and the final field:
type: Either DIRECT or INDIRECT.subtype:DIRECT: IDENTITY, TRANSFORMATION, or AGGREGATION.INDIRECT: JOIN, GROUP_BY, FILTER, SORT, WINDOW, or CONDITIONAL.description: A string describing the transformation.masking: A boolean indicating if the value was obfuscated.models:
orders:
fields:
customer_email_address_hash:
type: string
lineage:
inputFields:
- namespace: com.example.service.checkout
name: checkout_db.orders
field: email_address
transformations:
- type: DIRECT
subtype: TRANSFORMATION
description: The email address from the checkout order, hashed with SHA-256
masking: truequality object defines quality attributes, which are specific checks applied to data to ensure its integrity. These checks are intended to be executed by a data quality engine.