Open Data Contract Standard (ODCS)

repository·main·Indexed 21 days ago

https://github.com/bitol-io/open-data-contract-standard

A platform-agnostic, YAML-based standard (v3.1.0) for defining formal agreements between data producers and consumers. ODCS provides a structured format to document dataset schemas, data quality rules (accuracy, completeness, validity), service-level agreements (SLAs), pricing, stakeholders, and infrastructure details. It includes official JSON schemas for IDE validation and supports the .odcs.yaml file extension for automatic schema detection via SchemaStore.

Tokens: 30.9K
Snippets: 88
Records: 115
Agent score: 76%

What's inside Open Data Contract Standard

Overview of the Open Data Contract Standard (ODCS)
main
The Open Data Contract Standard (ODCS) defines the keys and values expected in a YAML-based data contract. It provides a structured way to describe datasets, including their schemas, data quality requirements, pricing, stakeholders, roles, service-level agreements (SLAs), and infrastructure.
Key principles of the standard:
- Platform Agnostic: The contract is designed to be usable across different data platforms without vendor lock-in.
- Standard Precedence: If a conflict arises between the written standard and the provided JSON schemas, the standard documentation takes precedence.
- YAML-based: The standard is primarily implemented using YAML files.
https://github.com/bitol-io/open-data-contract-standard/blob/main/docs/README.md
What is the Open Data Contract Standard (ODCS)?
main
The Open Data Contract Standard (ODCS) defines a structured agreement between a data producer and its consumers. It provides a standardized way to document the metadata, schema, quality requirements, and service levels associated with a data product. The current version is v3.1.0 and is licensed under Apache 2.0.
https://github.com/bitol-io/open-data-contract-standard/blob/main/README.md
Explore Open Data Contract Standard (ODCS) examples
main
The examples/ directory provides various excerpts of data contracts to illustrate specific sections, behaviors, and implementation patterns within the ODCS. You can use these examples to understand how to define different aspects of a data product, ranging from basic schema definitions to complex service-level agreements (SLAs) and data quality rules.
https://github.com/bitol-io/open-data-contract-standard/blob/main/docs/examples/README.md
Explore third-party ODCS software and tools
main
The Open Data Contract Standard (ODCS) is supported by a variety of third-party vendors and open-source tools. These solutions include data catalogs, quality platforms, enforcement gateways, and editors.
Key categories of tools include:
- Enforcement & Gateways: Tools like ContractGate provide real-time semantic enforcement at ingestion, while DataVow and vowl provide CLI-based validation and enforcement.
- Authoring & Editing: Use the Data Contract Editor (web-based) or FLUID Forge (Python CLI) to author and plan contracts. The Data Contract Playground allows for quick creation and validation.
- CLI Tooling: Data Contract CLI provides linting, testing against databases, and schema imports/exports. FLUID Forge and DataVow are also available for command-line workflows.
- Libraries & Templates: The open-data-contract-standard Python package allows for reading and writing ODCS YAML files. For non-programmatic authoring, an Excel Template is available.
https://github.com/bitol-io/open-data-contract-standard/blob/main/vendors.md
Use server-type-specific properties
main
Beyond the common properties, each server type can include specific configuration fields depending on the technology. For example, you might include host, port, database, or schema when configuring a database server. Refer to the specific documentation for the technology you are using to see which additional fields are supported.
https://github.com/bitol-io/open-data-contract-standard/blob/main/docs/infrastructure-servers.md
Understand ODCS v3 schema terminology
main
In ODCS v3, the schema moves away from strict table/column terminology to a more flexible model that supports both business representations and physical implementations. The core building blocks are:
- Objects: A structure of data. This can represent a table in an RDBMS, a document in a NoSQL database, or any other data container.
- Properties: Attributes of an object. This can represent a column in a table, a field in a JSON payload, or any other attribute.
- Elements: A generic term that refers to either an Object or a Property.
https://github.com/bitol-io/open-data-contract-standard/blob/main/docs/schema.md
Integrate ODCS with IDEs via SchemaStore
main
ODCS is supported by SchemaStore, which allows tools like VS Code and IntelliJ to provide automatic schema detection, autocompletion, and validation for ODCS files. Because ODCS is onboarded to SchemaStore, using the .odcs.yaml extension will trigger these IDE features automatically.
https://github.com/bitol-io/open-data-contract-standard/blob/main/schema/README.md
Define the `team` object in a data contract
main
The team object represents a group of people associated with a data contract. Starting with ODCS v3.1.0, the team structure is an object containing metadata about the team and a list of its members.
Note on Migration: In v2.x, this section was called stakeholders and used an array structure. While both the old array structure and the new object structure are valid in v3.1.0, the array structure is deprecated and will be removed in ODCS v4. You should use the object-based team structure for all new contracts.
```
team:
  id: tsc_team
  name: TSC
  description: The greatest team ever.
  members:
    - username: ceastwood
      role: Data Scientist
      dateIn: 2022-08-02
      dateOut: 2022-10-01
      replacedByUsername: mhopper
    - id: mhopper_member
      username: mhopper
      role: Data Scientist
      dateIn: 2022-10-01
    - id: daustin
      username: daustin
      role: Owner
      description: Keeper of the grail
      name: David Austin
      dateIn: 2022-10-01
```
https://github.com/bitol-io/open-data-contract-standard/blob/main/docs/team.md
Understand the components of a Data Contract
main
An ODCS data contract is composed of several key sections that define the lifecycle and operational parameters of the data. These sections include:
- Fundamentals: Core definitions and purpose.
- Schema: The structural definition of the data.
- References: External links or related data assets.
- Data quality: Rules and expectations for data accuracy and completeness.
- Support & communication channels: How to contact the producer.
- Pricing: Cost models for accessing the data.
- Team: Ownership and contact information.
- Roles: Responsibilities of different stakeholders.
- Service-level agreement (SLA): Guarantees on availability and performance.
- Infrastructures & servers: Technical environment details.
- Custom & other properties: Extensible fields for specific use cases.
https://github.com/bitol-io/open-data-contract-standard/blob/main/README.md
How to use references in ODCS v3.1.0
main
In ODCS v3.1.0, references allow you to create relationships between different parts of a data contract. Currently, references are exclusively supported for foreign key relationships.
To ensure stability during refactoring, it is recommended to use Fully Qualified Reference Notation, which relies on the id field and slash-separated paths rather than names. This makes references resilient to reordering or renaming of elements.
```

'schema/users_tbl/properties/user_id_pk'
```
https://github.com/bitol-io/open-data-contract-standard/blob/main/docs/references.md
Structure of an ODCS contract
main
An ODCS contract is organized into several functional sections. When building or consuming a contract, you can reference these specific domains to define different aspects of your data agreement:
1. Fundamentals: Core identity and basic properties of the contract.
2. Schema: The structural definition of the data.
3. References: Links to external datasets or related resources.
4. Data Quality: Rules and metrics for validating data integrity.
5. Support & Communication Channels: How to contact providers or report issues.
6. Pricing: Cost models and usage fees.
7. Team: Information about the organization or group providing the data.
8. Roles: Specific responsibilities of individuals or entities.
9. Service-Level Agreement (SLA): Guarantees regarding uptime, latency, or freshness.
10. Infrastructures & Servers: Technical details about where the data resides.
11. Custom & Other Properties: Extensibility for domain-specific needs.
https://github.com/bitol-io/open-data-contract-standard/blob/main/docs/README.md

Use Library metrics for standardized data quality checks

main

Library metrics are predefined metrics designed to be compatible with major data quality engines. They return a numeric value and require an operator (like mustBe, mustBeLessThan, or mustBeBetween) to validate the result.

Key features:

Type: The type: library can be omitted if the metric property is present.
Units: You can specify the unit as rows (default) or percent.
Arguments: Some metrics require an arguments object to define specific criteria (e.g., which values are considered 'missing').

Common Metrics Reference:

Metric	Level	Description	Arguments
`nullValues`	Property	Counts null values in a column/field	None
`missingValues`	Property	Counts values considered as missing (e.g., empty strings)	`missingValues`: Array of values
`invalidValues`	Property	Counts values not matching criteria	`validValues`: Array OR `pattern`: Regex
`duplicateValues`	Property	Counts duplicate values in a column	None
`duplicateValues`	Schema	Counts duplicate values across multiple columns	`properties`: Array of property names
`rowCount`	Schema	Counts total rows in a table/object store	None

properties:
  - name: order_id
    quality:
      - id: order_id_no_nulls
        metric: nullValues
        mustBe: 0
        unit: rows
        description: "There must be no null values in the column."