DOCSEARCHbyAlgolia
Submit Docs

Data Contract CLI

repository·main·Indexed 21 days ago

https://github.com/datacontract/datacontract-cli

An open-source Python command-line tool for managing, linting, testing, and exporting data contracts based on the Open Data Contract Standard (ODCS). It allows users to validate data quality and schema compliance, detect breaking changes, and synchronize with dbt projects. The tool can be used as a standalone CLI, within CI/CD pipelines, as a Python library, or as a REST API via Docker.

Tokens
104K
Snippets
346
Records
465
Agent score
74%

What's inside datacontract-cli

  1. Core capabilities of Data Contract CLI

    main

    The datacontract CLI is built around the Open Data Contract Standard (ODCS) and provides the following core capabilities:

    • Linting: Validates data contracts against the ODCS JSON Schema.
    • Connecting: Supports a wide range of data sources (Snowflake, BigQuery, Databricks, Postgres, Kafka, S3, etc.).
    • Testing: Verifies that actual data complies with the schema and quality attributes (e.g., freshness, row counts) defined in the contract.
    • Exporting: Converts contracts into 25+ formats, including SQL DDL, dbt, Avro, JSON Schema, HTML, and Protobuf.
    • Importing: Ingests existing schemas from SQL, dbt, BigQuery, Glue, Excel, etc., into an ODCS contract.

  2. Test BigQuery data contracts

    main

    BigQuery testing utilizes native type introspection. The CLI checks the declared physicalType against the actual column type retrieved from the BigQuery API.

    Behavioral Rules:

    • Timezones: Timezone variants of timestamps are considered interchangeable.
    • Parameters: Parameters like precision or length are only enforced if they are explicitly declared in the contract.
    • Fallback: If the physicalType cannot be interpreted as a valid BigQuery type, the CLI falls back to comparing the logicalType category.

  3. Test Trino data quality and schema

    main

    Trino testing utilizes native type introspection. The CLI checks the declared physicalType against the information_schema.columns in Trino.

    • Precision/Length: Enforced only if explicitly declared in the contract (e.g., varchar(10) matches varchar(10)).
    • Timestamps: Timezone variants of timestamps are treated as interchangeable.
    • Fallback: If a physicalType is not a valid Trino SQL type, the CLI falls back to comparing logical type categories.

  4. How type testing works for local files

    main

    When executing tests, type handling behavior depends on the format specified in the servers block. Note that physicalType is never checked against file sources; you should declare logicalType and use logicalTypeOptions for value constraints.

    formatTesting Behavior
    csvFiles are read as the contract's types. No type checks are generated; values that cannot be coerced result in a read error.
    jsonSimilar to CSV, but every record is also validated against a JSON Schema derived from the contract's logicalTypes (including format options like date-time or uuid).
    parquetColumn types are taken from the Parquet file; the contract's logicalType is checked by category (e.g., Check that field x has type y).
    deltaColumn types are taken from the Delta table; the contract's logicalType is checked by category.

  6. Understand Data Contract CLI configuration mapping

    main

    The Data Contract CLI uses environment variables to configure various data sources and general settings. These variables map to specific Config fields in the application.

    There are three ways to represent these configurations:

    1. Environment Variables: Prefixed with DATACONTRACT_ (e.g., DATACONTRACT_SNOWFLAKE_USERNAME).
    2. YAML Config Files: Use ${VAR}-style keys that drop the DATACONTRACT_ prefix and the section name (e.g., snowflake.username for DATACONTRACT_SNOWFLAKE_USERNAME).
    3. API Headers: Use the environment variable name in lowercase with dashes (e.g., datacontract-snowflake-username).

  8. Understand configuration precedence in Data Contract

    main

    Data Contract resolves configuration settings using a specific hierarchy. When multiple sources define the same option, the following precedence order applies (from highest to lowest):

    1. Explicit config: The Config object or dictionary passed directly to DataContract, the configuration file loaded via the --config-file flag, or specific datacontract-* headers in a request.
    2. Environment variables: Any settings not covered by the explicit configuration.
    3. .env file: Used to populate environment variables, but it never overrides an existing environment variable.

    This hierarchy allows for a flexible setup where a committed configuration file provides defaults, CI/CD pipelines override specific values via environment variables, and application code can override all settings via explicit configuration.

  9. Configure required, unique, and primary keys

    main

    You can enforce data integrity using the following property attributes:

    • required: true: Asserts no missing (null) values.
    • unique: true: Asserts no duplicate values.
    • primaryKey: true: Asserts both required and unique (no nulls and no duplicates).

    Composite Primary Keys

    To define a composite primary key, set primaryKey: true on multiple properties. The CLI treats them as a unique tuple. Use primaryKeyPosition to define the order of the members in the key.

    Example of a composite key:

    properties:
  - name: order_id
    primaryKey: true
    primaryKeyPosition: 1
  - name: line_number
    primaryKey: true
    primaryKeyPosition: 2

    This produces: order_id not null, line_number not null, and the combination (order_id, line_number) is unique.

  10. Test Postgres data contracts

    main

    Postgres testing uses native type introspection by checking the declared physicalType against information_schema.columns.

    Key Testing Behaviors

    • Type Distinction: On Postgres, text and varchar are distinct types. You must declare the exact type used by the column.
    • Interchangeable Types: decimal and numeric are treated as interchangeable. Similarly, different timezone variants of timestamps are interchangeable.
    • Precision/Length Enforcement: Length and precision are strictly enforced when declared. For example, varchar will match varchar(255), but varchar(255) will not match varchar(100).
    • Fallback: If a physicalType is provided that is not valid Postgres SQL, the CLI falls back to comparing the logicalType category.

    Note: This behavior also applies to Postgres-compatible databases configured with type: postgres (e.g., RisingWave).

  11. Manage credentials and connection details

    main

    Do not store credentials directly in the contract file.

    1. Connection details: Define these in the servers section of your contract.
    2. Credentials: Provide these via environment variables or a .env file located in your working directory.

    Check the specific reference page for your data source to see which environment variable names it expects.

  12. How dbt sync manages versioned models

    main

    The CLI supports dbt model versions. To manage multiple versions of a model via data contracts, create one contract file per version. The filename must include an integer version number preceded by a v (e.g., orders-v01.odcs.yaml).

    When datacontract dbt sync is executed with a subset of these versioned contracts, the versions not included in the command remain unchanged in the dbt project.