Soda Core

Soda Core uses a three-step process to determine if specific SQL features (like APPROX_PERCENTILE_DISC) are supported by your SQL Server instance. This ensures that the correct SQL syntax is generated based on your specific engine version and edition.

1. Detection: When a connection is opened via SqlServerDataSourceConnection, Soda Core eagerly probes the server for its server_major_version (via pyodbc.SQL_DBMS_VER) and its engine_edition (via SELECT CAST(SERVERPROPERTY('EngineEdition') AS INT)).
2. Sync: The SqlServerDataSourceImpl synchronizes these raw facts from the connection onto the SqlServerSqlDialect instance.
3. Derivation: The SqlServerSqlDialect uses these facts to calculate support. For example, it checks if the version is $\ge 16$ (SQL Server 2022) or if the engine edition matches Azure SQL Database (5) or Managed Instance (8).

Note on Offline/Snapshot Replay: If no server facts are available (e.g., during offline rendering or snapshot replay), Soda Core assumes the newest engine version and enables capabilities by default.

A data contract is a YAML file defining schema and data quality checks for a specific dataset.

Example contract.yml:

# contract.yml 

dataset: postgres_ds/db/schema/dataset

checks: # dataset level checks
  - schema:
  - row_count: 

columns: # columns block
  - name: id
    checks: # column level checks 
      - missing:
  - name: name
    checks:
      - missing:
          threshold:
            metric: percent
            must_be_less_than: 10
  - name: size
    checks:
      - invalid:
          valid_values: ['S', 'M', 'L'] 

Note: The dataset identifier (e.g., postgres_ds/db/schema/dataset) must match the name property in your data source configuration.

Verify the contract: Run a scan to evaluate the contract against your data.

soda contract verify -ds ds_config.yml -c contract.yml

• -ds, --data-source (Required): Path to the data source YAML file.
• -c, --contract (Required): Path to the data contract YAML file.

You can manage contracts centrally and execute checks remotely by connecting Soda Core to Soda Cloud.

1. Create a Soda Cloud config:

   soda cloud create -f sc_config.yml

   - `-f`, `--file` (Required): Output file path for the Soda Cloud YAML configuration.

2. **Test the Cloud connection:**
   ```bash
soda cloud test -sc sc_config.yml

• -sc, --soda-cloud (Required): Path to the Soda Cloud YAML file.

3. Publish a contract to Soda Cloud: Use this to set a contract as the source of truth in the cloud.

   soda contract publish -c contract.yml -sc sc_config.yml

4. **Publish verification results to Soda Cloud:**
   Add the `-p` flag to your verification command to upload results.
   ```bash
soda contract verify -ds ds_config.yml -c contract.yml -sc sc_config.yml -p

• -p, --publish (Optional): Publish results and contract to Soda Cloud. Requires "Manage contract" permission.

When using the sparkdf data source, you must specify a connection mode in your configuration. Soda supports four distinct connection modes for Spark DataFrames. You must provide exactly one of the following sets of properties to avoid configuration conflicts.

1. Existing Session Mode

Use this when you have an already instantiated SparkSession object in your Python environment (e.g., in a Databricks notebook or a local script).

• Key: spark_session (the actual SparkSession object)

2. Active Session Mode

Use this to automatically pick up the session currently active in the current thread via SparkSession.getActiveSession(). This is useful for reusing the existing spark object in notebooks without passing it explicitly.

• Key: use_active_session: true

3. Remote Session Mode (Spark Connect)

Use this to connect to a remote workspace (like Databricks) via Spark Connect. This mode builds a session using SparkSession.builder.remote(<uri>).getOrCreate().

• Keys:
  • host: The workspace host (e.g., dbc-12345.cloud.databricks.com). Note: You can provide the full URL with https://, and Soda will strip the scheme automatically.
  • token: Your Personal Access Token (PAT).
  • cluster_id: The ID of the cluster to route the session to.

4. New Session Mode

Use this to trigger the creation of a fresh Spark session.

• Key: new_session: true (default behavior)