Elementary OSS

The Elementary dbt package provides three core capabilities:

1. Elementary data tests: Provides advanced testing including anomalies in volume, freshness, and various data dimensions.
2. dbt artifacts: Automatically uploads dbt artifacts to tables in your target database.
3. on-run-end hooks: Uses dbt hooks to automate the upload of information generated during dbt runs.

Elementary provides a comprehensive observability suite including:

• dbt Artifacts & Run Results: Upload metadata, run, and test results to tables as part of your dbt jobs.
• Alerts: Send informative alerts to various channels and users.
• Data Observability Dashboard: A central interface to inspect data health, test results, model performance, and data lineage.
• End-to-end Data Lineage: Inspect dependencies, including Column Level Lineage and integration with BI tools.
• Data Catalog: Explore and discover datasets and manage documentation in code.

Elementary is a data observability platform specifically designed for dbt-first data organizations. It uses a dbt-native architecture to integrate into existing engineering workflows. The platform provides:

• Out-of-the-box monitoring for critical data issues.
• Coverage tools to increase data monitoring coverage.
• Integrations for end-to-end visibility across the data stack.
• Incident management tools to promote ownership and collaboration.
• Proactive data health tracking to help teams communicate data quality to stakeholders instead of reacting to failures.

Elementary provides automated Column-Level Lineage that integrates metadata from your data warehouse with BI tool integrations (such as Looker and Tableau). This lineage is enriched with the latest test and monitor results to support several workflows:

• Data Issue Debugging: Trace data issues back to their original sources.
• Incident Impact Analysis: Identify which downstream assets are affected by a specific data issue.
• Issue Prioritization: Triage issues based on their impact on critical downstream assets.
• Public Asset Health: Select an exposure and filter by upstream nodes to check the status of all upstream datasets.
• Change Impact Analysis: Determine which exposures will be affected by a planned schema or data change.
• Unused Dataset Detection: Identify datasets that have no downstream consumers to help reduce costs.

Elementary Messages is a system for building structured messages that can be rendered across multiple platforms (e.g., Slack, Microsoft Teams). It uses a minimal, platform-agnostic format that is then converted into specific output formats by formatters.

Because the system is designed for cross-platform compatibility, some specific styles, icons, or blocks may not be supported by every target platform.

Elementary OSS is an open-source CLI designed for dbt-native data observability. It functions by connecting to your data warehouse and reading metadata, artifacts, and test results collected by the Elementary dbt package.

Key capabilities include:

• Generating data observability reports.
• Surfacing anomalies and failed tests.
• Sending alerts to Slack and Microsoft Teams.
• Tracking performance trends for models and tests.

Elementary enables data validation and result tracking by leveraging the existing dbt testing ecosystem. It automatically captures and tracks results from:

• dbt tests: Any standard dbt tests defined in your project.
• dbt packages: Tests provided by packages like dbt-utils, dbt-expectations, and the Elementary dbt package.
• Custom dbt tests: Any custom dbt generic or singular tests you have developed.
• Custom SQL tests: Users can also create custom SQL tests specifically for Elementary.

This allows you to enforce explicit expectations on your data and monitor their execution over time within the Elementary platform.

Elementary provides several mechanisms to manage data observability notifications:

• Alert distribution rules: Route alerts to specific people or communication channels.
• Incident status and assignee: Manage the lifecycle and ownership of an issue.
• Owners and subscribers: Assign responsibility and ensure the right people are notified.
• Severity and tags: Categorize alerts to assist in prioritization and filtering.
• Alerts customization: Tailor alert details to your workflow.
• Group alerts to incidents: Automatically aggregate related events into a single incident.
• Alerts suppression: Mute or silence specific alerts to reduce noise.

The Elementary Databricks integration supports the following capabilities:

1. dbt Artifact Uploading: Uploading dbt artifacts, test results, and run results via the dbt package.
2. Data Observability Reports: Generation of observability reports.
3. Slack Alerts: Notifications for failures in models and tests.
4. Anomaly Detection: Running anomaly detection tests.

Note: Elementary schema changes tests are currently not supported for Databricks.