DataHub Documentation

repository·master·Indexed 27 days ago

https://github.com/datahub-project/datahub

An open-source AI data catalog for discovery, governance, and observability. This documentation covers the unified metadata graph, Metadata Model Extensions (including Custom Validators, Model Plugins, Custom Aspects, and Entity Extensions), and the DataHub Actions pipeline for event-driven workflows such as documentation propagation.

Tokens: 311.5K
Snippets: 644
Records: 1.6K
Agent score: 95%

What's inside DataHub

Overview of the ODCS Source
master
The Open Data Contract Standard (ODCS) source ingests YAML-based data contracts following the Linux Foundation / Bitol standard.
Key Modeling Behavior:
- Logical Datasets: Instead of modeling contracts as dataContract entities, the source models each schema[] entry in an ODCS file as a logical odcs Dataset.
- Physical Linking: When a schema[] entry resolves to a physical dataset (via servers[]), the source creates a PhysicalInstanceOf relationship, linking the physical dataset to the logical one using the logicalParent field.
- Assertions: Quality rules (quality[]) and schema compliance are attached to the logical dataset, not the physical one. DataHub propagates these expectations to physical instances via the logical-to-physical relationship.
Note on Logical Models: Logical Models are in private beta. To see logical odcs datasets and their assertions in the DataHub UI, you must enable the LOGICAL_MODELS_ENABLED feature flag (it is off by default).
https://github.com/datahub-project/datahub/blob/master/metadata-ingestion/docs/sources/odcs/README.md
Overview of Postgres Metadata Ingestion
master
The DataHub Postgres integration ingests core metadata from PostgreSQL instances, including datasets, tables, views, schema fields, and containers. It supports advanced features such as table- and column-level lineage, data profiling, and stateful deletion detection.
https://github.com/datahub-project/datahub/blob/master/metadata-ingestion/docs/sources/postgres/README.md
Overview of Grafana integration for DataHub
master
The DataHub integration for Grafana ingests business intelligence and analytics metadata. It captures BI entities including dashboards, charts, and datasets, along with their associated ownership context. Additionally, the integration supports table- and column-level lineage, ownership, tags, and stateful deletion detection.
https://github.com/datahub-project/datahub/blob/master/metadata-ingestion/docs/sources/grafana/README.md
Overview of Power BI Report Server integration
master
The DataHub integration for Power BI Report Server allows you to ingest business intelligence and analytics metadata into DataHub. It captures BI entities such as dashboards, charts, and datasets, along with related ownership context. Additionally, the integration supports ownership tracking and stateful deletion detection.
https://github.com/datahub-project/datahub/blob/master/metadata-ingestion/docs/sources/powerbi-report-server/README.md
Overview of Snaplogic Integration
master
The DataHub integration for Snaplogic allows you to ingest metadata from the Snaplogic streaming and integration platform. This integration captures streaming and integration entities such as topics, connectors, pipelines, or jobs, and provides table- and column-level lineage.
https://github.com/datahub-project/datahub/blob/master/metadata-ingestion/docs/sources/snaplogic/README.md
Overview of Airbyte Integration for DataHub
master
The Airbyte integration extracts metadata from Airbyte to provide visibility into data pipelines within DataHub. It captures information regarding connections, sources, destinations, streams, and job execution history. Crucially, it provides lineage between source and destination datasets at both the table and column levels.
https://github.com/datahub-project/datahub/blob/master/metadata-ingestion/docs/sources/airbyte/README.md
Overview of metadata-io-api
master
The metadata-io-api module is designed to isolate dependencies for external projects. It is intended to be used when implementing custom extensions for DataHub, such as custom validators, mutators, or side-effect handlers, ensuring that these extensions do not introduce dependency conflicts into the host project.
https://github.com/datahub-project/datahub/blob/master/metadata-io/metadata-io-api/README.txt
Overview of SQLAlchemy DataHub Integration
master
The DataHub integration for SQLAlchemy allows you to ingest metadata from SQLAlchemy-based data platforms into DataHub. It captures core metadata entities including datasets (tables/views), schema fields, and containers. Additionally, it supports capturing table- and column-level lineage, data profiling, and stateful deletion detection.
https://github.com/datahub-project/datahub/blob/master/metadata-ingestion/docs/sources/sqlalchemy/README.md
Overview of DataHub Apply integration
master
DataHub Apply is a utility designed for metadata-focused integrations. The DataHub connector for DataHub Apply allows you to apply metadata from YAML or JSON files directly to existing DataHub entities. Unlike discovery-based connectors, this integration uses input files to drive metadata updates.
https://github.com/datahub-project/datahub/blob/master/metadata-ingestion/docs/sources/datahubapply/README.md
Overview of Neo4J integration for DataHub
master
The DataHub integration for Neo4J allows for the ingestion of core metadata entities from Neo4J into DataHub. This includes datasets, tables, views, schema fields, and containers. Additionally, the integration supports stateful deletion detection to track removed assets.
https://github.com/datahub-project/datahub/blob/master/metadata-ingestion/docs/sources/neo4j/README.md
Overview of ClickHouse ingestion for DataHub
master
The DataHub integration for ClickHouse allows you to ingest metadata from ClickHouse data platforms into DataHub. The integration captures the following metadata entities:
- Core technical assets: Datasets, tables, and views.
- Schema information: Schema fields/columns.
- Containers: Organizational structures.
- Lineage: Table-level and column-level lineage (when supported and enabled).
- Usage & Quality: Usage statistics, data profiling, and stateful deletion detection.
https://github.com/datahub-project/datahub/blob/master/metadata-ingestion/docs/sources/clickhouse/README.md
Overview of Metadata File integration for DataHub
master
The Metadata File integration allows DataHub to ingest metadata from the Metadata File storage and lakehouse platform. This integration covers file and lakehouse metadata entities including datasets, paths, and containers. It also supports stateful deletion detection to track removed assets.
https://github.com/datahub-project/datahub/blob/master/metadata-ingestion/docs/sources/metadata-file/README.md