repository·main·Indexed 25 days ago
https://github.com/marquezproject/marquezAn open-source metadata service for the collection, aggregation, and visualization of data ecosystem metadata, focusing on data provenance, job runtime visibility, and dataset lifecycle management via the OpenLineage standard. Includes guides for Helm installation on Kubernetes, API and Web UI configuration, and usage of the Marquez Java and Python clients for reading metadata.
Column lineage provides granular metadata about how specific columns in a dataset are produced from input columns in other datasets. This is achieved using the ColumnLineageDatasetFacet from the OpenLineage spec.
Key features include:
transformationDescription: A string describing the transformation applied to the input columns.transformationType: Indicates the nature of the data. Supported values are IDENTITY (exact same as input) and MASKED (no original data available, such as a hash).Currently, column lineage is supported out-of-the-box for the Spark integration via OpenLineage.
Facets allow you to extend dataset, job, and run metadata. A custom facet must follow these rules:
_)._).When implementing a facet, you must include these base properties:
_producer: A URI identifying the producer of the metadata (e.g., a git URL with a tag or SHA)._schemaURL: The URL to the corresponding version of the schema definition.For a full list of standard run facets, refer to the OpenLineage standard facets specification.
Marquez's data model is built on the principles of immutability and versioning to ensure reproducible lineage. The core entities are:
A job represents a unit of work. It includes an owner, a unique name, a version, and an optional description. Jobs define dependencies via versioned inputs and artifacts via versioned outputs.
An immutable, read-only version of a job. It includes a unique link to the specific code used, ensuring build reproducibility. A Job Version maps specific input and output datasets to a job definition, which is critical for tracking how data flows through different jobs over time.
A dataset is a first-class value produced by job runs. It includes an owner, a unique name, a schema, a version, and an optional description. Datasets are grouped within a datasource (which represents the physical source).
An immutable, read-only version of a dataset. Each version has a unique ID representing a specific state in time. Marquez maintains a version pointer that is updated whenever a new change is committed. The current version is updated internally whenever a new distinct version ID is generated via Marquez's versioning function.
Marquez supports OpenLineage facets to extend metadata for datasets, jobs, and runs. Custom facets must follow a specific structure to ensure compatibility:
_producer: A URI identifying the producer of the metadata (e.g., a git URL)._schemaURL: A URL to the corresponding version of the schema definition._) to avoid naming conflicts with other facets._).In Marquez, OpenLineage facets (user-defined metadata) were originally queried by scanning the lineage_events table. Because the event column in lineage_events contains the raw JSON event, which can exceed 10MB per event, querying facets directly from this table can lead to Out-of-Memory (OOM) errors and poor performance.
To optimize this, Marquez introduces dedicated facet tables (dataset_facets, job_facets, and run_facets). These tables store extracted facets separately, allowing for efficient queries without loading massive raw event blobs into memory. Facets in these tables are append-only and follow a first-to-last received order, meaning newer facets override older ones during query-time merging using MapperUtils.toFacetsOrNull() logic.
Facets allow you to attach extensible metadata to OpenLineage entities. A CustomFacet is an object that can hold arbitrary properties. To ensure schema compliance, facets should ideally follow a BaseFacet structure which requires:
_producer: A URI identifying the producer of the metadata (e.g., a git URL)._schemaURL: The URL to the corresponding version of the schema definition.Example of a facet structure in JSON schema:
{
"_producer": {
"description": "URI identifying the producer of this metadata.",
"type": "string",
"format": "uri"
},
"_schemaURL": {
"description": "The URL to the corresponding version of the schema definition.",
"type": "string",
"format": "uri"
}
}To reduce database duplication, Marquez separates dataset schema versioning from dataset versioning.
When a job produces a dataset, Marquez compares the current schema against existing dataset_schema_versions. If the schema is identical, the new dataset_version simply references the existing dataset_schema_version. A new schema version is only created when the fields (name and type) actually change.
Equality Logic:
Two schemas are considered equal if they contain the same fields, keyed by name and type.
description is not considered a schema change.Facets allow you to extend dataset, job, and run metadata. Custom facets must follow a specific structure to ensure interoperability and avoid naming conflicts.
_)._) to prevent conflicts with custom fields. A valid BaseFacet requires:_producer: A URI identifying the producer of the metadata (e.g., a git URL)._schemaURL: The URL to the corresponding version of the schema definition.For a full list of standard run facets, refer to the OpenLineage standard facets specification.
The Dataset object represents a data entity in Marquez. Key fields include:
id: A composite identifier consisting of namespace and name.name: The logical name used for identification.physicalName: The actual location/name in the underlying system (e.g., public.mytable).fields: A list of schema elements. Each field has a name, type, and optional tags or description.facets: Metadata extensions._producer (URI) and a _schemaURL (URI) to identify the schema version. Field names within custom facets must not start with an underscore (_).currentVersion: A UUID representing the current version of the dataset metadata.{
"id": {
"namespace": "my-namespace",
"name": "my-dataset"
},
"type": "DB_TABLE",
"name": "my-dataset",
"physicalName": "public.mytable",
"createdAt": "2019-05-09T19:49:24.201361Z",
"updatedAt": "2019-05-09T19:49:24.201361Z",
"namespace": "my-namespace",
"sourceName": "my-source",
"fields": [
{
"name": "a",
"type": "INTEGER",
"tags": []
}
],
"tags": [],
"lastModifiedAt": null,
"description": "My first dataset!",
"facets": {},
"currentVersion": "b1d626a2-6d3a-475e-9ecf-943176d4a8c6"
}Marquez uses specific ID objects to uniquely identify datasets and jobs across namespaces.
namespace (string) and name (string).namespace (string) and name (string).These identifiers are used in the inputs and outputs arrays of a Job to define data lineage.
{
"title": "DatasetId",
"type": "object",
"properties": {
"namespace": { "type": "string" },
"name": { "type": "string" }
}
}