repository·main·Indexed 25 days ago
https://github.com/openlineage/openlineageAn open standard for collecting metadata and data lineage by instrumenting jobs during execution. It provides a unified model of jobs, runs, and datasets to enable consistent observability across diverse data tools. The documentation covers the Go SDK for managing run lifecycles and event emission, as well as Java transport dependencies for Amazon DataZone, GCP Lineage (Dataplex), and Google Cloud Storage.
The OpenLineage Java SDK is used to generate and emit OpenLineage events to backends. The SDK provides several core data structures and mechanisms:
RunEvent: Represents an event in a lineage run.RunState: Represents the state of a run.Run: Represents a specific execution.Job: Represents the logical unit of work.Dataset: Represents the data entities involved.Facets: Various metadata components that can be attached to Run, Job, or Dataset objects.Transport: An abstraction used to carry lineage events to target endpoints (e.g., via HTTP). The library provides various built-in transports.With the introduction of static lineage in version 1.0, the following use cases are now supported:
The OpenLineage test suite utilizes three distinct workflows to ensure compatibility across different scenarios. These workflows automate component testing, report generation, and failure detection. Depending on the trigger, the scope of tested components and the version of the OpenLineage specification used will vary:
Goal: Update compatibility data. Trigger: Periodic runs that check for new releases of OpenLineage or its components. Scope: Tests both Producers and Consumers. Component Selection: Only components with new releases are tested, or all components if a new OpenLineage release is detected. OpenLineage Versions: Uses official Release Versions. Additional Actions: Notifies about new failures, updates test reports, and updates compatibility information.
Goal: Notify developers of potential backward compatibility issues when the specification changes.
Trigger: Periodic runs that check for updates to the specification in the OpenLineage main branch.
Scope: Tests Producers and Consumer Input Events.
Component Selection: All Producers and Consumer Input Events are tested.
OpenLineage Versions: Uses the Latest snapshot version from the main branch.
Additional Actions: Notifies about new failures.
Goal: Ensure that changes made to the test suite itself do not introduce new failures. Trigger: A Pull Request (PR) submitted to the Test Suite repository. Scope: Tests Producers, Consumers, and Consumer Input Events. Component Selection: All Producers, Consumers, and Consumer Input Events are tested. OpenLineage Versions: Uses official Release Versions.
The OpenLineage Compatibility Tests are a suite of automated checks designed to validate how different versions of OpenLineage interact with various producers (components that emit events) and consumers (components that ingest events).
These tests are hosted in a dedicated GitHub repository and use GitHub Actions to continuously verify compatibility. The results are used to generate compatibility tables in the official documentation.
Validation Types:
The following integration packages are maintained locally within this repository:
Note: The Apache Airflow integration is maintained in the official Apache Airflow repository rather than locally here.
OpenLineage can be implemented by using official clients to send events to an HTTP backend or by using existing integrations that automatically collect metadata as OpenLineage events.
If you are building custom tools, you can use the following clients to emit OpenLineage events via HTTP:
Metadata can be collected automatically from common data tools using these integrations:
The OpenLineage Spark Agent uses JVM instrumentation to automatically emit OpenLineage metadata from Spark applications. This allows you to capture lineage information without manually instrumenting your Spark code.
For detailed setup, refer to the official documentation links:
The OpenLineage Python client is organized into several functional modules for interacting with the OpenLineage specification:
openlineage.client.client: The primary entry point for client operations.openlineage.client.event_v2: Handles OpenLineage event models (v2).openlineage.client.facet & openlineage.client.facet_v2: Modules for managing facets (metadata extensions).openlineage.client.run: Provides abstractions for representing job runs.openlineage.client.serde: Handles serialization and deserialization of OpenLineage data.openlineage.client.transport: Contains various transport implementations for sending metadata (e.g., HTTP, Kafka, File, Console).The Spark3 module provides common code for Apache Spark 3.x.y versions where x >= 2. This includes support for:
The OpenLineage Python client can be configured using four main sections:
Configuration is read only at client creation time. The precedence (from highest to lowest) is:
OpenLineageClient constructor via the config parameter.openlineage.yml file.OPENLINEAGE__ prefix.OPENLINEAGE_URL).If no configuration is provided, the client defaults to ConsoleTransport, printing events to the console.