DOCSEARCHbyAlgolia
Submit Docs

Smart Data Lake Builder (SDLB) Documentation

website·Indexed 32 days ago

https://smartdatalake.ch/

Documentation for Smart Data Lake Builder (SDLB), a tool for building data pipelines. It covers core concepts including DataObjects, Actions, and transformers, as well as schema inference, execution modes, and data quality metrics. The guide provides details on extending the system via custom traits, configuring pipelines using HOCON, and deploying across various environments including Spark, Snowflake, and Azure.

Tokens
59.2K
Snippets
232
Records
374
Agent score
57%

What's inside Smart Data Lake Builder

  1. Overview of SDLB UI capabilities

    The SDLB UI is a technical data catalog designed to visualize, track, and manage data pipelines. It provides visibility into data pipeline execution (workflows) and showcases dependencies through a lineage graph. Key features include a workflow dashboard for execution metrics, a configuration viewer for metadata and documentation, and schema/column statistics for DataObjects.

  2. Overview of CustomDataFrameAction

    CustomDataFrameAction is used to transform data between many input and many output DataObjects using DataFrames. It is the primary action for operations that cannot be expressed as a 1:1 relation, such as joins, unions, and fan-outs. While many-to-many (n:m) relations are possible, it is recommended to stick to n:1 or 1:m to maintain accurate lineage and impact analysis. For simple 1:1 relations, CopyAction is preferred.

  3. Overview of the SDLB Remote Agent feature

    The Agent feature (Experimental) allows a main SDLB instance to send execution instructions to a Remote Agent. The Remote Agent executes the action in its own environment and returns only the schema of the resulting data (via an empty SparkSubFeed) to the main instance. The main instance then fetches the actual data using its own connection. This architecture is typically used to bridge environments, such as copying data from an on-premise source to the cloud.

  4. Overview of Smart Data Lake Builder (SDLB)

    Smart Data Lake Builder (SDLB) is an open-source automation framework implemented in Java/Scala designed to simplify the loading and transforming of data. It is built on Apache Hadoop and Apache Spark and supports a wide variety of data sources and file formats, including HadoopFS, Hive, DeltaLake, JDBC, Splunk, Webservice, SFTP, Kafka, JMS, Excel, and Access. It is primarily used to create data pipelines for Lakehouse architectures, Data Products, and Data Mesh implementations.

  5. Overview of SDLB Execution Engines

    An execution engine in SDLB is a technology or library used to transform data. Data is transported between DataObjects and Actions using a structure called a SubFeed. SDLB can combine different execution engines within a single data pipeline or job.

    CategoryExecution EngineSubFeed NameEngine ConnectionDescriptionSupported ActionsSupported DataObjects
    Java-Byte-StreamFile EngineFileSubFeed-Transfers Byte-Streams without content knowledgeFileTransferAction, CustomFileActionHadoopFileDataObjects, WebserviceFileDataObject, SFtpFileDataObject
    Generic DataFrame APISpark EngineSparkSubFeedSparkClassicConnectionSpark DataFrame API in a session inside the SDLB processCopyAction, CustomDataFrameAction, UpsertAction, HistorizeActionHadoop/SparkFileDataObject, AccessTableDataObject, AirbyteDataObject, CustomDfDataObject, DeltaLakeTableDataObject, HiveTableDataObject, IcebergTableDataObject, JdbcTableDataObject, JmsDataObject, KafkaTopicDataObject, SnowflakeTableDataObject, SplunkDataObject, TickTockHiveTableDataObject
    Generic DataFrame APISpark Connect EngineSparkConnectSubFeedSparkConnectConnectionSpark DataFrame API on a remote Spark Connect serverCopyAction, CustomDataFrameAction, UpsertAction, HistorizeActionSparkConnectTableDataObject, DeltaLakeTableDataObject, IcebergTableDataObject
    Generic DataFrame APISnowflake-Snowpark EngineSnowparkSubFeed-Snowflake Snowpark DataFrame APICopyAction, CustomDataFrameActionSnowflakeTableDataObject
    ParameterParameter EngineParameterSubFeed-Coordinates work outside SDLB (e.g., script results, ML training)CustomScriptAction, MLflowTrainAction inputs/outputsall DataObjects

  6. Overview of Smart Data Lake Builder core capabilities

    Smart Data Lake Builder provides an automated approach to data pipelines to reduce manual effort and increase efficiency. It utilizes a declarative, metadata-driven architecture that supports a spectrum of development needs, from low-code configuration pipelines to complex custom code integrations in various languages. The platform is open-source (GPL-3.0), preventing vendor lock-in and supporting flexible deployment across local environments, on-premises infrastructure, and public hyperscalers.

  7. Overview of available SDLB Actions

    Actions define the dependencies and transformations between input and output DataObjects. While most Actions follow a 1:1 relationship, some support n:m (many-to-many) mappings. It is recommended to use 1:1, 1:n, or n:1 mappings to maintain clear metadata lineage unless n:m is strictly necessary.

  8. Deploy Smart Data Lake Builder

    Smart Data Lake Builder is designed to be environment-agnostic and can be deployed locally, on a private server, or within a large cluster using on-premise or cloud infrastructure. It can be executed as a Java application (for example, via spark-submit) or run within a container.

  9. Understand the SmartDataLakeBuilder execution phases

    SmartDataLakeBuilder (SDLB) uses an 'early validation' approach to fail as quickly as possible if errors exist. Every run progresses through four distinct phases:

    1. Parse configuration: Validates configuration files for missing required attributes or misspelled blocks (e.g., actions {}). Provides typo suggestions.
    2. DAG prepare: Validates preconditions, including the existence of Connections and DataObject structures.
    3. DAG init: Validates the entire lineage of Actions. For Spark Actions, it validates the DataFrame lineage; if a column referenced in a later Action does not exist, the execution fails here.
    4. DAG exec: The only phase where data is effectively transferred. Execution Modes are applied to select data and execute Actions.

  10. Distinguish between Execution Engines and Execution Environments

    The Execution Environment is where the JVM runs (e.g., a Spark Cluster via spark-submit), while the Execution Engine is the library used to transform data (e.g., Snowpark). SDLB chooses the engine independently of the environment based on DataObjects, Actions, Transformations, and engine connections.

    Example: If an action uses only SnowflakeTableDataObjects, SDLB will use the Snowpark-Engine, even if the JVM is running inside a Spark Cluster. The Spark Connect engine further decouples this, as the Execution Environment no longer requires a Spark installation—only a small container is needed to run the SDLB job while processing happens on the remote Spark Connect server.

  11. Use UpsertAction to maintain current state (SCD1)

    UpsertAction copies data from one input to one output DataObject, keeping only the latest version of every record identified by the output table's primary key. This implements the Slowly Changing Dimension Type 1 (SCD1) pattern: when a record changes in the source, the existing version is overwritten. Unlike CopyAction, records no longer delivered by the source are preserved in the output. For full history (SCD2), use HistorizeAction instead.

  12. Apply transformers to Spark actions

    Transformers can be added to Spark-related actions to modify data before it is passed to the next action or written to a Data Object. Multiple transformers can be defined for a single action; they are processed sequentially, where the output of one transformer becomes the input for the next. Transformers are categorized into two types: 1-to-1 (one input Data Object to one output) and many-to-many (one or more inputs to potentially many outputs). For better lineage and impact analysis, it is recommended to create actions with only one output Data Object.
    SampleWhitelistAction {
    type = CopyAction
    inputId = someInput
    outputId = someOutput
    transformers = [{
        type = WhitelistTransformer
        columnWhitelist = [includedColumn1, anotherColumn2]
    }]
}