Databricks CLI

The my_lakeflow_pipelines project is a template-generated project for managing Lakeflow pipelines. It is organized into two main directories:

• src/: Contains the SQL source code for the pipelines.
• resources/: Contains resource configurations such as jobs and pipelines.

You can work on this project in three ways:

1. Directly in Databricks workspace: Using the Databricks web interface.
2. Locally with an IDE: Using extensions like the Databricks VS Code extension.
3. Via Command Line: Using the Databricks CLI.

The my_lakeflow_integrations bundle provides custom integrations for Lakeflow Jobs. It includes runnable examples and job definitions located in src/my_lakeflow_integrations/examples/ and resources/ respectively.

Available examples:

• slack.send_slack_message (Function): Posts a message to Slack using a Unity Catalog connection.
• wait_for_run_sensor.WaitForRunSensor (Sensor): A sensor that polls a job run until it reaches a terminal state.

The lakeflow-pipelines template is a specialized version of the 'default' template. It is specifically designed for use within the Databricks Workspace UI to facilitate the creation of Lakeflow pipelines.

For a complete list of the properties and configurations that are automatically defaulted when using this template, refer to the databricks_template_schema.json file.

Projects generated using the default-python template follow a specific directory structure:

• src/: Contains the Python source code.
  • src/my_default_python/: Shared Python code intended for use by Databricks jobs and pipelines.
• resources/: Contains resource configurations such as jobs and pipelines.
• tests/: Contains unit tests for the shared Python code.
• fixtures/: Contains data set fixtures, primarily used for testing.

A my_default_python pipeline is organized into several functional directories:

• explorations/: Contains ad-hoc notebooks for exploring processed data.
• transformations/: The core directory containing all dataset definitions and transformations. By convention, each dataset is defined in its own separate file.
• utilities/ (optional): Contains shared utility functions and Python modules.
• data_sources/ (optional): Contains view definitions that describe the source data for the pipeline.

The lakeflow_project pipeline is organized into several functional directories:

• explorations/: Contains ad-hoc notebooks for exploring processed data.
• transformations/: Contains all dataset definitions and transformations. This is the primary location for core logic.
• utilities/ (optional): Contains shared utility functions and Python modules.
• data_sources/ (optional): Contains view definitions that describe the source data for the pipeline.

The my_default_python project follows this directory structure:

• src/: Python source code.
  • src/my_default_python/: Shared Python code intended for use by Databricks jobs and pipelines.
• resources/: Configuration files for Databricks resources (e.g., jobs, pipelines).
• tests/: Unit tests for the shared Python code.
• fixtures/: Data set fixtures, primarily used for testing.

A my_sql_project pipeline is organized into specific directories that separate logic, data definitions, and exploration:

• transformations/: The core directory containing all dataset definitions and transformations. By convention, each dataset is defined in its own separate file.
• explorations/: Contains ad-hoc notebooks used for exploring the data processed by the pipeline.
• data_sources/ (optional): Contains view definitions that describe the source data for the pipeline.
• utilities/ (optional): Contains shared utility functions and Python modules used across the pipeline.

The my_python_project pipeline source code is organized into the following directory structure:

• explorations/: Contains ad-hoc notebooks for data exploration.
• transformations/: Contains all dataset definitions and transformations. This is the primary location for core logic.
• utilities/ (optional): Contains utility functions and Python modules.
• data_sources/ (optional): Contains view definitions describing source data.

The databricks-bundles package provides Python support for Declarative Automation Bundles. This allows developers to manage Databricks resources using a declarative approach within Python environments. The package is organized into several functional modules:

• databricks.bundles.core: Core bundle functionality.
• databricks.bundles.jobs: Management of Databricks Jobs.
• databricks.bundles.pipelines: Management of Databricks Pipelines.
• databricks.bundles.schemas: Management of schemas.
• databricks.bundles.volumes: Management of Unity Catalog Volumes.