dbx (Databricks CLI eXtensions)

dbx supports two distinct prefix types for resolving file paths during deployment. The choice depends on whether you need to access the file via object storage APIs (like Spark) or via a local file system interface (like Python's pathlib).

Standard References (file://)

• Prefix: file://<path_in_project>
• Resolution: Resolves to <artifact storage prefix>/<path_in_project> (e.g., dbfs://..., s3://..., abfss://...).
• Best Use Case: Workflow properties like init scripts or spark_python_file references.
• Access Method: Use object storage compatible APIs (e.g., Spark APIs).

FUSE References (file:fuse://)

• Prefix: file:fuse://<path_in_project>
• Resolution: Resolves to /dbfs/<artifact storage prefix>/<path_in_project>.
• Best Use Case: Passing files to libraries that only support local file system access.
• Constraint: FUSE-based paths only work with dbfs://-based artifact locations (both mounted and non-mounted).

# Accessing a Standard reference via Spark
standard_referenced_path = "dbfs://some/path"
def read_text_payload(_path):
    return "\n".join(spark.read.format("text").load(standard_referenced_path).select("value").toPandas()["value"])

# Accessing a FUSE reference via pathlib
from pathlib import Path
fuse_based_path = "/dbfs/some/path"
payload = Path(fuse_based_path).read_text()

dbx can reference a Cluster Policy in a cluster definition, and it will automatically resolve and merge the policy's fixed properties into the cluster definition during deployment.

When resolution occurs

Resolution is triggered if:

1. policy_id or the legacy policy_name is provided in the new_cluster definition.
2. policy_id starts with the cluster-policy:// prefix.

Resolution Logic

1. Transformation: The policy definition is traversed, and only fixed properties are selected to be converted into a Jobs API compatible format.
2. Deep Update: The policy's fixed properties are deeply merged into the cluster definition.
3. Conflict Check: If the cluster definition contains any keys that are also marked as fixed in the policy, dbx will throw an error.

Limitations

dbx only resolves and verifies fixed policy elements. It does not resolve Forbidden, Limiting, or Allowlist policies; those are handled only during the actual workflow deployment API call by Databricks.

In the context of dbx, DevOps for workflows is the automated process of managing, testing, and rolling out code changes for data or ML workflows.

Core Terminology

• Task: A single unit of work within a workflow (e.g., an ETL task processing tables or an ML task preparing a model). Tasks can have upstream and downstream dependencies.
• Workflow: A composition of one or more tasks. Workflows are characterized by their trigger (e.g., scheduled, event-driven), their type (batch or streaming), and their defined inputs and outputs (ideally Delta tables).
• CI (Continuous Integration): The process of verifying code changes within a development environment. This includes running unit tests, integration tests, and linters to ensure code quality.
• CD (Continuous Delivery): The process of propagating verified code changes to production environments and managing releases.
• DevOps: The combined automated process of accepting code changes and rolling out releases across different environments.

Minimal DevOps Requirements

To implement a DevOps process for workflows, you need:

1. Git-like Repository: A place to store workflow code (e.g., GitHub, GitLab, Azure Repos).
2. Triggers: Events (like a push to a branch or a merge to main) that initiate pipelines.
3. CI/CD Pipelines: Automated steps defined in a YAML-like format (using providers like GitHub Actions or Jenkins) to execute tasks like dependency installation, testing, and artifact deployment.

All-purpose clusters are designed for interactive usage (Notebook exploration, ETL development, ML modeling). In the dbx workflow, they are recommended for the development loop rather than automated production workflows.

Recommended use cases:

• Using dbx execute when developing Python-based workflows.
• Using dbx sync repo for Notebook-based development.

Warnings for Python and JVM package-based projects: Do not use dbx deploy or dbx launch with Python package-based or JVM-based projects on all-purpose clusters. Using existing_cluster_id or existing_cluster_name in your deployment file for these project types can lead to undefined behavior during library installation (e.g., the job may fail to launch if a version of the library is already installed, even if a newer version is provided).

For Python package-based projects, follow the development loop described in the Python package development guide. For JVM projects, follow the JVM development guide.

dbx supports three workflow types. If you do not explicitly specify the workflow_type field, dbx infers it based on the presence of the tasks section:

• jobs-v2.1: Inferred if a tasks section is provided.
• jobs-v2.0: Inferred if the tasks section is NOT provided.
• pipeline: Used for Delta Live Tables (DLT) pipelines.

You can explicitly set the workflow_type field to one of these values: jobs-v2.1, jobs-v2.0, or pipeline.

Supported formats include standard Databricks Jobs API (v2.0 and v2.1) and Delta Live Tables pipeline formats.

build:
  python: "pip"

environments:
  default:
    workflows:

      # Inferred as jobs-v2.1 because 'tasks' is present
      - name: "workflow-in-v2.1-format"
        tasks:
          - task_key: "task1"
            python_wheel_task:
              package_name: "some-pkg"
              entry_point: "some-ep"

      # Inferred as jobs-v2.0 because 'tasks' is absent
      - name: "workflow-in-v2.0-format"
        spark_python_task:
          python_file: "file://some/file.py"

      # Explicitly set as a DLT pipeline
      - name: "workflow-in-pipeline-format"
        target: "some-target-db"
        workflow_type: "pipeline"
        libraries:
          - notebook:
              path: "/Repos/some/path"