Databricks Terraform Provider

The databricks_share resource manages a Delta Sharing share, which is a read-only collection of tables, views, volumes, and other objects shared with recipients.

Important Constraints:

• This resource must be used with a workspace-level provider.
• In Unity Catalog-enabled workspaces, a share is a securable object contained within a databricks_metastore. Removing a share from the metastore revokes access for all recipients.
• The object blocks are treated as an ordered list. To ensure consistent and readable Terraform diffs, it is recommended to define object blocks in alphabetical order by their name argument.

Key Object Types:

• TABLE, VIEW, MATERIALIZED_VIEW, STREAMING_TABLE, FOREIGN_TABLE (Use shared_as for aliasing).
• VOLUME, MODEL, NOTEBOOK_FILE, FUNCTION (Use string_shared_as for aliasing).
• SCHEMA (Can include all current and future tables).
• NOTEBOOK_FILE (Requires base64 encoded content).

resource "databricks_share" "example" {
  name = "my_share"
  object {
    name             = "catalog.schema.table"
    data_object_type = "TABLE"
  }
}

The databricks_postgres_branch resource allows you to create and manage independent Postgres database environments within a project. These branches use copy-on-write to share storage with a parent branch, making them ideal for isolated development, testing, point-in-time recovery, and safe data exploration.

Hierarchy and Lifecycle

• Hierarchy: A branch belongs to a databricks_postgres_project. A branch can contain endpoints, databases, and roles.
• Lifecycle: This resource is used for managing updates to existing branches. It does not control the creation or deletion of the branch itself; the branch's lifecycle is controlled by the parent project. Removing the resource from Terraform only removes it from the state, not from Databricks. To delete an actual branch, you must delete the project it belongs to.
• Root Branch: Every project has an implicitly created root branch named production. To manage this existing branch in Terraform, you must set replace_existing = true.

resource "databricks_postgres_project" "this" {
  project_id = "my-project"
  spec = {
    pg_version   = 17
    display_name = "My Project"
  }
}

# Managing the implicit 'production' branch
resource "databricks_postgres_branch" "production" {
  branch_id = "production"
  parent    = databricks_postgres_project.this.name
  spec = {
    no_expiry = true
  }
  replace_existing = true
}

The databricks_alert_v2 resource manages Databricks SQL alerts. These alerts monitor query results on a schedule and trigger notifications when specific conditions (thresholds) are met.

Key Capabilities

• Alert Evaluation: Compare query results against fixed values or other columns using operators like GREATER_THAN, EQUAL, etc. You can also define behavior for empty results.
• Notifications: Send alerts to user email addresses or configured notification destinations. You can configure alerts to notify when an alert returns to a normal state (notify_on_ok) and set a retrigger_seconds interval to control notification frequency.
• Scheduling: Uses Quartz cron syntax for scheduling, with support for specific timezones and pause/unpause status.
• Identity Management: Use the run_as block to specify if the alert runs as a specific workspace user or a service principal (requires servicePrincipal/user role).

resource "databricks_alert_v2" "basic_alert" {
  display_name = "High Error Rate Alert"
  query_text   = "SELECT count(*) as error_count FROM logs WHERE level = 'ERROR' AND timestamp > now() - interval 1 hour"
  warehouse_id = "a7066a8ef796be84"
  parent_path  = "/Users/user@example.com"
  
  evaluation = {
    source = {
      name        = "error_count"
      display     = "Error Count"
      aggregation = "COUNT"
    }
    comparison_operator = "GREATER_THAN"
    threshold = {
      value = {
        double_value = 100
      }
    }
    empty_result_state = "OK"
    
    notification = {
      subscriptions = [
          {
            user_email = "user@example.com"
          }
      ]
      notify_on_ok = true
    }
  }
  
  schedule = {
    quartz_cron_schedule = "0 0/15 * * * ?"  # Every 15 minutes
    timezone_id          = "America/Los_Angeles"
    pause_status         = "UNPAUSED"
  }
}

The databricks_feature_engineering_kafka_config resource (currently in Private Preview) allows you to define a Kafka configuration for machine learning feature engineering. This configuration enables streaming data ingestion from Kafka topics into a Databricks-managed Delta table for querying and training.

Key capabilities include:

• Consuming from specific topics, patterns, or partitions.
• Defining schemas for Kafka keys and values (Avro, JSON, or Protobuf).
• Configuring Mutual-TLS (mTLS) authentication using Unity Catalog volumes.
• Setting up backfill sources from existing Delta tables to populate historical data.
• Managing data ingestion into a managed Delta table destination.

# Example usage pattern (requires specific arguments based on your Kafka setup)
resource "databricks_feature_engineering_kafka_config" "this" {
  bootstrap_servers = "host1:9092,host2:9092"
  subscription_mode  = "subscribe"
  subscription_details = {
    subscribe = "my_topic"
  }
  auth_config = {
    # ... auth configuration ...
  }
  # ... schema and ingestion configuration ...
}

The databricks_sql_global_config resource configures the security policy, instance profiles, and data access properties for all databricks_sql_endpoint resources within a workspace.

Critical Requirements & Behaviors:

• Administrator Privileges: You must be an administrator to use this resource.
• Workspace-level Provider: This resource can only be used with a workspace-level provider.
• Restart Impact: Changing any parameters of this resource will restart all running databricks_sql_endpoint instances in the workspace.

Use this resource to enforce global settings like security_policy, data_access_config (for Hive metastores or Hadoop configurations), and sql_config_params (to override default session behaviors).

resource "databricks_sql_global_config" "this" {
  security_policy      = "DATA_ACCESS_CONTROL"
  instance_profile_arn = "arn:...."
  data_access_config = {
    "spark.sql.session.timeZone" : "UTC"
  }
}