DOCSEARCHbyAlgolia
Submit Docs

Databricks SDK for Python

repository·main·Indexed 20 days ago

https://github.com/databricks/databricks-sdk-py

A Python library providing a Pythonic interface to all public Databricks REST APIs for the Databricks Lakehouse. It features the WorkspaceClient for service API access, built-in support for retries, pagination, and long-running operations via Wait objects. The SDK supports multiple authentication flows, including Databricks native (PAT, OIDC), Azure, GCP, and OAuth Authorization Code flow with PKCE.

Tokens
250.4K
Snippets
833
Records
1.3K
Agent score
69%

What's inside databricks-sdk-py

  1. Overview of Databricks SDK for Python

    main
    The Databricks SDK for Python (Beta) provides a programmatic way to interact with the Databricks Lakehouse using Python. It covers all public Databricks REST API operations, allowing developers to automate and accelerate development tasks. While supported for production, users should be aware that future releases may include interface changes.

  2. Manage Databricks SQL assets with the Databricks SDK

    main

    The Databricks SDK for Python provides programmatic access to manage Databricks SQL assets. You can use the SDK to interact with the following components:

    • Warehouses: Manage SQL warehouses used for executing queries.
    • Queries & Query History: Create, manage, and inspect queries and their execution history.
    • Dashboards & Widgets: Manage dashboards and the individual widgets that compose them.
    • Alerts: Configure and manage SQL alerts (including legacy and v2 versions).
    • Statement Execution: Interact with the statement execution API.
    • Permissions: Manage Databricks SQL-specific permissions.
    • Data Sources & Visualizations: Manage data sources and query visualizations.

  5. Configure Unity Catalog governance with the Databricks SDK

    main

    The Databricks SDK for Python allows you to manage Unity Catalog components to implement data governance. You can programmatically configure and manage the following resources:

    • Metastores: The top-level container for Unity Catalog.
    • Metastore Assignments: Mapping metastores to specific workspaces.
    • Catalogs: The logical grouping of schemas.
    • Schemas: Logical groupings within a catalog.
    • Tables: The actual data assets.
    • External Locations: Managed access to cloud storage.
    • Storage Credentials: Identity and access management for cloud storage used by Unity Catalog.

  7. Postgres service dataclasses in Databricks SDK

    main
    The databricks.sdk.service.postgres module contains dataclasses used to represent API requests and responses for Postgres-related services. These classes include specifications (Spec), statuses (Status), and metadata (OperationMetadata) for various resources like Branches, Catalogs, Databases, Endpoints, Projects, and Roles.

  8. Configure Spark Declarative Pipelines via databricks.sdk.service.pipelines

    main
    The databricks.sdk.service.pipelines module provides dataclasses used to represent API requests and responses for Spark Declarative Pipelines. These classes allow you to define pipeline specifications, including ingestion sources, transformers, triggers, and cluster configurations.

  9. Dataclasses in databricks.sdk.service.database

    main

    The databricks.sdk.service.database module contains several dataclasses used to represent API requests and responses for database-related services. Key classes include:

    • Instance Management: DatabaseInstance, DatabaseInstanceRef, DatabaseInstanceRole, DatabaseInstanceRoleAttributes.
    • Catalog & Table Management: DatabaseCatalog, DatabaseTable, SyncedDatabaseTable, SyncedTableSpec.
    • Sync & Pipeline Info: DeltaTableSyncInfo, NewPipelineSpec, SyncedTablePipelineProgress, SyncedTablePosition.
    • Response Objects: ListDatabaseCatalogsResponse, ListDatabaseInstanceRolesResponse, ListDatabaseInstancesResponse, ListSyncedDatabaseTablesResponse.

  10. Manage bundle deployment metadata with BundleDeploymentsAPI

    main

    The BundleDeploymentsAPI service in databricks.sdk.service.bundledeployments is used to manage the lifecycle and metadata of bundle deployments. This includes creating deployments, managing versions, tracking resource operations, and handling deployment locks.

    Key capabilities include:

    • Deployment Lifecycle: Create, get, list, and delete deployments.
    • Version Management: Create new versions (which acquires an exclusive lock), list versions, and complete versions (which releases the lock).
    • Resource & Operation Tracking: Create and update resource operations under a specific version, and manage deployment-level resources.
    • Concurrency Control: Uses version IDs and sequence IDs to prevent concurrent deployments and ensure optimistic concurrency during updates.

  11. Manage identities and workspace access with WorkspaceIamV2API

    main
    The WorkspaceIamV2API (accessible via w.workspace_iam_v2) is used to manage identities (users, service principals, and groups) and their specific access/entitlements within a Databricks workspace. It provides methods to create, delete, update, and list these identities, as well as manage group memberships and workspace assignments.

  12. Manage Unity Catalog entity tag assignments

    main

    Use the EntityTagAssignmentsAPI (accessible via w.entity_tag_assignments) to organize and categorize Unity Catalog entities using tags. Supported entities include catalogs, schemas, tables (including views), columns, and volumes.

    Required Privileges

    To create, update, or delete tags, you must own the entity or have the following privileges:

    • APPLY TAG on the entity
    • USE SCHEMA on the entity's parent schema
    • USE CATALOG on the entity's parent catalog

    If you are working with governed tags, you must also have the ASSIGN or MANAGE permission on the tag policy.