repository·main·Indexed 42 days ago
https://github.com/databrickslabs/overwatchA deprecated telemetry unification tool for Databricks and Spark deployments designed to provide governance and actionable intelligence. It enables the retrieval of platform data—including jobs, clusters, and SQL history—as Spark DataFrames and provides utilities for persisting this data into Delta tables with support for parallel API calls and automated merge operations.
Use the following SBT commands to clean and build the project. Use package for standard packaging or assembly to create a fat JAR.
sbt clean package
sbt clean assemblyTo ensure compatibility with the project, your development environment must match the dependencies used by Databricks runtime 10.4 LTS. Ensure you have the following versions installed:
SBT (Scala Build Tool) can be installed in two ways:
Critical Requirement: Regardless of the installation method, ensure that your SBT installation is explicitly pointing to the correct Java installation (Zulu 8.56.0.21) to avoid build failures.
The Migration class supports three distinct types of configuration storage. The logic used to update the storage_prefix is determined by the format of the configPath provided:
| Format | Detection Logic | Update Mechanism |
|---|---|---|
| CSV File | Ends with .csv | Reads CSV, updates storage_prefix column where etl_database_name matches, and overwrites the file. |
| Delta Path | Contains / (and not .csv) | Executes a Spark SQL UPDATE statement on the Delta location: update delta.$configPath set storage_prefix = '$target_storage_prefix' WHERE etl_database_name = '$sourceETLDB' |
| Delta Table | No / detected | Executes a Spark SQL UPDATE statement on the table: update $configPath set storage_prefix = '$target_storage_prefix' WHERE etl_database_name = '$sourceETLDB' |
When optimizing certain primary targets, PostProcessor automatically includes related intermediate tables to ensure data consistency and prevent orphaned data. This prevents forcing an optimize on every single pipeline run by tying intermediate tables to their leader tables.
Specifically:
audit_log_bronze is marked for optimization, the following are also included: processedEventLogs, clusterEventsErrorsTarget, pipelineStateTarget, cloudMachineDetail, and dbuCostDetail.spark_events_bronze is marked, processedEventLogs is also included.Optimization parameters for spark_events_bronze are specialized: it uses a smaller maxFileSizeMB = 32, while other tables use maxFileSizeMB = 128.
Overwatch is provided AS-IS and is not formally supported by Databricks with Service Level Agreements (SLAs).
To protect against accidental data loss, Overwatch requires an explicit Spark configuration to be set before running destructive commands (like fastDropTargets).
You must set the Spark configuration overwatch.permit.db.destruction to a value exactly matching the name of the target database you intend to modify.
If this configuration is missing or incorrect, a com.databricks.labs.overwatch.utils.BadConfigException will be thrown.
-- Example: If your target database is 'my_prod_db'
SET overwatch.permit.db.destruction = my_prod_db;The ApiEnv object (passed to ApiCallV2) controls the core authentication and connection settings:
workspaceURL: The base URL of the Databricks workspace.rawToken: The PAT token used for Bearer authentication.packageVersion: Used in the User-Agent header.successBatchSize: The number of responses to accumulate before writing a micro-batch to the successTempPath (if batchPersist is enabled).errorBatchSize: The number of errors to accumulate before writing to the error path.threadPoolSize: The number of threads used during parallel API calls.apiWaitingTime: The timeout threshold for waiting for parallel calls to complete.In multi-workspace deployments where target.requiresLocking is true, Overwatch uses a global table lock to prevent race conditions during parallel writes. If a table is locked, the process will wait until the lock is released or the timeout is reached.
You can control this timeout by setting the following Spark configuration on your cluster:
overwatch.tableLockTimeout (milliseconds)
1200000 (20 minutes).TABLE LOCK TIMEOUT errors.spark.conf.set("overwatch.tableLockTimeout", "3600000") // Set to 1 hourThe Migration.process method automates the migration of an Overwatch ETL database and its associated consumer database to a new target location. It performs a deep snapshot of the source, updates the configuration files (CSV, Delta path, or Delta table) with the new storage prefix, and then drops the original source and consumer databases.
Workflow:
DEEP clone of the source ETL database to the migrateRootPath.storage_prefix in the configuration to match the new target path for the specific sourceETLDB.CASCADE.Parameters:
sourceETLDB: The name of the source ETL database.migrateRootPath: The target path where the migration will be performed.configPath: The path to the configuration file (supports .csv files, Delta paths, or Delta table names).tablesToExclude: (Optional) A string representing table names to exclude from the migration. Defaults to an empty string.// Example usage of the Migration process
Migration.process(
sourceETLDB = "my_source_etl_db",
migrateRootPath = "/mnt/target/path/",
configPath = "/mnt/configs/overwatch_config.csv",
tablesToExclude = "table1,table2"
)