repository·main·Indexed 41 days ago
https://github.com/databrickslabs/lakebridgeAn experimental open-source project from Databricks Labs for data lake exploration and bridging. It includes a Lakebridge Analyzer that assigns complexity scores (LOW, MEDIUM, HIGH, VERY HIGH) to SQL objects and ETL jobs from platforms such as Teradata, DataStage, Talend, SSIS, Alteryx, BODS, Pentaho, and SAS to prioritize migration efforts.
Databricks Labs Lakebridge is an open-source project by Databricks Labs. It is provided for exploration and experimentation. Note that it does not come with formal Databricks Service Level Agreements (SLAs) and is provided AS-IS.
For detailed technical documentation, visit the official documentation site: https://databrickslabs.github.io/lakebridge/
Lakebridge provides specialized conversion guides for migrating logic from various enterprise ETL and data warehouse platforms to Databricks. Currently supported source systems include:
Morpheus is Lakebridge's AST-based SQL transpiler that converts source SQL into Databricks SQL. It provides a strong correctness guarantee: if a file is transpiled without any errors or warnings, it is guaranteed to produce equivalent results on Databricks as the original file did on the source platform.
Supported Source Dialects:
mssql: Microsoft SQL Server, Azure SQL Database, Azure SQL Managed Instance, Amazon RDS for SQL Serversnowflake: Snowflake (including dbt repointing)synapse: Azure Synapse Analytics (dedicated SQL pools)When NOT to use Morpheus:
The Lakebridge Profiler is an experimental feature designed to extract and analyze metadata from database systems to aid in migration planning. It provides insights into source environments by capturing:
Note: The Profiler is currently an Experimental feature. Prerequisites for connection and metrics collection vary by source system.
Lakebridge is a toolkit for managing SQL migrations to Databricks through three primary phases:
The Analyzer classifies SQL objects into four complexity levels. Use these to plan your manual review effort:
Note: For best results with the Analyzer, run the SQL Splitter first if your files contain multiple objects (e.g., stored procedures and tables mixed), so the Analyzer can provide per-object granularity.
The Lakebridge Analyzer assigns one of four complexity levels to each SQL object or ETL job. These scores are used to prioritize migration effort, estimate timelines, and identify objects requiring manual review.
| Level | Meaning | Typical Action |
|---|---|---|
| LOW | Simple object; few statements, no loops, no complex constructs | Transpile directly; little to no manual review expected |
| MEDIUM | Moderate complexity; some loops, 10–30 statements, limited advanced SQL | Transpile; review transpiler warnings before deploying |
| HIGH | High complexity; many loops, 30–50 statements, or advanced constructs (XML, PIVOT) | Plan for manual review after transpilation; test thoroughly |
| VERY HIGH | Highest complexity; heavy loop nesting, 50+ statements, or extreme PIVOT/XML usage | Allocate dedicated review time; consider using Switch for intent-aware conversion |
For target_type=notebook or target_type=sdp, Switch executes the following sequential steps:
analyze_input_files: Scans input directories and stores metadata/content in a Delta table.convert_with_llm: Sends content to a Model Serving Endpoint using prompts to generate new code.validate_sdp is called.validate_python_notebook is called.fix_syntax_with_llm: Automatically attempts to correct errors found during validation.split_code_into_cells: Breaks raw Python code into logical notebook cells (imports, functions, etc.).export_to_notebook: Writes the final .py notebooks to the output directory.convert_notebook_to_sql (Optional): If sql_output_dir is provided, converts the Python notebooks into SQL notebooks.Redshift-specific string and conditional functions are converted as follows:
REGEXP_COUNT(str, pattern) is converted to a combination of LENGTH and REGEXP_REPLACE: LENGTH(str) - LENGTH(REGEXP_REPLACE(str, pattern, '')).DECODE(expr, v1, r1, v2, r2, default) is converted to an inline CASE WHEN expr=v1 THEN r1 WHEN expr=v2 THEN r2 ELSE default END statement.CHARINDEX(substr, str) is converted to INSTR(str, substr) (note that the argument order is swapped).Example conversion:
-- Redshift
SELECT REGEXP_COUNT(email, '[A-Z]'), DECODE(val, 1, 'A', 2, 'B', 'C');
-- Databricks
SELECT LENGTH(email) - LENGTH(REGEXP_REPLACE(email, '[A-Z]', '')), CASE WHEN val=1 THEN 'A' WHEN val=2 THEN 'B' ELSE 'C' END;-- Redshift
UPDATE customers
SET loyalty_points = loyalty_points +
CASE
WHEN REGEXP_COUNT(email, '[A-Z]') > 3 THEN 100
WHEN REGEXP_COUNT(email, '[0-9]') > 2 THEN 50
ELSE 25
END,
status = DECODE(
REGEXP_COUNT(phone, '[0-9]'),
10, 'VERIFIED',
11, 'VERIFIED',
'PENDING'
)
WHERE customer_id IN (
SELECT DISTINCT o.customer_id
FROM orders o
JOIN order_items oi ON o.order_id = oi.order_id
WHERE o.order_date >= CURRENT_DATE - INTERVAL '30 days'
AND o.total_amount > 500
);
-- Databricks
UPDATE customers
SET loyalty_points = loyalty_points +
CASE
WHEN LENGTH(email) - LENGTH(REGEXP_REPLACE(email, '[A-Z]', '')) > 3 THEN 100
WHEN LENGTH(email) - LENGTH(REGEXP_REPLACE(email, '[0-9]', '')) > 2 THEN 50
ELSE 25
END,
status = case when LENGTH(phone) - LENGTH(REGEXP_REPLACE(phone, '[0-9]', ''))=10 then 'VERIFIED' when LENGTH(phone) - LENGTH(REGEXP_REPLACE(phone, '[0-9]', ''))=11 then 'VERIFIED' else 'PENDING' END
WHERE customer_id IN (
SELECT DISTINCT o.customer_id
FROM orders o
JOIN order_items oi ON o.order_id = oi.order_id
WHERE o.order_date >= CURRENT_DATE - INTERVAL '30 days'
AND o.total_amount > 500
);BladeBridge execution is driven by a configuration file. You use this file to define the parameters for your migration project, including:
Lakebridge is a toolkit for migrating data workloads to Databricks, organized into three distinct phases. You can use these phases independently depending on your migration stage:
When to use which:
Choose a report_type based on the level of reconciliation required:
schema: Validates that data types between source and target are the same or compatible. Outputs: schema_comparison, schema_difference.row: Reconciles data at the row level using hash value comparison. Use this when there are no join columns. Outputs: missing_in_src, missing_in_tgt.data: Reconciles data at both row and column level using join_columns to identify specific mismatches. Outputs: mismatch_data, missing_in_src, missing_in_tgt, threshold_mismatch, mismatch_columns.all: A combination of data and schema reports.