DOCSEARCHbyAlgolia
Submit Docs

Semantic Link Labs

repository·main·Indexed 20 days ago

https://github.com/microsoft/semantic-link-labs

A Python library for Microsoft Fabric notebooks that extends Semantic Link. It provides tools for automating the management of semantic models, reports, capacities, and lakehouses. Key features include migrating Import/DirectQuery models to Direct Lake, automating P SKU to F SKU capacity migrations via labs.migrate_capacities, and performing Best Practice Analyzer (BPA) tasks using labs.run_model_bpa_bulk.

Tokens
38.2K
Snippets
153
Records
173
Agent score
66%

What's inside semantic-link-labs

  1. Migrate Import/DirectQuery models to Direct Lake

    main

    Semantic Link Labs provides an automated process to migrate existing Power BI Import or DirectQuery semantic models to a new Direct Lake model. This is particularly useful for models using Power Query, as it offloads logic to Dataflows Gen2 and creates delta tables in a Fabric lakehouse.

    Prerequisites

    • XMLA Read/Write: Must be enabled for your capacity.
    • Lakehouse: A lakehouse must exist in your Fabric workspace.
    • Workspace Settings: Enable Users can edit data models in the Power BI service (Workspace -> Workspace Settings -> General -> Data model settings).

    Migration Steps

    1. Download the Notebook: Use the Migration to Direct Lake.ipynb notebook.
    2. Setup Workspace: Ensure you are in the 'Fabric' developer experience in your workspace.
    3. Import Notebook: In your workspace, select New -> Import notebook and upload the downloaded file.
    4. Add Lakehouse: Add your target lakehouse to the notebook.
    5. Execute Notebook: Follow the internal notebook instructions to:
      • Create a Power Query Template (.pqt) file.
      • Sync files via OneLake file explorer.
      • Create a Dataflow Gen2 from the .pqt file within your lakehouse (this ensures automatic destination mapping).
      • Publish the Dataflow and wait for delta tables to be created.
      • Generate the new Direct Lake semantic model.
      • Rebind existing reports to the new model.

    What is migrated

    • Standard Objects: Tables, columns, calculation groups, measures, relationships, hierarchies, roles, RLS, perspectives, and translations.
    • Calculated Tables: Migrated as data tables with DAX expressions stored as model annotations (a workaround for Direct Lake limitations).
    • Field Parameters: Migrated as calculated tables.

    Limitations

    • Calculated Columns: Not migrated.
    • Auto date/time tables: Not migrated (disable this in Power BI Desktop and use proper date tables instead).
    • Broken References: References to calculated columns in Field Parameters or DAX expressions will be removed or will break.

  2. Format code with Black

    main

    This project uses black for code formatting. Ensure your code is formatted before submitting a pull request.

    Install Black

    pip install black==25.1.0

    Run Formatting

    Run this command from the repository root to format the src directory:

    python -m black src
    pip install black==25.1.0
python -m black src

  3. Bulk import helper notebooks from GitHub

    main

    You can automatically download and import all available helper notebooks from the semantic-link-labs repository into your chosen Fabric workspace using the labs.import_notebook_from_web function.

    Note: You must specify a workspace_name (as a string) where you want the notebooks to be saved. If left as None, the behavior depends on the underlying implementation (typically defaults to the current workspace).

    import sempy_labs as labs
import requests

workspace_name = None # Update this to the workspace in which you want to save the notebooks
api_url = "https://api.github.com/repos/microsoft/semantic-link-labs/contents/notebooks"
response = requests.get(api_url)
files = response.json()
notebook_files = {file['name'][:-6]: file['html_url'] for file in files if file['name'].endswith('.ipynb')}

for file_name, file_url in notebook_files.items():
    labs.import_notebook_from_web(notebook_name=file_name, url=file_url, workspace=workspace_name)

  4. Import Semantic Link Labs into a Fabric notebook

    main

    After installing the library (e.g., via %pip install semantic-link-labs), use the following import statements to access the various modules and sub-packages. The library is organized into specialized modules for different Fabric components like lakehouses, reports, semantic models, and more.

    import sempy_labs as labs
import sempy_labs.lakehouse as lake
import sempy_labs.report as rep
from sempy_labs import admin, deployment_pipeline, directlake, environment, eventstream, graph, graph_model, migration, mirrored_azure_databricks_catalog, ml_model, snowflake_database, sql_database, theme, variable_library, warehouse, semantic_model, eventhouse, dataflow
from sempy_labs.tom import connect_semantic_model
from sempy_labs.report import connect_report

  5. Build and test the semantic-link-labs .whl file

    main

    To contribute to the project or test local changes in a Fabric notebook, you must build a wheel (.whl) file and install it via the notebook's Resources section.

    1. Build the wheel file

    In your local cloned repository terminal, run:

    python -m build

    This creates a .whl file in the dist folder.

    2. Install in a Fabric Notebook

    1. Open a notebook in a Fabric workspace.
    2. In the Explorer tab, navigate to Resources (do not use 'Files').
    3. Upload the .whl file from your dist folder.
    4. Click the ellipsis (...) next to the uploaded file and select Copy relative path.
    5. In a notebook cell, run the following command, replacing the placeholder with your copied path:
    %pip install "<COPIED_RELATIVE_PATH>"

  6. Load Semantic Link Labs into a custom Fabric environment

    main

    To avoid running %pip install in every notebook session, you can add semantic-link-labs to a custom Fabric environment. This makes the library persistently available in your workspace.

    1. Create a custom environment

    1. Navigate to your Fabric workspace.
    2. Click New -> More options.
    3. Within Data Science, click Environment.
    4. Name your environment and click Create.

    2. Add semantic-link-labs as a library

    1. Within Public libraries, click Add from PyPI.
    2. Enter semantic-link-labs.
    3. Click Save at the top right.
    4. Click Publish at the top right.
    5. Click Publish All.

    3. Update your notebook

    1. Navigate to your Notebook.
    2. Select your newly created environment from the Environment dropdown in the navigation bar at the top of the notebook.

    Note: You must wait for the environment to finish publishing before updating your notebook.

  7. Set up a local development environment

    main

    To contribute to Semantic Link Labs, set up a local Python virtual environment and install the necessary build tools.

    Create and Activate Virtual Environment

    Windows:

    python -m venv venv
.\venv\Scripts\Activate.ps1

    Mac/Linux:

    python3 -m venv venv
source venv/bin/activate

    Install Build Tools

    Once the environment is active, install the build module:

    pip install build
    python -m venv venv
# Windows activation:
.\venv\Scripts\Activate.ps1
# Mac activation:
source venv/bin/activate

pip install build

  9. Manage Fabric mirrored databases

    main
    The sempy_labs._mirrored_databases module provides a set of wrapper functions for managing Fabric mirrored databases via REST APIs. You can list, create, delete, and manage the mirroring lifecycle (start/stop) of databases. Most functions accept an optional workspace parameter (name or UUID); if omitted, the library resolves to the workspace of the attached lakehouse or the current notebook.

  10. Configure ADLS Gen2 for Semantic Model Management

    main

    Before performing backup, copy, or restore operations, you must ensure both your source and target workspaces are associated with an Azure Data Lake Storage (ADLS) Gen2 storage account.

    Steps to configure:

    1. Create an ADLS Gen2 storage account in the Azure Portal.
    2. In your Fabric workspace, navigate to Workspace settings.
    3. Select Azure connections.
    4. Within the Azure Data Lake Gen2 Storage section, click Configure.
    5. Enter your Subscription, Resource Group, and Storage Account details.
    6. Click Save.

    Repeat these steps for both the source and target workspaces.

  11. Requirements for Capacity Migration

    main

    Before performing capacity migrations, ensure you meet the following requirements:

    • Azure Subscription: An active subscription is required.
    • App Registration: An Azure App must be registered with the Contributor role at the scope where resources will be created (often the subscription level).
    • Azure Key Vault:
      • Must be set up within your Azure Subscription.
      • Must store secrets for Tenant ID, Client ID (Application ID), and Client Secret.
      • The user executing the notebook must have the Key Vault Secrets User role.
    • Fabric Permissions: The user should be a tenant admin to ensure authority to manage capacities and workspaces.

  12. Rebind reports to the new Direct Lake semantic model

    main

    Once the Direct Lake model is ready, you must rebind your reports to use the new dataset instead of the old one.

    To rebind all reports in a workspace: Use rep.report_rebind_all.

    To rebind a single specific report: Use rep.report_rebind.

    # Rebind all reports
rep.report_rebind_all(
    dataset=dataset_name,
    dataset_workspace=workspace_name,
    new_dataset=new_dataset_name,
    new_dataset_workspace=new_dataset_workspace_name,
    report_workspace=None
)

# Rebind one report
report_name = 'MyReport'
rep.report_rebind(
    report=report_name,
    dataset=new_dataset_name,
    report_workspace=workspace_name,
    dataset_workspace=new_dataset_workspace_name)