DOCSEARCHbyAlgolia
Submit Docs

Microsoft Fabric CI/CD

repository·main·Indexed 43 days ago

https://github.com/microsoft/fabric-cicd

A Python library that enables code-first CI/CD automation for Microsoft Fabric workspaces. It allows developers to manage deployments via source control and integrate workspaces into deployment frameworks without interacting directly with Microsoft Fabric APIs.

Tokens
36.5K
Snippets
79
Records
130
Agent score
55%

What's inside fabric-cicd

  1. Overview of fabric-cicd

    main
    fabric-cicd is a Python library designed for Microsoft Fabric workspaces. It enables code-first Continuous Integration / Continuous Deployment (CI/CD) automations, allowing developers to integrate source-controlled workspaces into deployment frameworks without interacting directly with Microsoft Fabric APIs.

  2. Organize your deployment directory structure

    main

    The library deploys items from a directory containing files and directories committed via the Fabric Source Control UI. The repository_directory should contain these committed items and an optional parameter.yml file.

    Expected structure:

    /<your-directory>
    /<item-name>.<item-type>
        ...
    /<item-name>.<item-type>
        ...
    /<workspace-subfolder>
        /<item-name>.<item-type>
            ...
        /<item-name>.<item-type>
            ...
    /parameter.yml
    /<your-directory>
    /<item-name>.<item-type>
        ...
    /<item-name>.<item-type>
        ...
    /<workspace-subfolder>
        /<item-name>.<item-type>
            ...
        /<item-name>.<item-type>
            ...
    /parameter.yml

  3. Core concepts and deployment expectations of fabric-cicd

    main

    When using fabric-cicd for Microsoft Fabric CI/CD, keep the following architectural behaviors in mind:

    • Full Deployment Model: The library performs a full deployment every time. It does not calculate or consider commit diffs to perform incremental updates.
    • Identity Context: Deployments are executed into the tenant of the identity currently running the code.
    • Item Constraints: The library only supports Microsoft Fabric items that have Source Control enabled and provide Public Create/Update APIs.
    • Parameter Replacement: If you use a parameter.yml file for environment-specific values, you must provide the environment parameter to FabricWorkspace to enable replacement.

  4. How fabric-cicd deployment works

    main

    The fabric-cicd library operates on a full deployment model. It does not perform diffing or inspect commit history; instead, it ensures the target workspace converges to the exact state defined in your repository.

    The Deployment Process:

    1. The library reads item definition files from your repository directory.
    2. It passes the content of these files as payloads to the Fabric REST APIs.
    3. The APIs create or update the corresponding items in the target workspace.
    4. Every item type specified in item_type_in_scope is processed on every run.

    Managing Orphaned Items: Because the library performs a full deployment of what is in the repo, items that exist in the workspace but have been deleted from the repository will remain there (orphaned). To ensure the workspace perfectly matches the repository, use the unpublish_all_orphan_items() method to remove these items.

  5. Authentication requirements for fabric-cicd

    main

    The fabric-cicd library requires an explicit token_credential parameter when initializing the FabricWorkspace object.

    Important Security Changes:

    • The Default Credential (DefaultAzureCredential fallback) is no longer supported.
    • Implicit Fabric Notebook authentication (omitting the token_credential parameter) is no longer supported.

    fabric-cicd supports any object that implements the TokenCredential interface. Avoid hardcoding credentials; instead, use environment variables or secret management services. For Service Principal (SPN) authentication, use az login --service-principal or Connect-AzAccount -ServicePrincipal in your CLI/PowerShell flows.

  6. Configure environment-specific values in `config.yml`

    main

    All configuration fields in config.yml support environment-specific mappings. Instead of a single value, you can provide a dictionary where keys are environment names (e.g., dev, test, prod).

    Example mapping:

    core:
    workspace_id:
        dev: "dev-id"
        prod: "prod-id"

    Behavior for missing environments:

    • Required fields (core.workspace_id, core.workspace, core.repository_directory): If the target environment is missing from the mapping, a validation error is raised.
    • Important optional fields (core.item_types_in_scope, core.parameter): If the environment is missing, a warning is logged and the setting is skipped.
    • Other optional fields (e.g., publish.skip, publish.exclude_regex): If the environment is missing, a debug message is logged (if debug logging is enabled) and the setting is skipped.
    core:
    workspace_id:
        dev: "dev-workspace-id"
        test: "test-workspace-id"
        prod: "prod-workspace-id"

publish:
    skip:
        dev: true
        # test and prod default to false

  7. Manage Variable Library environments

    main

    The active value set of a Variable Library is determined by the environment field passed into the FabricWorkspace object. If no environment is specified, the active Value Set remains unchanged.

    Constraints and Troubleshooting

    • No Programmatic Renaming: Variable Libraries do not support programmatically changing the name of the active value set.
    • Deployment Failures: If an active value set is renamed or removed after the initial deployment, the deployment will fail.
    • Resolution: If a deployment fails due to value set changes, manual intervention in the Fabric UI is required before restarting the deployment.

  8. Parameterize Reports for Semantic Model connections

    main

    When deploying Reports, how they connect to Semantic Models depends on the reference type:

    • External Semantic Models: Reports connected to Semantic Models outside the same workspace always point to the original model. You must parameterize these in the parameter.yml file.
    • Internal Semantic Models (byPath): Reports using byPath references to Semantic Models within the same workspace are automatically re-pointed to the new item in the target workspace.
    • Internal/External Semantic Models (byConnection): Reports using byConnection can be updated using find_replace or key_value_replace parameters in parameter.yml to modify workspace IDs, semantic model names, or semantic model IDs in the connection string.

  9. Filter parameterization targets with `item_type`, `item_name`, and `file_path`

    main

    You can restrict where find_replace operations are applied by using optional filter fields:

    • item_type: Limits matching to specific Fabric item types (e.g., Notebook). Types must be valid within the scope of deployment.
    • item_name: Limits matching to specific item names. This can be a single string or an array of strings. Names must match the exact names in the repository_directory.
    • file_path: Limits matching to specific files. Supports:
      • Absolute paths: Starting from the drive root.
      • Relative paths: Relative to the repository_directory.
      • Wildcard paths: Uses glob patterns like * (any characters) or ** (any directory depth).

    Note: String input values should be wrapped in quotes, and special characters like \ in file_path must be escaped.

    find_replace:
    - find_value: "my-dev-server"
      replace_value:
          PPE: "my-ppe-server"
          PROD: "my-prod-server"
      ignore_case: "true"
      item_type: "Notebook"
      item_name: ["Hello World", "Hello World Subfolder"]
      file_path:
          - "/Hello World.Notebook/notebook-content.py"
          - "/subfolder/Hello World Subfolder.Notebook/notebook-content.py"

  10. How dynamic replacement variables work

    main

    Fabric-cicd supports dynamic replacement variables that allow you to resolve IDs at deployment time without hardcoding them in your parameter.yml file. This is essential when deploying items that depend on other items being deployed in the same process.

    Common Dynamic Variables:

    • $items.<ItemType>.<ItemName>.$id: Resolves the ID of a specific item (e.g., $items.Lakehouse.Example_LH.$id).
    • $workspace.$id: Resolves the ID of the target workspace.
    • $items.Lakehouse.<lakehouse_name>.$sqlendpointid: Resolves the SQL Endpoint ID for a specific Lakehouse.

    Usage Note: When using find_replace with regex, the regex pattern must include a capture group (), and the find_value must always match group 1. The value captured in this group will be dynamically replaced by the variable provided in replace_value.

  11. Use the `_ALL_` environment key for universal replacements

    main

    The _ALL_ key (case-insensitive) can be used within the replace_value dictionary of find_replace, key_value_replace, or spark_pool parameters. It applies the replacement to all target environments.

    Constraint: When using _ALL_, it must be the only key in the replace_value dictionary. Using ALL (without underscores) is treated as a standard environment key.

    find_replace:
    - find_value: "db52be81-c2b2-4261-84fa-840c67f4bbd0"
      replace_value:
          _ALL_: "$items.Lakehouse.Example_LH.$id"

  12. Use dynamic replacement variables for workspace and item metadata

    main

    Fabric-CICD supports dynamic variables that resolve to metadata from the target workspace or deployed items. These can be used in the replace_value field of find_replace and key_value_replace parameters.

    Workspace Variables

    Used to resolve properties of the target workspace or a specific named workspace.

    • $workspace.$id or $workspace.id: Target workspace ID.
    • $workspace.$name: Target workspace display name.
    • $workspace.$name_encoded: URL-encoded display name.
    • $workspace.<name>.$id: ID of a specific workspace named <name>.
    • $workspace.<name>.$items.<item_type>.<item_name>.$<attribute>: Attribute of a specific item in a named workspace.

    Item Attribute Variables

    Replaces a value with the attribute of a specific item in the target workspace. Syntax: $items.<item_type>.<item_name>.$<attribute>.

    AttributeSupported ItemsExample
    $idAll$items.Notebook.MyNotebook.$id
    $sqlendpointLakehouse, MirroredDatabase, SQLDatabase, Warehouse$items.Lakehouse.MyLH.$sqlendpoint
    $sqlendpointidLakehouse, MirroredDatabase$items.Lakehouse.MyLH.$sqlendpointid
    $queryserviceuriEventhouse$items.Eventhouse.MyEH.$queryserviceuri

    Important Constraints:

    • Bulk Publish: Dynamic variables are not supported in bulk publish mode. Using them will cause a fallback to standard publishing.
    • Case Sensitivity: Item types and names are case-sensitive. Attributes must be lowercase.
    • Eager Resolution: If any dynamic variable is used, $sqlendpoint and $sqlendpointid are resolved for every compatible item in the target workspace. Ensure these items are fully provisioned before deploying to avoid failure.