Microsoft Fabric Samples

PII SparkShield is a PySpark-based solution designed for detecting and anonymizing Personally Identifiable Information (PII) within Microsoft Fabric. It leverages Microsoft Presidio for NLP-based detection, Faker for synthetic data generation, and PySpark for scalable processing.

It supports three primary approaches:

1. Detection & Redaction: Using Microsoft Presidio to identify PII in structured or unstructured data and replacing it with placeholders (e.g., REDACTED).
2. Synthetic Data Generation: Using Presidio to find PII and Faker to replace it with realistic synthetic data.
3. Built-in PySpark Functions: Using native Spark functions for hashing (e.g., sha2) and masking (e.g., partial hiding of phone numbers) for structured data.

This tutorial series provides a practical guide for building, evaluating, and optimizing Retrieval-Augmented Generation (RAG) applications using Microsoft Fabric. The guide is divided into three core tutorials:

1. Building Retrieval Augmented Generation in Fabric: Learn to set up a RAG pipeline, connect data sources, use Fabric retrieval tools, and integrate with Azure OpenAI.
2. Evaluating the performance of RAG Applications on Fabric: Learn to assess query accuracy, response quality, and system performance to drive data-driven improvements.
3. Exploring Azure AI search in Fabric: Learn to use AI-powered search to enhance RAG applications by retrieving relevant information from large datasets.

For the building phase, you can choose between using Fabric's built-in Azure OpenAI endpoints (no subscription keys required) or bringing your own Azure OpenAI subscription keys.

This tutorial guides you through building advanced GenAI-powered applications that extract knowledge from tribal conversational data (such as online forums). The primary use case is automatically generating FAQs or an adaptive knowledge base to optimize retrieval performance for RAG (Retrieval-Augmented Generation) systems.

Key technical details:

• LLM Endpoint: Azure OpenAI GPT4o
• Platform: Microsoft Fabric
• Data Source: Synthetic Microsoft Teams conversational data (structured like Microsoft Graph API).
• Approach: Uses a set of human-defined themes to generate questions and discussion threads via an LLM to simulate real-world forum data.

The community-samples directory contains community-contributed projects that demonstrate various Microsoft Fabric capabilities.

⚠️ Warning: These samples are provided 'as-is' and are not officially supported by Microsoft. Proceed with caution before running them in your environment.

For support or questions, visit the Microsoft Fabric Community.

The microsoft/fabric-samples repository provides hands-on examples, code, and sample data to help you learn and implement Microsoft Fabric. The repository is organized into three main categories of samples:

• Documentation Samples: Located in the docs-samples folder, these are the official examples used in Microsoft Fabric documentation and Microsoft Learn training content.
• Features Samples: Located in the features-samples folder, these demonstrate how to use specific Microsoft Fabric features.
• Community Samples: Located in the community-samples folder, these are real-world scenarios and creative implementations provided by the community.

The project follows this directory layout:

• src/index.tsx: Application entry point.
• src/App.tsx: Main application component containing logic and UI.
• src/App.css: Application styles.
• src/components/: Directory for custom React components.
• src/operations/: Directory for GraphQL queries and mutations.
• src/Client.ts: Apollo Client configuration (endpoint, cache, and auth headers).
• src/authConfig.ts: Authentication and endpoint configuration.
• graphqlrc.yml: Configuration for GraphQL Intellisense and auto-completion.
• codegen.yml: Configuration for GraphQL code generation.

The server provides two primary MCP tools to interact with your GraphQL API.

Important Workflow: You must run introspect-schema before attempting to run queries. The client needs the schema to understand the available data structures.

1. introspect-schema

• Description: Retrieves the full GraphQL schema from the backend.
• Parameters: None
• Returns: The full GraphQL schema as JSON.

2. query-graphql

• Description: Executes an arbitrary GraphQL query against the backend.
• Parameters:
  • query (string, required): The GraphQL query string.
  • variables (object, optional): GraphQL variables.
• Returns: The query result as JSON.

The end-to-end architecture for enriching Power BI reports with machine learning in Fabric follows these seven steps:

1. Semantic model: Defines the core business logic and metrics.
2. Semantic Link: Provides direct access to the semantic model from Fabric notebooks.
3. Fabric ML experiment: Used to train and evaluate the machine learning model (e.g., a churn prediction model).
4. Batch scoring: Applies the model at scale and saves the resulting predictions to OneLake.
5. Real-time scoring endpoints: Supports low-latency inference for immediate predictions.
6. Dataflow Gen2: Calls the real-time scoring endpoint to enrich data during the ingestion process.
7. Power BI Report: Visualizes the churn risk and drives business actions.

All components are natively integrated within Microsoft Fabric, sharing the same security, governance, and storage layers.

The notebooks in this repository utilize two key Spark optimization techniques to handle large-scale PII processing:

1. Pandas UDFs (Vectorized UDFs): Instead of standard Python UDFs which process data row-by-row, Pandas UDFs process data in batches using Pandas Series. This significantly reduces serialization overhead and leverages vectorized operations, making them much faster for large datasets.

2. Broadcasting: For heavy objects like NLP models or Presidio engines, broadcasting is used to serialize the object only once and send it to each worker node once. This prevents Spark from serializing and sending a separate copy of the model to every single task, which reduces memory usage, network traffic, and job startup time.