Azure Search OpenAI Demo

The SentenceTextSplitter (located in ../app/backend/prepdocslib/textsplitter.py) is the primary engine for chunking text in the data ingestion pipeline. It is designed to create semantically coherent chunks for Azure AI Search by following these principles:

• Sentence Alignment: Chunks align with sentence boundaries using punctuation (. ! ? and CJK equivalents).
• Token & Character Limits: Enforces a hard limit of 500 tokens per chunk and a soft guideline of 1,000 characters (with a 20% overflow tolerance).
• Atomic Figures: <figure>...</figure> blocks are treated as atomic units. They are never split and are always attached to preceding text if available.
• Boundary Repair: Attempts to re-stitch sentences split by page boundaries.
• Semantic Overlap: Appends a small prefix from the next chunk to the current one to improve retrieval recall.
• Normalization: Performs minimal whitespace trimming to prevent trivial overflows without altering original formatting.

The sample-app-aoai-chatGPT repository (https://github.com/Microsoft/sample-app-aoai-chatGPT/) is designed for users setting up via Azure OpenAI Studio and the Azure Portal.

Key Differences

• RAG Approach: This repository uses multiple RAG approaches that chain results from multiple API calls (Azure OpenAI and ACS) for greater flexibility. sample-app-aoai-chatGPT uses the built-in data sources option in the Azure OpenAI API.
• Experimental Nature: This repository is more experimental as it is not tied to the Azure OpenAI Studio.

Feature Comparison

Technology Comparison

The Azure Chat Solution Accelerator (https://github.com/microsoft/azurechat) is a popular alternative.

Key Differences

• Stack: azurechat uses a full JavaScript/TypeScript stack with a Node.js backend, whereas this repository uses Python (Quart).
• Focus: azurechat emphasizes user personalization, while this repository focuses on enterprise scenarios like document-level Access Control Lists (ACLs) and evaluation.

Feature Comparison

Technology Comparison

The data ingestion pipeline transforms raw documents into searchable content in Azure AI Search through three main stages:

1. Document extraction: Extracts text, tables (converted to HTML), and figures (with bounding boxes) from source files. It uses Azure Document Intelligence for complex formats (PDF, HTML, DOCX, PPTX, XLSX, Images) or local parsers (PyPDF, BeautifulSoup, etc.) for simpler ones.
2. Figure processing (Optional): If multimodal support is enabled, figures are cropped as PNGs, described using GPT-4 Vision or Azure AI Content Understanding, uploaded to Blob Storage, and optionally embedded.
3. Text processing: Merges figure descriptions back into the text, splits content into semantic chunks (default ~1000 characters with 10% overlap), and computes vector embeddings using Azure OpenAI models.

Finally, the chunks are indexed into Azure AI Search as individual documents.

The recommended way to add document-level access control is using Cloud Ingestion with Azure Data Lake Storage (ADLS) Gen2. This method uses Azure Functions to extract POSIX-style ACLs from files and index them into Azure AI Search.

Workflow

1. Store documents in an ADLS Gen2 account with hierarchical namespace enabled.
2. Set ACLs on files/folders to control user/group access.
3. An Azure AI Search indexer triggers Azure Functions (Document Extractor, Figure Processor, Text Processor) to parse documents and extract ACLs.
4. Extracted User IDs (oids) and Group IDs (groups) are stored in the search index.

Setup Instructions

1. Enable cloud ingestion and ACLs in your environment:

azd env set USE_CLOUD_INGESTION true
azd env set USE_CLOUD_INGESTION_ACLS true
azd env set AZURE_USE_AUTHENTICATION true
azd env set AZURE_ENFORCE_ACCESS_CONTROL true

2. Deploy the application:

azd up

3. Upload documents with ACLs: You can use the provided script to upload sample data with ACLs:

python scripts/adlsgen2setup.py './data/*' --data-access-control './scripts/sampleacls.json' -v

4. Trigger ingestion:

./scripts/setup_cloud_ingestion.sh