Overview of the GenAI Azure Hosting Template
mainThis repository provides a template for hosting and providing APIs for Hugging Face models (via vLLM) and Azure OpenAI models on Microsoft Azure. It is designed to efficiently manage inference workloads using a scalable architecture.
Architecture Components
Common Components:
- Azure API Management (APIM): Acts as the single entry point. It handles external request management, including source IP restrictions, API key authentication, and request routing to the Application Gateway via private IP.
- Azure Application Gateway (APGW): Performs load balancing to backend inference VMs. It uses a WAF policy to validate the
x-apim-secretheader added by APIM, ensuring only authorized APIM instances can communicate with it.
vLLM-Compatible Model API Group:
- Virtual Machine Scale Set (VMSS) / Virtual Machine (VM): Runs inference containers that retrieve models from Hugging Face. The VMSS configuration allows for flexible scaling by adding more VMs.
- Azure Automation: Handles the periodic starting and stopping of VMs to manage costs.
OpenAI API Group:
- Azure Functions: A serverless environment that orchestrates Azure OpenAI APIs. It communicates with Azure OpenAI via private endpoints.
- Azure OpenAI Service: Provides OpenAI APIs, including Chat Completions, Responses, and Code Interpreter-based data analysis (Excel/CSV).