DOCSEARCHbyAlgolia
Submit Docs

AI Gateway Labs

repository·main·Indexed 21 days ago

https://github.com/azure-samples/ai-gateway

A collection of hands-on learning resources and tools for building enterprise-grade AI applications. It demonstrates the use of Azure API Management to manage AI models, agents, and tools via MCP, with a focus on security, reliability, and cost control. The repository includes Jupyter notebooks, Bicep files, and APIM policies, covering topics such as advanced load balancing, backend circuit breaking, content filtering, and developer tooling for AI Service APIs.

Tokens
185.9K
Snippets
436
Records
687
Agent score
75%

What's inside ai-gateway

  1. Overview of AI Gateway Foundry Models Lab

    main

    The AI Gateway Foundry Models lab is a playground for testing the AI Gateway tier of Azure API Management (public preview). This tier allows you to publish Microsoft Foundry models behind a single, stable gateway endpoint.

    Key capabilities demonstrated include:

    • Governance via Policy Cards: Instead of XML, use policy cards to manage token rate limits (leaky bucket), quotas, Azure AI Content Safety, and model fallback.
    • Standardized API Formats: Call models using OpenAI Chat Completions and Responses formats.
    • Observability: Monitor token-usage metrics via OpenTelemetry, flowing into Application Insights and managed Prometheus, broken down by model and subscription.
    • Streaming: Support for streaming responses and measuring Time-to-First-Token (TTFT).

    Note: The AI Gateway tier is in public preview and is currently available in East US 2 and Sweden Central.

  2. Overview of Foundry Models Evals

    main

    The Foundry Models Evals lab demonstrates how to implement LLMOps workflows by extracting LLM request/response data from Azure API Management (APIM) and running quality evaluations in Microsoft Foundry.

    Workflow:

    1. Extract: Query the ApiManagementGatewayLlmLog in APIM to retrieve prompts and completions.
    2. Transform: Convert these logs into a JSONL format compatible with Microsoft Foundry.
    3. Evaluate: Use the Microsoft Foundry SDK to run cloud evaluations using built-in evaluators such as coherence, fluency, and groundedness.

  3. Overview of Realtime Audio with MCP Agents

    main

    This lab provides a playground to experiment with the Azure OpenAI Realtime API for both text and audio interactions. It integrates the Model Context Protocol (MCP) with Azure API Management (APIM) to enable 'plug & play' tool capabilities for Large Language Models (LLMs).

    Key features include:

    • Tool Integration: Uses MCP servers to provide tools for weather, Spotify, and ServiceNow.
    • Secure Authentication: Leverages the APIM credential manager for managing OAuth 2.0 tokens for backend tools and client token validation to ensure end-to-end authorization.
    • MCP Servers included:
      • Weather Service: Tools to get cities by country and retrieve weather info.
      • Spotify Service: Tools for authentication (via APIM Credential Manager), retrieving playlists/latest releases, and controlling playback.
      • ServiceNow Incidents: Tools for authentication (via APIM Credential Manager), listing, retrieving, and creating incidents.

  4. Overview of Backend Pool Load Balancing with APIM and OpenAI

    main

    This lab demonstrates how to use Azure API Management (APIM) built-in backend pool functionality to load balance requests across multiple Azure OpenAI endpoints using Terraform for infrastructure as code.

    It specifically implements a prioritized PTU (Provisioned Throughput Unit) with fallback consumption scenario. In this configuration:

    1. A Priority 1 (highest) backend is used first.
    2. Once the Priority 1 backend is exhausted, the system gracefully falls back to two equally-weighted Priority 2 backends.

  6. Overview of Serverless GPU Inference Lab

    main

    This lab demonstrates how to deploy the SGLang high-performance LLM serving framework on Azure Container Apps using NVIDIA A100 GPU workload profiles. The deployment exposes an OpenAI-compatible inference endpoint through Azure API Management (APIM) for secure, managed access.

    Key capabilities demonstrated:

    • Deploying GPU-accelerated Container Apps using Consumption-GPU-NC24-A100 workload profiles.
    • Running SGLang via its official Docker image.
    • Exposing SGLang's OpenAI-compatible API through APIM with subscription key authentication.
    • Interacting with self-hosted models using the standard OpenAI Python SDK.
    • Supporting streaming for real-time token generation.

  7. Overview of the FinOps Framework Lab

    main

    The FinOps Framework lab demonstrates how to implement comprehensive cost tracking and optimization for AI services using Azure API Management (APIM).

    Key capabilities include:

    • Cost Control: Using APIM token limit policies applied to specific products.
    • Automated Governance: Integrating Azure Monitor alerts with Logic Apps to automatically disable APIM subscriptions that exceed defined cost quotas.
    • Visibility: Building dashboards for cost visibility and implementing chargeback models for different teams or customers.

  8. Overview of MCP A2A Agents Lab

    main

    The MCP A2A Agents lab is a playground for experimenting with Agent-to-Agent (A2A) enabled agents using the Model Context Protocol (MCP) via Azure API Management (APIM).

    It demonstrates a heterogeneous multi-agent system where agents built with different orchestrators—specifically Semantic Kernel and Autogen—communicate through APIM for authentication and authorization. The lab uses APIM's credential manager to handle OAuth 2.0 tokens for backend tools and client token validation to ensure end-to-end security.

    Included MCP servers:

    • Basic oncall service: Provides tools to retrieve a list of random on-call personnel, their status, and time zones.
    • Basic weather service: Provides tools to find cities in a country and retrieve weather information for a specific city.
    • GitHub Issues MCP Server: Uses APIM Credential Manager for GitHub OAuth to retrieve user info and list repository issues.

  9. Overview of AI Foundry Deepseek Lab

    main

    This lab provides a playground to test the Deepseek R1 model using the Azure AI Model Inference API.

    It demonstrates how to manage and monitor token usage in an enterprise environment by integrating Azure API Management (APIM) with Azure AI Foundry. The architecture utilizes two specific APIM LLM policies:

    • llm-token-limit: To manage and restrict token consumption.
    • llm-emit-token-metric: To monitor and emit token usage metrics.

  10. Overview of AI Gateway Tools

    main

    The AI Gateway Tools provide specialized capabilities for interacting with AI Foundry model APIs:

    • Tracing: Enables invocation of AI Foundry model APIs with tracing enabled. It returns detailed tracing information for monitoring and debugging.
    • Streaming: Enables invocation of AI Foundry model APIs with streaming enabled. It returns the model response in chunks rather than a single block.
    • AI-Gateway Mock server: A simulation environment designed to mimic the behavior and responses of the OpenAI API. This is useful for testing integrations with APIM (API Management) or other use cases without hitting live endpoints. The server implementation in mock-server/app.py can be customized for specific testing requirements.

  11. Overview of Foundry IQ Agent Service Integration

    main

    This lab demonstrates how to integrate the Foundry Agent Service with a Foundry IQ Knowledge Base using the Model Context Protocol (MCP). In this architecture, Azure API Management (APIM) acts as the AI Gateway, managing all agent inference and embedding traffic.

    Core Components

    • Azure AI Search Knowledge Base: An agentic retrieval pipeline utilizing both vector and semantic search.
    • APIM AI Gateway: Provides managed identity authentication, token metrics emission, token rate limiting, load balancing, and centralized observability via Application Insights.
    • Foundry Agent: An agent equipped with the knowledge_base_retrieve MCP tool, enabling it to provide grounded answers with citations.

    Supported Invocation Patterns

    • Conversations API: A simplified invocation pattern.
    • Classic Agent API: A pattern designed for multi-turn threads.

  12. Overview of the Gemini + MCP Agents + Content Safety lab

    main

    This lab provides a playground to experiment with the Model Context Protocol using Azure API Management (APIM). It demonstrates how to enable plug-and-play tool access for LLMs while integrating third-party models like Google Gemini.

    Key capabilities demonstrated:

    • Multi-Cloud AI: Exposing Gemini capabilities alongside Azure OpenAI through a unified gateway.
    • MCP Server Integration: Implementing MCP servers to provide tools to LLMs.
    • Governance: Enforcing Content Safety and monitoring Token metrics through the AI Gateway to create a single pane of glass for AI services.

    Included MCP Servers:

    • Basic oncall service: Provides tools to retrieve a list of random people currently on-call, including their status and time zone.
    • Basic weather service: Provides tools to retrieve cities for a given country and random weather information for a specified city.